feat: Add duckdb connector (#308)

* refactor(duckdb): extract shared json-safe bigint helper

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(duckdb): add and register the duckdb primary connector

Add KtxDuckDbDialect, KtxDuckDbScanConnector (local file-backed, read-only,
never-create, main-schema introspection via information_schema and
duckdb_constraints() for foreign keys), and register the duckdb driver across
the dialect factory, driver registry, connection-type enum, warehouse descriptor,
config schema, scan normalization, connection test drivers, and status display.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(duckdb): route live-database ingest through the DuckDB connector

Add the DuckDB live-database introspection bridge and dispatch duckdb
connections to it in local-adapters, matching the SQLite path. Repoint the
config-rejection test off duckdb (now a valid driver) onto the no-driver case.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(duckdb): add duckdb to the setup database flow

Offer DuckDB in the interactive checklist and via ktx setup --database duckdb,
with a file-path prompt and duckdb-local default connection id, parallel to SQLite.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(duckdb): attach native duckdb files in federation

Native .duckdb members ATTACH with (READ_ONLY) and no TYPE/INSTALL/LOAD, since
the duckdb format needs no extension. attachTypeForDriver returns null for the
native case; buildAttachStatements builds load statements from non-null types
only and emits a conditional ATTACH clause.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(duckdb): document the duckdb primary-source connector

Add a DuckDB section to the primary-sources integration page (config, read-only
never-create behavior, main-schema scope, federation) and update the
supported-driver assertion in dialects.test.ts to include duckdb.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(duckdb): use single-namespace display shape for main-only refs

DuckDB v1 introspects the main schema and sets db=null on every table, so its
display refs are single-namespace like SQLite. The ansi shape emitted a 1-part
table display it then refused to parse, breaking column-level display resolution.
Switch the dialect to the sqlite display shape and add a round-trip test plus a
composite-foreign-key test that were missing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(duckdb): resolve connector dialect via getDialectForDriver

Route the connector's dialect through the shared factory like every other
connector, now that duckdb is registered. Single construction path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(duckdb): skip schema picker for single-file duckdb setup

DuckDB is a single-file, single-namespace ('main') database like SQLite,
but the setup scope step only skipped the schema picker for sqlite. DuckDB
fell into the multi-schema path with an empty schema list, rendering a
broken picker ("No matches found" for main). Extend the file-based-driver
early-return to cover duckdb so it ingests every table directly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(duckdb): reuse shared config helper and derive scope skip

Route duckdb path resolution through the shared resolveStringReference
helper instead of a local third copy of env:/file: handling. Derive the
setup scope-picker skip from SCOPE_DISCOVERY_SPECS membership rather than
a hardcoded sqlite/duckdb driver list.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(duckdb): use a genuinely unknown driver in the rejection test

The merged "rejects unknown drivers" test used `driver: duckdb` as its
unknown-driver stand-in, which stopped being unknown once this branch
added the duckdb connector. Switch to `nonsense` so it again exercises
the unsupported-driver config error.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(duckdb): cover dialect, connector, and live-introspection branches

Codecov flagged uncovered branches as dead code; all are real connector,
dialect, and live-ingest behavior. Add unit tests instead of removing them.

- dialect: precedence ladder, sample/clause builders, profiling expressions
- connector: url/env config forms, error throws, never-create guard,
  cardinality cap branches, table-scope empty/non-empty paths
- live-introspection: full-schema and table-scope extraction

Functions 100%, lines ~99% across the duckdb connector dir.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: add DuckDB to supported-driver references

The DuckDB connector PR documented the connector itself but left the
scattered supported-driver enumerations stale. Add duckdb to the
federation concept page (participation table, activation, table naming,
limitations), the ktx setup CLI reference, the ktx.yaml warehouse-driver
table, the primary-sources field reference, and the quickstart driver
list (which also restores the missing ClickHouse entry).

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Andrey Avtomonov <andreybavt@gmail.com>
This commit is contained in:
Kevin Messiaen 2026-07-01 19:06:02 +07:00 committed by GitHub
parent f21594c42a
commit 3c4fcc27c7
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
39 changed files with 1366 additions and 59 deletions

View file

@ -688,7 +688,7 @@ describe('runKtxConnection', () => {
await initKtxProject({ projectDir });
await writeFile(
join(projectDir, 'ktx.yaml'),
'connections:\n mystery:\n driver: duckdb\n',
'connections:\n mystery:\n driver: nonsense\n',
'utf-8',
);
const io = makeIo();

View file

@ -0,0 +1,280 @@
import { DuckDBInstance } from '@duckdb/node-api';
import { mkdtemp, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { pathToFileURL } from 'node:url';
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import {
KtxDuckDbScanConnector,
duckDbDatabasePathFromConfig,
isKtxDuckDbConnectionConfig,
} from '../../../src/connectors/duckdb/connector.js';
import { tableRefSet } from '../../../src/context/scan/table-ref.js';
let dir: string;
let dbPath: string;
beforeAll(async () => {
dir = await mkdtemp(join(tmpdir(), 'ktx-duckdb-'));
dbPath = join(dir, 'warehouse.duckdb');
const instance = await DuckDBInstance.create(dbPath);
const connection = await instance.connect();
await connection.run('CREATE TABLE customers (id BIGINT PRIMARY KEY, name VARCHAR, big BIGINT)');
await connection.run(
`INSERT INTO customers VALUES (1, 'Ada', 9223372036854775807), (2, 'Lin', 10)`,
);
await connection.run('CREATE TABLE orders (id BIGINT, customer_id BIGINT REFERENCES customers(id))');
await connection.run('INSERT INTO orders VALUES (1, 1), (2, 2)');
// Composite primary key + composite foreign key, to exercise the parallel
// unnest() zip of constraint/referenced column names in readForeignKeys.
await connection.run('CREATE TABLE regions (country VARCHAR, code VARCHAR, PRIMARY KEY (country, code))');
await connection.run(
'CREATE TABLE stores (id BIGINT, country VARCHAR, code VARCHAR, FOREIGN KEY (country, code) REFERENCES regions(country, code))',
);
await connection.run('CREATE TABLE empty_table (id BIGINT)');
connection.closeSync();
instance.closeSync();
});
afterAll(async () => {
await rm(dir, { recursive: true, force: true });
});
function connector(connection: Record<string, unknown> = { driver: 'duckdb', path: dbPath }) {
return new KtxDuckDbScanConnector({ connectionId: 'warehouse', connection, projectDir: dir });
}
describe('isKtxDuckDbConnectionConfig', () => {
it('accepts duckdb driver, rejects others', () => {
expect(isKtxDuckDbConnectionConfig({ driver: 'duckdb' })).toBe(true);
expect(isKtxDuckDbConnectionConfig({ driver: 'sqlite' })).toBe(false);
});
});
describe('duckDbDatabasePathFromConfig', () => {
it('resolves a relative path against projectDir', () => {
const resolved = duckDbDatabasePathFromConfig({
connectionId: 'warehouse',
projectDir: dir,
connection: { driver: 'duckdb', path: 'warehouse.duckdb' },
});
expect(resolved).toBe(dbPath);
});
it('derives the path from a file: url', () => {
const resolved = duckDbDatabasePathFromConfig({
connectionId: 'warehouse',
connection: { driver: 'duckdb', url: pathToFileURL(dbPath).href },
});
expect(resolved).toBe(dbPath);
});
it('derives the path from a duckdb: url', () => {
const resolved = duckDbDatabasePathFromConfig({
connectionId: 'warehouse',
connection: { driver: 'duckdb', url: `duckdb://${dbPath}` },
});
expect(resolved).toBe(dbPath);
});
it('resolves an env: reference in path', () => {
process.env.KTX_TEST_DUCKDB_PATH = dbPath;
try {
const resolved = duckDbDatabasePathFromConfig({
connectionId: 'warehouse',
connection: { driver: 'duckdb', path: 'env:KTX_TEST_DUCKDB_PATH' },
});
expect(resolved).toBe(dbPath);
} finally {
delete process.env.KTX_TEST_DUCKDB_PATH;
}
});
it('rejects a non-duckdb driver', () => {
expect(() =>
duckDbDatabasePathFromConfig({
connectionId: 'warehouse',
connection: { driver: 'sqlite', path: dbPath },
}),
).toThrow(/cannot run driver "sqlite"/);
});
it('requires a path or url', () => {
expect(() =>
duckDbDatabasePathFromConfig({
connectionId: 'warehouse',
connection: { driver: 'duckdb' },
}),
).toThrow(/requires connections\.warehouse\.path or url/);
});
});
describe('KtxDuckDbScanConnector', () => {
it('testConnection succeeds for an existing file', async () => {
const c = connector();
expect(await c.testConnection()).toEqual({ success: true });
await c.cleanup();
});
it('testConnection fails (never creating) for a missing file', async () => {
const c = connector({ driver: 'duckdb', path: join(dir, 'absent.duckdb') });
const result = await c.testConnection();
expect(result.success).toBe(false);
await c.cleanup();
});
it('introspects main-schema tables, columns, and foreign keys', async () => {
const c = connector();
const snapshot = await c.introspect({ connectionId: 'warehouse', driver: 'duckdb' }, { runId: 't' });
const names = snapshot.tables.map((t) => t.name).sort();
expect(names).toEqual(['customers', 'empty_table', 'orders', 'regions', 'stores']);
const orders = snapshot.tables.find((t) => t.name === 'orders');
expect(orders?.foreignKeys[0]).toMatchObject({ fromColumn: 'customer_id', toTable: 'customers', toColumn: 'id' });
await c.cleanup();
});
it('maps a composite foreign key column-for-column to the referenced table', async () => {
const c = connector();
const snapshot = await c.introspect({ connectionId: 'warehouse', driver: 'duckdb' }, { runId: 't' });
const stores = snapshot.tables.find((t) => t.name === 'stores');
const fks = stores?.foreignKeys.map((fk) => ({ fromColumn: fk.fromColumn, toTable: fk.toTable, toColumn: fk.toColumn }));
expect(fks).toEqual([
{ fromColumn: 'country', toTable: 'regions', toColumn: 'country' },
{ fromColumn: 'code', toTable: 'regions', toColumn: 'code' },
]);
await c.cleanup();
});
it('lists tables', async () => {
const c = connector();
const tables = (await c.listTables()).map((t) => t.name).sort();
expect(tables).toEqual(['customers', 'empty_table', 'orders', 'regions', 'stores']);
await c.cleanup();
});
it('samples a table', async () => {
const c = connector();
const sample = await c.sampleTable(
{ connectionId: 'warehouse', table: { name: 'customers', catalog: null, db: null }, limit: 1 },
{ runId: 't' },
);
expect(sample.rows.length).toBe(1);
await c.cleanup();
});
it('stringifies BIGINT beyond 2^53 in read-only results', async () => {
const c = connector();
const result = await c.executeReadOnly(
{ connectionId: 'warehouse', sql: 'SELECT big FROM customers WHERE id = 1', maxRows: 10 },
{ runId: 't' },
);
expect(result.rows[0][0]).toBe('9223372036854775807');
await c.cleanup();
});
it('rejects non-read-only SQL', async () => {
const c = connector();
await expect(
c.executeReadOnly({ connectionId: 'warehouse', sql: 'DELETE FROM customers', maxRows: 10 }, { runId: 't' }),
).rejects.toThrow();
await c.cleanup();
});
it('returns distinct values under the cardinality cap', async () => {
const c = connector();
const distinct = await c.getColumnDistinctValues({ name: 'customers', catalog: null, db: null }, 'name', {
maxCardinality: 10,
limit: 10,
});
expect(distinct?.values?.sort()).toEqual(['Ada', 'Lin']);
await c.cleanup();
});
it('withholds values but reports the count when cardinality exceeds the cap', async () => {
const c = connector();
const distinct = await c.getColumnDistinctValues({ name: 'customers', catalog: null, db: null }, 'name', {
maxCardinality: 1,
limit: 10,
});
expect(distinct).toEqual({ values: null, cardinality: 2 });
await c.cleanup();
});
it('samples a single column, dropping null rows', async () => {
const c = connector();
const sample = await c.sampleColumn(
{ connectionId: 'warehouse', table: { name: 'customers', catalog: null, db: null }, column: 'name', limit: 10 },
{ runId: 't' },
);
expect(sample.values.sort()).toEqual(['Ada', 'Lin']);
expect(sample.nullCount).toBeNull();
await c.cleanup();
});
it('counts table rows', async () => {
const c = connector();
expect(await c.getTableRowCount('customers')).toBe(2);
await c.cleanup();
});
it('lists only the main schema and reports no column stats', async () => {
const c = connector();
expect(await c.listSchemas()).toEqual(['main']);
expect(await c.columnStats({ connectionId: 'warehouse', table: { name: 'customers', catalog: null, db: null }, column: 'id' }, { runId: 't' })).toBeNull();
await c.cleanup();
});
it('rejects operations for a mismatched connection id', async () => {
const c = connector();
await expect(
c.executeReadOnly({ connectionId: 'other', sql: 'SELECT 1', maxRows: 1 }, { runId: 't' }),
).rejects.toThrow(/cannot serve connection other/);
await c.cleanup();
});
it('exposes the dialect identifier quoting', () => {
expect(connector().quoteIdentifier('a"b')).toBe('"a""b"');
});
// Opening a connection must never create the file: the db() guard throws
// rather than letting DuckDBInstance.create() materialize a missing path.
it('refuses to open (never creating) a missing file when a query runs', async () => {
const c = connector({ driver: 'duckdb', path: join(dir, 'absent.duckdb') });
await expect(c.listTables()).rejects.toThrow(/File not found/);
await c.cleanup();
});
it('returns no tables for an empty table scope', async () => {
const c = connector();
const snapshot = await c.introspect(
{ connectionId: 'warehouse', driver: 'duckdb', tableScope: new Set() },
{ runId: 't' },
);
expect(snapshot.tables).toEqual([]);
await c.cleanup();
});
it('restricts introspection to the named tables in a non-empty scope', async () => {
const c = connector();
const snapshot = await c.introspect(
{
connectionId: 'warehouse',
driver: 'duckdb',
tableScope: tableRefSet([{ catalog: null, db: null, name: 'customers' }]),
},
{ runId: 't' },
);
expect(snapshot.tables.map((t) => t.name)).toEqual(['customers']);
await c.cleanup();
});
it('reports zero cardinality and an empty value list for an empty column', async () => {
const c = connector();
const distinct = await c.getColumnDistinctValues({ name: 'empty_table', catalog: null, db: null }, 'id', {
maxCardinality: 10,
limit: 10,
});
expect(distinct).toEqual({ values: [], cardinality: 0 });
await c.cleanup();
});
});

View file

@ -0,0 +1,108 @@
import { describe, expect, it } from 'vitest';
import { KtxDuckDbDialect } from '../../../src/connectors/duckdb/dialect.js';
describe('KtxDuckDbDialect', () => {
const dialect = new KtxDuckDbDialect();
it('quotes identifiers with double quotes and escapes embedded quotes', () => {
expect(dialect.quoteIdentifier('order"s')).toBe('"order""s"');
});
it('maps integer types to number dimension', () => {
expect(dialect.mapToDimensionType('BIGINT')).toBe('number');
expect(dialect.mapToDimensionType('DOUBLE')).toBe('number');
});
it('maps timestamp types to time dimension', () => {
expect(dialect.mapToDimensionType('TIMESTAMP')).toBe('time');
expect(dialect.mapToDimensionType('DATE')).toBe('time');
});
it('maps text types to string dimension', () => {
expect(dialect.mapToDimensionType('VARCHAR')).toBe('string');
});
it('maps boolean types to boolean dimension', () => {
expect(dialect.mapToDimensionType('BOOLEAN')).toBe('boolean');
expect(dialect.mapToDimensionType('BOOL')).toBe('boolean');
});
it('falls back to string for an empty or unknown native type', () => {
expect(dialect.mapToDimensionType('')).toBe('string');
expect(dialect.mapToDimensionType('JSON')).toBe('string');
});
// The precedence ladder strips parameters before substring rules fire, so a
// parameterized DECIMAL still resolves through the numeric branch rather than
// the string fallback.
it('strips type parameters before resolving the dimension', () => {
expect(dialect.mapToDimensionType('DECIMAL(10,2)')).toBe('number');
expect(dialect.mapToDimensionType('VARCHAR(255)')).toBe('string');
});
// Types absent from the exact-match table still resolve via substring rules:
// TIMESTAMP_NS (time), UINT128/HUGEINT-like (number), and lowercase input.
it('resolves unlisted types through substring matching, case-insensitively', () => {
expect(dialect.mapToDimensionType('timestamp_ns')).toBe('time');
expect(dialect.mapToDimensionType('INT128')).toBe('number');
expect(dialect.mapToDimensionType(' double ')).toBe('number');
});
it('generates a limited sample query', () => {
expect(dialect.generateSampleQuery('"t"', 5)).toBe('SELECT * FROM "t" LIMIT 5');
});
it('quotes selected columns in a sample query', () => {
expect(dialect.generateSampleQuery('"t"', 5, ['a', 'b'])).toBe('SELECT "a", "b" FROM "t" LIMIT 5');
});
it('builds a non-null, non-blank column sample query', () => {
expect(dialect.generateColumnSampleQuery('"t"', 'email', 3)).toBe(
`SELECT "email" FROM "t" WHERE "email" IS NOT NULL AND TRIM(CAST("email" AS VARCHAR)) != '' LIMIT 3`,
);
});
// A degenerate sample percentage (<=0 or >=1) means "no sampling", so both the
// random filter and the TABLESAMPLE clause must collapse to an empty string.
it('returns empty sample clauses outside the (0,1) range and real clauses inside it', () => {
expect(dialect.getRandomSampleFilter(0)).toBe('');
expect(dialect.getRandomSampleFilter(1)).toBe('');
expect(dialect.getRandomSampleFilter(0.25)).toBe('RANDOM() < 0.25');
expect(dialect.getTableSampleClause(0)).toBe('');
expect(dialect.getTableSampleClause(0.1)).toBe('USING SAMPLE 10 PERCENT (bernoulli)');
});
// A type missing from the exact-match table but containing BOOL still resolves
// through the substring branch rather than the string fallback.
it('resolves a BOOL-substring type to boolean', () => {
expect(dialect.mapToDimensionType('MYBOOL')).toBe('boolean');
});
it('builds limit/offset, sample-value aggregation, and randomized cardinality clauses', () => {
expect(dialect.getLimitOffsetClause(10, 5)).toContain('LIMIT 10');
expect(dialect.getSampleValueAggregation('SELECT 1')).toContain('STRING_AGG');
expect(dialect.generateRandomizedCardinalitySampleQuery('"t"', 'c', 100)).toContain('USING SAMPLE 100 ROWS');
});
it('exposes profiling expressions and a null column-statistics query', () => {
expect(dialect.getNullCountExpression('c')).toBe('SUM(CASE WHEN c IS NULL THEN 1 ELSE 0 END)');
expect(dialect.getDistinctCountExpression('c')).toBe('COUNT(DISTINCT c)');
expect(dialect.textLengthExpression('c')).toBe('LENGTH(CAST(c AS VARCHAR))');
expect(dialect.castToText('c')).toBe('CAST(c AS VARCHAR)');
expect(dialect.mapDataType('BIGINT')).toBe('BIGINT');
expect(dialect.getTopClause(5)).toBe('');
expect(dialect.generateColumnStatisticsQuery('main', 't')).toBeNull();
});
// Guards the single-namespace (db=null) display shape: v1 introspects only
// `main`, so a display ref must round-trip as a bare table name. An ANSI shape
// would emit a 1-part name it then refuses to parse, breaking column lookups.
it('round-trips a single-namespace display ref and reports a 1-part column shape', () => {
const table = { catalog: null, db: null, name: 'orders' };
const display = dialect.formatDisplayRef(table);
expect(display).toBe('orders');
expect(dialect.parseDisplayRef(display)).toMatchObject({ name: 'orders' });
expect(dialect.columnDisplayTablePartCount()).toBe(1);
expect(dialect.formatTableName(table)).toBe('"orders"');
});
});

View file

@ -135,6 +135,14 @@ describe('federatedAttachTarget', () => {
expect(target).toContain('ssl_mode=REQUIRED');
});
it('resolves a duckdb member to its database file path', () => {
const target = federatedAttachTarget(
{ connectionId: 'dux', driver: 'duckdb', projectDir: '/p', connection: { driver: 'duckdb', path: 'a.duckdb' } },
{},
);
expect(target).toBe('/p/a.duckdb');
});
it('throws for an unsupported driver', () => {
expect(() => federatedAttachTarget(member({ driver: 'snowflake', connection: { driver: 'snowflake' } }), {})).toThrow(
/cannot be attached/i,

View file

@ -67,4 +67,27 @@ describe('buildAttachStatements', () => {
);
expect(stmts.at(-1)).toBe('ATTACH \'postgresql://u:it\'\'s@h/db\' AS "pg" (TYPE postgres, READ_ONLY);');
});
it('attaches a native duckdb member with no TYPE and no INSTALL/LOAD', () => {
const statements = buildAttachStatements(
[{ connectionId: 'dux', driver: 'duckdb', projectDir: '/p', connection: { driver: 'duckdb', path: '/p/a.duckdb' } }],
{},
);
expect(statements.some((s) => s.startsWith('INSTALL'))).toBe(false);
expect(statements.find((s) => s.startsWith('ATTACH'))).toContain('(READ_ONLY)');
expect(statements.find((s) => s.startsWith('ATTACH'))).not.toContain('TYPE');
});
it('mixes a duckdb member with a postgres member, loading only postgres', () => {
const statements = buildAttachStatements(
[
{ connectionId: 'dux', driver: 'duckdb', projectDir: '/p', connection: { driver: 'duckdb', path: '/p/a.duckdb' } },
{ connectionId: 'pg', driver: 'postgres', projectDir: '/p', connection: { driver: 'postgres', url: 'postgres://h/db' } },
],
{},
);
expect(statements).toContain('INSTALL postgres; LOAD postgres;');
expect(statements.some((s) => s.includes('INSTALL duckdb'))).toBe(false);
expect(statements.filter((s) => s.startsWith('ATTACH')).length).toBe(2);
});
});

View file

@ -0,0 +1,45 @@
import { DuckDBInstance } from '@duckdb/node-api';
import { mkdtemp, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import { createDuckDbLiveDatabaseIntrospection } from '../../../src/connectors/duckdb/live-database-introspection.js';
import { tableRefSet } from '../../../src/context/scan/table-ref.js';
let dir: string;
let dbPath: string;
beforeAll(async () => {
dir = await mkdtemp(join(tmpdir(), 'ktx-duckdb-live-'));
dbPath = join(dir, 'warehouse.duckdb');
const instance = await DuckDBInstance.create(dbPath);
const connection = await instance.connect();
await connection.run('CREATE TABLE customers (id BIGINT, name VARCHAR)');
await connection.run('CREATE TABLE orders (id BIGINT)');
connection.closeSync();
instance.closeSync();
});
afterAll(async () => {
await rm(dir, { recursive: true, force: true });
});
function port() {
return createDuckDbLiveDatabaseIntrospection({
projectDir: dir,
connections: { warehouse: { driver: 'duckdb', path: dbPath } },
});
}
describe('createDuckDbLiveDatabaseIntrospection', () => {
it('extracts the full schema for a connection', async () => {
const snapshot = await port().extractSchema('warehouse');
expect(snapshot.tables.map((t) => t.name).sort()).toEqual(['customers', 'orders']);
});
it('restricts extraction to a table scope', async () => {
const tableScope = tableRefSet([{ catalog: null, db: null, name: 'customers' }]);
const snapshot = await port().extractSchema('warehouse', { tableScope });
expect(snapshot.tables.map((t) => t.name)).toEqual(['customers']);
});
});

View file

@ -0,0 +1,17 @@
import { describe, expect, it } from 'vitest';
import { jsonSafeBigint, toJsonSafeRows } from '../../../src/connectors/shared/duckdb-json-safe.js';
describe('duckdb json-safe bigint', () => {
it('keeps safe-range bigints as numbers', () => {
expect(jsonSafeBigint(42n)).toBe(42);
});
it('stringifies bigints beyond Number.MAX_SAFE_INTEGER', () => {
const big = BigInt(Number.MAX_SAFE_INTEGER) + 10n;
expect(jsonSafeBigint(big)).toBe(big.toString());
});
it('converts only bigint cells in a row matrix', () => {
expect(toJsonSafeRows([[1n, 'a', null]])).toEqual([[1, 'a', null]]);
});
});

View file

@ -305,7 +305,7 @@ describe('getDialectForDriver', () => {
it('throws with a supported-driver list for unknown drivers', () => {
expect(() => getDialectForDriver('oracle')).toThrow(
'Unsupported driver "oracle". Supported drivers: bigquery, clickhouse, mongodb, mysql, postgres, snowflake, sqlite, sqlserver',
'Unsupported driver "oracle". Supported drivers: bigquery, clickhouse, duckdb, mongodb, mysql, postgres, snowflake, sqlite, sqlserver',
);
});

View file

@ -22,6 +22,7 @@ const connectionFixtures: Record<KtxConnectionDriver, FixtureFactory> = {
schemas: ['public'],
}),
sqlite: () => ({ driver: 'sqlite', path: 'warehouse.db' }),
duckdb: (projectDir) => ({ driver: 'duckdb', path: join(projectDir, 'warehouse.duckdb') }),
mongodb: () => ({
driver: 'mongodb',
url: 'mongodb://localhost:27017/app',
@ -101,6 +102,7 @@ describe('driverRegistrations', () => {
expect(listSupportedDrivers()).toEqual([
'bigquery',
'clickhouse',
'duckdb',
'mongodb',
'mysql',
'postgres',
@ -138,7 +140,7 @@ describe('driverRegistrations', () => {
expect(connector.listTables).toEqual(expect.any(Function));
await connector.cleanup?.();
if (registration.driver === 'sqlite') {
if (registration.driver === 'sqlite' || registration.driver === 'duckdb') {
expect(registration.scopeConfigKey).toBeNull();
} else {
expect(registration.scopeConfigKey).not.toBeNull();

View file

@ -33,8 +33,7 @@ describe('per-dialect SQL notes', () => {
});
it('does not author notes for unreachable dialects', () => {
// duckdb/databricks appear in the resolver map but no connector produces them.
expect(DIALECTS_WITH_NOTES).not.toContain('duckdb');
// databricks appears in the resolver map but no connector produces it.
expect(DIALECTS_WITH_NOTES).not.toContain('databricks');
});

View file

@ -92,7 +92,7 @@ describe('createKtxCliScanConnector', () => {
expect(bigQueryMock.constructorInputs[0]).not.toHaveProperty('maxBytesBilled');
});
it('rejects daemon-only fallback driver configs at config parse time', async () => {
it('resolves a duckdb connection to the DuckDB scan connector', async () => {
await initKtxProject({ projectDir: tempDir });
await writeFile(
join(tempDir, 'ktx.yaml'),
@ -105,10 +105,12 @@ describe('createKtxCliScanConnector', () => {
].join('\n'),
'utf-8',
);
const project = await loadKtxProject({ projectDir: tempDir });
await expect(loadKtxProject({ projectDir: tempDir })).rejects.toThrow(
/connections\.warehouse\.driver:.*Invalid discriminator value/,
);
const connector = await createKtxCliScanConnector(project, 'warehouse');
expect(connector.id).toBe('duckdb:warehouse');
expect(connector.driver).toBe('duckdb');
});
it('rejects connection blocks with no driver field at config parse time', async () => {

View file

@ -245,6 +245,7 @@ describe('setup databases step', () => {
{ value: 'sqlserver', label: 'SQL Server' },
{ value: 'mongodb', label: 'MongoDB' },
{ value: 'sqlite', label: 'SQLite' },
{ value: 'duckdb', label: 'DuckDB' },
],
required: true,
});
@ -3515,4 +3516,72 @@ describe('setup databases step', () => {
expect(io.stdout()).toContain('ktx cannot work until you add a database.');
expect(await readFile(join(tempDir, 'ktx.yaml'), 'utf-8')).not.toContain('completed_steps:');
});
it('adds one non-interactive DuckDB connection from --database-url without prompting', async () => {
const io = makeIo();
const prompts = makePromptAdapter({});
const testConnection = vi.fn(async () => 0);
const scanConnection = vi.fn(async () => 0);
const result = await runKtxSetupDatabasesStep(
{
projectDir: tempDir,
inputMode: 'disabled',
databaseDrivers: ['duckdb'],
databaseConnectionId: 'duckdb-local',
databaseUrl: './warehouse.duckdb',
databaseSchemas: [],
skipDatabases: false,
},
io.io,
{ prompts, testConnection, scanConnection },
);
expect(result.status).toBe('ready');
expect(prompts.text).not.toHaveBeenCalled();
expect(testConnection).toHaveBeenCalledWith(tempDir, 'duckdb-local', expect.anything());
expect(scanConnection).toHaveBeenCalledWith(tempDir, 'duckdb-local', expect.anything());
const config = parseKtxProjectConfig(await readFile(join(tempDir, 'ktx.yaml'), 'utf-8'));
expect(config.connections['duckdb-local']).toEqual({
driver: 'duckdb',
path: './warehouse.duckdb',
});
expect(config.setup).toEqual({
database_connection_ids: ['duckdb-local'],
});
expect((await readKtxSetupState(tempDir)).completed_steps).toContain('databases');
});
it('adds an interactive DuckDB connection without prompting for a schema', async () => {
const prompts = makePromptAdapter({
selectValues: ['no'],
textValues: ['', './warehouse.duckdb'],
});
const pickers = makePickerStubs();
const result = await runKtxSetupDatabasesStep(
{
projectDir: tempDir,
inputMode: 'auto',
databaseDrivers: ['duckdb'],
databaseSchemas: [],
skipDatabases: false,
},
makeIo().io,
{
prompts,
testConnection: vi.fn(async () => 0),
scanConnection: vi.fn(async () => 0),
pickDatabaseScope: pickers.pickDatabaseScope,
},
);
expect(result.status).toBe('ready');
expect(pickers.scopeCalls).toHaveLength(0);
const config = parseKtxProjectConfig(await readFile(join(tempDir, 'ktx.yaml'), 'utf-8'));
expect(config.connections['duckdb-local']).toEqual({
driver: 'duckdb',
path: './warehouse.duckdb',
});
});
});