mirror of
https://github.com/Kaelio/ktx.git
synced 2026-07-04 10:52:13 +02:00
* refactor(duckdb): extract shared json-safe bigint helper
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(duckdb): add and register the duckdb primary connector
Add KtxDuckDbDialect, KtxDuckDbScanConnector (local file-backed, read-only,
never-create, main-schema introspection via information_schema and
duckdb_constraints() for foreign keys), and register the duckdb driver across
the dialect factory, driver registry, connection-type enum, warehouse descriptor,
config schema, scan normalization, connection test drivers, and status display.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(duckdb): route live-database ingest through the DuckDB connector
Add the DuckDB live-database introspection bridge and dispatch duckdb
connections to it in local-adapters, matching the SQLite path. Repoint the
config-rejection test off duckdb (now a valid driver) onto the no-driver case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(duckdb): add duckdb to the setup database flow
Offer DuckDB in the interactive checklist and via ktx setup --database duckdb,
with a file-path prompt and duckdb-local default connection id, parallel to SQLite.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(duckdb): attach native duckdb files in federation
Native .duckdb members ATTACH with (READ_ONLY) and no TYPE/INSTALL/LOAD, since
the duckdb format needs no extension. attachTypeForDriver returns null for the
native case; buildAttachStatements builds load statements from non-null types
only and emits a conditional ATTACH clause.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(duckdb): document the duckdb primary-source connector
Add a DuckDB section to the primary-sources integration page (config, read-only
never-create behavior, main-schema scope, federation) and update the
supported-driver assertion in dialects.test.ts to include duckdb.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(duckdb): use single-namespace display shape for main-only refs
DuckDB v1 introspects the main schema and sets db=null on every table, so its
display refs are single-namespace like SQLite. The ansi shape emitted a 1-part
table display it then refused to parse, breaking column-level display resolution.
Switch the dialect to the sqlite display shape and add a round-trip test plus a
composite-foreign-key test that were missing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(duckdb): resolve connector dialect via getDialectForDriver
Route the connector's dialect through the shared factory like every other
connector, now that duckdb is registered. Single construction path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(duckdb): skip schema picker for single-file duckdb setup
DuckDB is a single-file, single-namespace ('main') database like SQLite,
but the setup scope step only skipped the schema picker for sqlite. DuckDB
fell into the multi-schema path with an empty schema list, rendering a
broken picker ("No matches found" for main). Extend the file-based-driver
early-return to cover duckdb so it ingests every table directly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(duckdb): reuse shared config helper and derive scope skip
Route duckdb path resolution through the shared resolveStringReference
helper instead of a local third copy of env:/file: handling. Derive the
setup scope-picker skip from SCOPE_DISCOVERY_SPECS membership rather than
a hardcoded sqlite/duckdb driver list.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(duckdb): use a genuinely unknown driver in the rejection test
The merged "rejects unknown drivers" test used `driver: duckdb` as its
unknown-driver stand-in, which stopped being unknown once this branch
added the duckdb connector. Switch to `nonsense` so it again exercises
the unsupported-driver config error.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(duckdb): cover dialect, connector, and live-introspection branches
Codecov flagged uncovered branches as dead code; all are real connector,
dialect, and live-ingest behavior. Add unit tests instead of removing them.
- dialect: precedence ladder, sample/clause builders, profiling expressions
- connector: url/env config forms, error throws, never-create guard,
cardinality cap branches, table-scope empty/non-empty paths
- live-introspection: full-schema and table-scope extraction
Functions 100%, lines ~99% across the duckdb connector dir.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: add DuckDB to supported-driver references
The DuckDB connector PR documented the connector itself but left the
scattered supported-driver enumerations stale. Add duckdb to the
federation concept page (participation table, activation, table naming,
limitations), the ktx setup CLI reference, the ktx.yaml warehouse-driver
table, the primary-sources field reference, and the quickstart driver
list (which also restores the missing ClickHouse entry).
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Andrey Avtomonov <andreybavt@gmail.com>
134 lines
4 KiB
TypeScript
134 lines
4 KiB
TypeScript
import { mkdtemp, rm, writeFile } from 'node:fs/promises';
|
|
import { tmpdir } from 'node:os';
|
|
import { join } from 'node:path';
|
|
import { initKtxProject, loadKtxProject } from '../src/context/project/project.js';
|
|
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
|
import { createKtxCliScanConnector } from '../src/local-scan-connectors.js';
|
|
|
|
const bigQueryMock = vi.hoisted(() => ({
|
|
constructorInputs: [] as Array<{
|
|
connectionId: string;
|
|
connection: unknown;
|
|
}>,
|
|
}));
|
|
|
|
vi.mock('../src/connectors/bigquery/connector.js', () => ({
|
|
isKtxBigQueryConnectionConfig: (connection: { driver?: unknown } | undefined) =>
|
|
String(connection?.driver ?? '').toLowerCase() === 'bigquery',
|
|
KtxBigQueryScanConnector: class {
|
|
readonly id: string;
|
|
readonly driver = 'bigquery';
|
|
|
|
constructor(options: { connectionId: string; connection: unknown }) {
|
|
bigQueryMock.constructorInputs.push(options);
|
|
this.id = `bigquery:${options.connectionId}`;
|
|
}
|
|
},
|
|
}));
|
|
|
|
describe('createKtxCliScanConnector', () => {
|
|
let tempDir: string;
|
|
|
|
beforeEach(async () => {
|
|
bigQueryMock.constructorInputs.length = 0;
|
|
tempDir = await mkdtemp(join(tmpdir(), 'ktx-cli-scan-connector-'));
|
|
});
|
|
|
|
afterEach(async () => {
|
|
await rm(tempDir, { recursive: true, force: true });
|
|
});
|
|
|
|
it('creates a native sqlite connector from standalone config', async () => {
|
|
await initKtxProject({ projectDir: tempDir });
|
|
await writeFile(
|
|
join(tempDir, 'ktx.yaml'),
|
|
[
|
|
'connections:',
|
|
' warehouse:',
|
|
' driver: sqlite',
|
|
' path: warehouse.db',
|
|
'',
|
|
].join('\n'),
|
|
'utf-8',
|
|
);
|
|
const project = await loadKtxProject({ projectDir: tempDir });
|
|
|
|
const connector = await createKtxCliScanConnector(project, 'warehouse');
|
|
|
|
expect(connector.id).toBe('sqlite:warehouse');
|
|
expect(connector.driver).toBe('sqlite');
|
|
});
|
|
|
|
it('passes canonical BigQuery YAML scan limits through to the connector', async () => {
|
|
await initKtxProject({ projectDir: tempDir });
|
|
await writeFile(
|
|
join(tempDir, 'ktx.yaml'),
|
|
[
|
|
'connections:',
|
|
' warehouse:',
|
|
' driver: bigquery',
|
|
' dataset_id: analytics',
|
|
' max_bytes_billed: "987654321"',
|
|
' query_timeout_ms: 30000',
|
|
'',
|
|
].join('\n'),
|
|
'utf-8',
|
|
);
|
|
const project = await loadKtxProject({ projectDir: tempDir });
|
|
|
|
const connector = await createKtxCliScanConnector(project, 'warehouse');
|
|
|
|
expect(connector.id).toBe('bigquery:warehouse');
|
|
expect(connector.driver).toBe('bigquery');
|
|
expect(bigQueryMock.constructorInputs).toEqual([
|
|
expect.objectContaining({
|
|
connectionId: 'warehouse',
|
|
connection: expect.objectContaining({
|
|
max_bytes_billed: '987654321',
|
|
query_timeout_ms: 30000,
|
|
}),
|
|
}),
|
|
]);
|
|
expect(bigQueryMock.constructorInputs[0]).not.toHaveProperty('maxBytesBilled');
|
|
});
|
|
|
|
it('resolves a duckdb connection to the DuckDB scan connector', async () => {
|
|
await initKtxProject({ projectDir: tempDir });
|
|
await writeFile(
|
|
join(tempDir, 'ktx.yaml'),
|
|
[
|
|
'connections:',
|
|
' warehouse:',
|
|
' driver: duckdb',
|
|
' path: warehouse.duckdb',
|
|
'',
|
|
].join('\n'),
|
|
'utf-8',
|
|
);
|
|
const project = await loadKtxProject({ projectDir: tempDir });
|
|
|
|
const connector = await createKtxCliScanConnector(project, 'warehouse');
|
|
|
|
expect(connector.id).toBe('duckdb:warehouse');
|
|
expect(connector.driver).toBe('duckdb');
|
|
});
|
|
|
|
it('rejects connection blocks with no driver field at config parse time', async () => {
|
|
await initKtxProject({ projectDir: tempDir });
|
|
await writeFile(
|
|
join(tempDir, 'ktx.yaml'),
|
|
[
|
|
'connections:',
|
|
' warehouse:',
|
|
' type: postgres',
|
|
' url: postgresql://example/db',
|
|
'',
|
|
].join('\n'),
|
|
'utf-8',
|
|
);
|
|
|
|
await expect(loadKtxProject({ projectDir: tempDir })).rejects.toThrow(
|
|
/connections\.warehouse\.driver:.*Invalid discriminator value/,
|
|
);
|
|
});
|
|
});
|