feat(connectors): add MongoDB connector (#305) (#310)

* refactor(connectors): split KtxDialect into core and KtxSqlDialect

Separate the dialect contract into a driver-agnostic core (display/ref
formatting and type mapping) and a SQL-only extension (query generators).
The catalog and entity-details paths resolve the core dialect for any
snapshot driver, so it must stay free of SQL generation; this is the
prerequisite refactor for adding non-SQL primary sources.

- KtxDialect keeps type, formatDisplayRef, parseDisplayRef,
  columnDisplayTablePartCount, mapDataType, mapToDimensionType
- KtxSqlDialect extends it with quoteIdentifier, formatTableName, and the
  query/sample/statistics generators; the 7 SQL dialects implement it
- add getSqlDialectForDriver for SQL drivers; the 7 connectors and the
  relationship-benchmark harness consume it
- thread the relationship pipeline (profiling/validation/composite/
  discovery) as KtxSqlDialect | null so a non-SQL source skips coverage SQL
  and its candidates stay in review; local-enrichment builds the SQL
  dialect only when the connector advertises readOnlySql

Pure extraction: no behavior change for the existing 7 drivers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(connectors): add MongoDB connector for issue #305

Add a read-only MongoDB connector that treats a database as a primary
context source: collections map to tables and inferred top-level fields to
columns. MongoDB is the first non-SQL source (readOnlySql: false), so
ktx sql and metric compilation do not apply, but its collections flow
through ingest, descriptions, and relationship discovery.

- schema-inference: infer a flat column schema from the most recent
  sample_size documents (by _id desc, or order_by for non-ObjectId keys).
  Union BSON types per field, mark multi-type fields mixed (string), keep
  sub-documents/arrays as a single opaque json column, derive nullability
  from presence, treat _id as the primary key
- connector: KtxMongoDbScanConnector behind an injectable client seam;
  strictly read-only (find/listCollections/estimatedDocumentCount only),
  no executeReadOnly; resolves env:/file: via resolveKtxConfigReference
- core-only KtxMongoDbDialect and a live-database introspection adapter
- wire the mongodb driver: driver union, dialect registry, driver
  registration (scopeConfigKey databases), mongodbConnectionSchema,
  connection-drivers, normalizeDriver, the live-database route, and the
  ktx setup picker. ktx sql is refused by the read-only SQL capability gate
- tests: schema inference, connector snapshot via a fake client, dialect,
  driver-schema parsing, and the ktx sql rejection

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(integrations): document the MongoDB primary source

Add a MongoDB section to the primary-sources reference: connection config
(url, databases, enabled_tables, sample_size, order_by), mongodb+srv/TLS/
Atlas notes, the schema-inference explainer, a features matrix, and the
non-SQL caveat. Update the frontmatter and connection field reference.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(connectors): address review blockers on the MongoDB connector

- introspect: skip estimatedDocumentCount for views. The count command is
  rejected on a MongoDB view (CommandNotSupportedOnView), so counting a view
  aborted introspect for the whole connection; compute estimatedRows only for
  real collections, as ClickHouse does.
- sl: refuse a semantic-layer query against a non-SQL connection instead of
  defaulting it to the Postgres dialect. compileLocalSlQuery (the shared CLI +
  MCP path) now rejects a driver with no SQL dialect via the new
  isSqlQueryableDriver authority, keeping MongoDB context-only per issue #305.
- tests: cover input.tableScope and the empty-scope skip for the Mongo
  connector (the scan layer does not post-filter), the view no-count path, and
  the ktx sl query refusal for a mongodb connection.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* polish(mongodb): compute sampled nullCount and document sampling caveats

Address the non-blocking review notes:

- sampleColumn now counts null/absent values over the sampled window instead of
  returning nullCount: null, since the documents are already in hand
- warn that a custom order_by must be indexed (an unindexed sort hits MongoDB's
  in-memory sort limit on large collections) in the connection schema and docs
- note that sampled values for nested fields are stringified, not faithfully
  serialized, so the json opacity is deliberate

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(examples): add a MongoDB connector example

A manual, container-backed example mirroring examples/postgres-historic:

- docker-compose.yml + init/seed.js seed a representative dataset (nested
  documents, arrays, a Decimal128, a mixed-type field, a nullable field, an
  ObjectId reference, and a view) on first container start
- scripts/smoke.sh + introspect-smoke.mjs assert the connector's inferred
  schema with no LLM credentials — the same introspection entry point ktx
  ingest's database-schema stage uses, including the view-no-count path
- README.md documents the smoke and a full keyless ktx ingest run
  (claude-code LLM + managed sentence-transformers embeddings)

Works with Docker Compose or podman compose. Verified end to end.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: ignore examples/** in knip to fix dead-code false positives

The MongoDB connector example files (examples/mongodb/init/seed.js and
examples/mongodb/scripts/introspect-smoke.mjs) are used at runtime but were
flagged as unused by knip. Add examples/** to the ignore array, matching the
existing .context/** entry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0114qQV8fJ5a5ME3XbMVRzbL

* fix(mongodb): refuse non-SQL connections before SQL analysis

`ktx sql` and the MCP sql_execution tool resolved a SQL-analysis dialect
(falling back to Postgres for a non-SQL driver) and ran read-only
validation before the connector capability gate refused the connection.
For a MongoDB connection that spun up the parser/daemon and produced
Postgres parser diagnostics instead of a clean non-SQL refusal.

Route both entry points through a shared assertSqlQueryableConnection
guard before dialect selection, mirroring compileLocalSlQuery. The
federated duckdb path has no driver and is exempted at each call site.
Add CLI and MCP regression tests asserting validation/connector work
never starts for a MongoDB connection.

* fix(mongodb): pass CI gates (dialect boundary, secrets, setup test)

Three latent failures in the connector surfaced once CI ran on the branch:

- connector.ts imported the concrete KtxMongoDbDialect, which the connector
  dialect-import boundary forbids. Route it through getDialectForDriver('mongodb')
  and widen inferKtxMongoCollectionColumns to the base KtxDialect (it only uses
  mapDataType/mapToDimensionType).
- detect-secrets flagged a test ObjectId hex and the mongodb+srv example URL;
  annotate both with allowlist pragmas.
- the "shows every supported database" setup test omitted the new MongoDB option.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Luca Martial <48870843+luca-martial@users.noreply.github.com>
Co-authored-by: Luca Martial <lucamrtl@gmail.com>
Co-authored-by: Andrey Avtomonov <andreybavt@gmail.com>
This commit is contained in:
Pintouch 2026-06-29 15:17:56 +02:00 committed by GitHub
parent 4f084186f1
commit 2afab61417
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
59 changed files with 1971 additions and 129 deletions

View file

@ -0,0 +1,241 @@
import { describe, expect, it, vi } from 'vitest';
import {
isKtxMongoDbConnectionConfig,
KtxMongoDbScanConnector,
type KtxMongoClient,
type KtxMongoClientFactory,
type KtxMongoDbConnectionConfig,
type KtxMongoListedCollection,
} from '../../../src/connectors/mongodb/connector.js';
import { createMongoDbLiveDatabaseIntrospection } from '../../../src/connectors/mongodb/live-database-introspection.js';
import { executeProjectReadOnlySql } from '../../../src/context/connections/project-sql-executor.js';
import { tableRefSet } from '../../../src/context/scan/table-ref.js';
import type { KtxLocalProject } from '../../../src/context/project/project.js';
import type { KtxMongoDocument } from '../../../src/connectors/mongodb/schema-inference.js';
function objectId(hex: string): unknown {
return { _bsontype: 'ObjectId', toString: () => hex };
}
const COLLECTIONS: Record<string, KtxMongoListedCollection[]> = {
app: [
{ name: 'users' },
{ name: 'orders' },
{ name: 'system.views' },
],
};
const DOCUMENTS: Record<string, KtxMongoDocument[]> = {
users: [
{ _id: objectId('a1'), email: 'a@x.com', age: 31, address: { city: 'NY' } },
{ _id: objectId('a2'), email: 'b@x.com' },
],
orders: [{ _id: objectId('b1'), total: 9.99 }],
};
function fakeClientFactory(): { factory: KtxMongoClientFactory; client: KtxMongoClient } {
const client: KtxMongoClient = {
listCollections: vi.fn(async (databaseName: string) => COLLECTIONS[databaseName] ?? []),
estimatedDocumentCount: vi.fn(async (_databaseName: string, collectionName: string) =>
collectionName === 'users' ? 2 : 1,
),
find: vi.fn(async (_databaseName: string, collectionName: string) => DOCUMENTS[collectionName] ?? []),
ping: vi.fn(async () => undefined),
close: vi.fn(async () => undefined),
};
return { factory: { create: vi.fn(() => client) }, client };
}
function connector(connection: KtxMongoDbConnectionConfig, factory: KtxMongoClientFactory): KtxMongoDbScanConnector {
return new KtxMongoDbScanConnector({ connectionId: 'mongo-prod', connection, clientFactory: factory });
}
const baseConnection: KtxMongoDbConnectionConfig = {
driver: 'mongodb',
url: 'mongodb://localhost:27017/app',
databases: ['app'],
};
describe('isKtxMongoDbConnectionConfig', () => {
it('matches only the mongodb driver', () => {
expect(isKtxMongoDbConnectionConfig({ driver: 'mongodb' })).toBe(true);
expect(isKtxMongoDbConnectionConfig({ driver: 'postgres' })).toBe(false);
expect(isKtxMongoDbConnectionConfig(undefined)).toBe(false);
});
});
describe('KtxMongoDbScanConnector capabilities', () => {
it('advertises a non-SQL, sampling-capable source and exposes no SQL execution', () => {
const { factory } = fakeClientFactory();
const c = connector(baseConnection, factory);
expect(c.driver).toBe('mongodb');
expect(c.capabilities.readOnlySql).toBe(false);
expect(c.capabilities.formalForeignKeys).toBe(false);
expect(c.capabilities.tableSampling).toBe(true);
expect(c.capabilities.columnSampling).toBe(true);
expect(c.capabilities.nestedAnalysis).toBe(true);
expect('executeReadOnly' in c).toBe(false);
});
});
describe('KtxMongoDbScanConnector construction', () => {
it('requires a url', () => {
const { factory } = fakeClientFactory();
expect(() => connector({ driver: 'mongodb', databases: ['app'] }, factory)).toThrow(/requires connections\.mongo-prod\.url/);
});
it('resolves env: url references and derives the database from the url path', () => {
const { factory, client } = fakeClientFactory();
const c = new KtxMongoDbScanConnector({
connectionId: 'mongo-prod',
connection: { driver: 'mongodb', url: 'env:MONGO_URL' },
clientFactory: factory,
env: { MONGO_URL: 'mongodb://localhost:27017/app' },
});
return c.introspect({ connectionId: 'mongo-prod', driver: 'mongodb' }, { runId: 't' }).then((snapshot) => {
expect(snapshot.scope.schemas).toEqual(['app']);
expect(client.listCollections).toHaveBeenCalledWith('app');
});
});
it('refuses a non-mongodb driver', () => {
const { factory } = fakeClientFactory();
expect(() => connector({ driver: 'postgres' } as KtxMongoDbConnectionConfig, factory)).toThrow(/cannot run driver "postgres"/);
});
});
describe('KtxMongoDbScanConnector.introspect', () => {
it('maps collections to tables, infers columns, and excludes system collections', async () => {
const { factory } = fakeClientFactory();
const snapshot = await connector(baseConnection, factory).introspect(
{ connectionId: 'mongo-prod', driver: 'mongodb' },
{ runId: 't' },
);
expect(snapshot.driver).toBe('mongodb');
expect(snapshot.tables.map((table) => table.name).sort()).toEqual(['orders', 'users']);
const users = snapshot.tables.find((table) => table.name === 'users')!;
expect(users.db).toBe('app');
expect(users.estimatedRows).toBe(2);
expect(users.foreignKeys).toEqual([]);
const idColumn = users.columns.find((column) => column.name === '_id')!;
expect(idColumn.primaryKey).toBe(true);
expect(idColumn.nullable).toBe(false);
const email = users.columns.find((column) => column.name === 'email')!;
expect(email.nullable).toBe(false); // present in every sampled document
const age = users.columns.find((column) => column.name === 'age')!;
expect(age.nullable).toBe(true); // missing from the second document
const address = users.columns.find((column) => column.name === 'address')!;
expect(address.normalizedType).toBe('json');
});
it('honors the enabled_tables allowlist', async () => {
const { factory } = fakeClientFactory();
const snapshot = await connector(
{ ...baseConnection, enabled_tables: ['app.users'] },
factory,
).introspect({ connectionId: 'mongo-prod', driver: 'mongodb' }, { runId: 't' });
expect(snapshot.tables.map((table) => table.name)).toEqual(['users']);
});
it('restricts introspection to input.tableScope (the scan layer does not post-filter)', async () => {
const { factory } = fakeClientFactory();
const snapshot = await connector(baseConnection, factory).introspect(
{
connectionId: 'mongo-prod',
driver: 'mongodb',
tableScope: tableRefSet([{ catalog: null, db: 'app', name: 'users' }]),
},
{ runId: 't' },
);
expect(snapshot.tables.map((table) => table.name)).toEqual(['users']);
});
it('yields zero tables for a database whose scoped set is empty', async () => {
const { factory } = fakeClientFactory();
const snapshot = await connector(baseConnection, factory).introspect(
{
connectionId: 'mongo-prod',
driver: 'mongodb',
tableScope: tableRefSet([{ catalog: null, db: 'other', name: 'users' }]),
},
{ runId: 't' },
);
expect(snapshot.tables).toEqual([]);
});
it('does not count documents on a view (estimatedDocumentCount fails on views)', async () => {
const collections: KtxMongoListedCollection[] = [
{ name: 'users' },
{ name: 'active_users', type: 'view' },
];
const client: KtxMongoClient = {
listCollections: vi.fn(async () => collections),
// Real MongoDB rejects a count command on a view with CommandNotSupportedOnView.
estimatedDocumentCount: vi.fn(async (_db: string, name: string) => {
if (name === 'active_users') {
throw new Error('CommandNotSupportedOnView: count is not supported on a view');
}
return 2;
}),
find: vi.fn(async (_db: string, name: string) => DOCUMENTS[name] ?? DOCUMENTS.users!),
ping: vi.fn(async () => undefined),
close: vi.fn(async () => undefined),
};
const snapshot = await connector(baseConnection, { create: vi.fn(() => client) }).introspect(
{ connectionId: 'mongo-prod', driver: 'mongodb' },
{ runId: 't' },
);
const view = snapshot.tables.find((table) => table.name === 'active_users')!;
expect(view.kind).toBe('view');
expect(view.estimatedRows).toBeNull();
expect(client.estimatedDocumentCount).not.toHaveBeenCalledWith('app', 'active_users');
});
});
describe('KtxMongoDbScanConnector sampling', () => {
it('samples a column, flattening nested values and counting nulls over the window', async () => {
const { factory } = fakeClientFactory();
const result = await connector(baseConnection, factory).sampleColumn(
{ connectionId: 'mongo-prod', table: { catalog: null, db: 'app', name: 'users' }, column: 'address', limit: 10 },
{ runId: 't' },
);
expect(result.values).toEqual([JSON.stringify({ city: 'NY' })]);
// address is present in the first sampled document and absent from the second
expect(result.nullCount).toBe(1);
});
});
describe('createMongoDbLiveDatabaseIntrospection', () => {
it('extracts a schema through the live-database port', async () => {
const { factory } = fakeClientFactory();
const port = createMongoDbLiveDatabaseIntrospection({
connections: {
'mongo-prod': { driver: 'mongodb', url: 'mongodb://localhost:27017/app', databases: ['app'] },
},
clientFactory: factory,
});
const snapshot = await port.extractSchema('mongo-prod');
expect(snapshot.driver).toBe('mongodb');
expect(snapshot.tables.length).toBe(2);
});
});
describe('ktx sql against a MongoDB connection', () => {
it('is rejected by the read-only SQL capability gate', async () => {
const { factory } = fakeClientFactory();
const project = { config: { connections: {} }, projectDir: '/tmp' } as unknown as KtxLocalProject;
await expect(
executeProjectReadOnlySql({
project,
input: { connectionId: 'mongo-prod', connection: undefined, sql: 'SELECT 1', maxRows: 1 },
createConnector: () => connector(baseConnection, factory),
}),
).rejects.toThrow(/does not support read-only SQL execution/);
});
});

View file

@ -0,0 +1,33 @@
import { describe, expect, it } from 'vitest';
import { KtxMongoDbDialect } from '../../../src/connectors/mongodb/dialect.js';
import { getDialectForDriver, getSqlDialectForDriver } from '../../../src/context/connections/dialects.js';
const dialect = new KtxMongoDbDialect();
describe('KtxMongoDbDialect', () => {
it('formats and parses db.collection display refs', () => {
expect(dialect.formatDisplayRef({ catalog: null, db: 'app', name: 'users' })).toBe('app.users');
expect(dialect.parseDisplayRef('app.users')).toEqual({ catalog: null, db: 'app', name: 'users' });
expect(dialect.columnDisplayTablePartCount()).toBe(2);
});
it('maps nested types to opaque json and scalars to dimension types', () => {
expect(dialect.mapDataType('object')).toBe('json');
expect(dialect.mapDataType('array')).toBe('json');
expect(dialect.mapDataType('objectId')).toBe('objectid');
expect(dialect.mapToDimensionType('date')).toBe('time');
expect(dialect.mapToDimensionType('long')).toBe('number');
expect(dialect.mapToDimensionType('mixed')).toBe('string');
expect(dialect.mapToDimensionType('object')).toBe('string');
});
});
describe('dialect registry', () => {
it('resolves a core dialect for mongodb', () => {
expect(getDialectForDriver('mongodb').type).toBe('mongodb');
});
it('refuses a SQL dialect for mongodb', () => {
expect(() => getSqlDialectForDriver('mongodb')).toThrow(/no SQL dialect/);
});
});

View file

@ -0,0 +1,103 @@
import { describe, expect, it } from 'vitest';
import { KtxMongoDbDialect } from '../../../src/connectors/mongodb/dialect.js';
import {
bsonTypeOf,
inferKtxMongoCollectionColumns,
type KtxMongoDocument,
} from '../../../src/connectors/mongodb/schema-inference.js';
const dialect = new KtxMongoDbDialect();
function objectId(): unknown {
return { _bsontype: 'ObjectId', toString: () => '64b7f0c2a1b2c3d4e5f60718' }; // pragma: allowlist secret
}
function decimal128(value: string): unknown {
return { _bsontype: 'Decimal128', toString: () => value };
}
function infer(documents: KtxMongoDocument[]) {
const columns = inferKtxMongoCollectionColumns(documents, dialect);
return new Map(columns.map((column) => [column.name, column]));
}
describe('bsonTypeOf', () => {
it('maps JS and BSON runtime values to canonical type names', () => {
expect(bsonTypeOf(objectId())).toBe('objectId');
expect(bsonTypeOf('hi')).toBe('string');
expect(bsonTypeOf(7)).toBe('int');
expect(bsonTypeOf(7.5)).toBe('double');
expect(bsonTypeOf(true)).toBe('bool');
expect(bsonTypeOf(new Date())).toBe('date');
expect(bsonTypeOf(decimal128('1.5'))).toBe('decimal');
expect(bsonTypeOf({ city: 'NY' })).toBe('object');
expect(bsonTypeOf([1, 2])).toBe('array');
expect(bsonTypeOf(null)).toBe('null');
});
});
describe('inferKtxMongoCollectionColumns', () => {
it('treats _id as the non-nullable primary key', () => {
const columns = infer([{ _id: objectId(), name: 'a' }]);
const id = columns.get('_id')!;
expect(id.primaryKey).toBe(true);
expect(id.nullable).toBe(false);
expect(id.dimensionType).toBe('string');
expect(id.normalizedType).toBe('objectid');
});
it('derives nullability from field presence and observed nulls', () => {
const columns = infer([
{ _id: objectId(), email: 'a@x.com', deleted_at: null },
{ _id: objectId(), email: 'b@x.com' },
]);
// present in every document, never null -> not nullable
expect(columns.get('email')!.nullable).toBe(false);
// missing in one document and null in another -> nullable
expect(columns.get('deleted_at')!.nullable).toBe(true);
});
it('maps scalar BSON types to dimension types', () => {
const columns = infer([
{ _id: objectId(), age: 30, score: 9.5, active: true, created: new Date(), balance: decimal128('10.00') },
]);
expect(columns.get('age')!.dimensionType).toBe('number');
expect(columns.get('score')!.dimensionType).toBe('number');
expect(columns.get('active')!.dimensionType).toBe('boolean');
expect(columns.get('created')!.dimensionType).toBe('time');
expect(columns.get('balance')!.dimensionType).toBe('number');
});
it('marks a field seen with more than one type as mixed and treats it as a string', () => {
const columns = infer([
{ _id: objectId(), ref: 'abc' },
{ _id: objectId(), ref: 123 },
]);
const ref = columns.get('ref')!;
expect(ref.nativeType).toBe('mixed');
expect(ref.normalizedType).toBe('mixed');
expect(ref.dimensionType).toBe('string');
});
it('keeps sub-documents and arrays as a single opaque json column', () => {
const columns = infer([
{ _id: objectId(), address: { city: 'NY', zip: '10001' }, tags: ['a', 'b'] },
]);
const address = columns.get('address')!;
expect(address.nativeType).toBe('object');
expect(address.normalizedType).toBe('json');
expect(address.dimensionType).toBe('string');
const tags = columns.get('tags')!;
expect(tags.nativeType).toBe('array');
expect(tags.normalizedType).toBe('json');
});
it('preserves first-seen field order', () => {
const columns = inferKtxMongoCollectionColumns(
[{ _id: objectId(), b: 1, a: 2 }],
dialect,
);
expect(columns.map((column) => column.name)).toEqual(['_id', 'b', 'a']);
});
});

View file

@ -1,5 +1,5 @@
import { describe, expect, it } from 'vitest';
import { getDialectForDriver } from '../../../src/context/connections/dialects.js';
import { getDialectForDriver, getSqlDialectForDriver } from '../../../src/context/connections/dialects.js';
import type { KtxConnectionDriver, KtxTableRef } from '../../../src/context/scan/types.js';
interface DialectFixture {
@ -248,8 +248,8 @@ const fixtures: DialectFixture[] = [
];
describe('getDialectForDriver', () => {
it.each(fixtures)('returns a full KtxDialect for $driver', (fixture) => {
const dialect = getDialectForDriver(fixture.driver);
it.each(fixtures)('returns a full KtxSqlDialect for $driver', (fixture) => {
const dialect = getSqlDialectForDriver(fixture.driver);
const column = dialect.quoteIdentifier('status');
expect(dialect.type).toBe(fixture.driver);
@ -305,12 +305,12 @@ describe('getDialectForDriver', () => {
it('throws with a supported-driver list for unknown drivers', () => {
expect(() => getDialectForDriver('oracle')).toThrow(
'Unsupported warehouse driver "oracle". Supported drivers: bigquery, clickhouse, mysql, postgres, sqlite, snowflake, sqlserver',
'Unsupported driver "oracle". Supported drivers: bigquery, clickhouse, mongodb, mysql, postgres, snowflake, sqlite, sqlserver',
);
});
it('rejects legacy driver aliases', () => {
expect(() => getDialectForDriver('postgresql')).toThrow('Unsupported warehouse driver "postgresql"');
expect(() => getDialectForDriver('sqlite3')).toThrow('Unsupported warehouse driver "sqlite3"');
expect(() => getDialectForDriver('postgresql')).toThrow('Unsupported driver "postgresql"');
expect(() => getDialectForDriver('sqlite3')).toThrow('Unsupported driver "sqlite3"');
});
});

View file

@ -22,6 +22,11 @@ const connectionFixtures: Record<KtxConnectionDriver, FixtureFactory> = {
schemas: ['public'],
}),
sqlite: () => ({ driver: 'sqlite', path: 'warehouse.db' }),
mongodb: () => ({
driver: 'mongodb',
url: 'mongodb://localhost:27017/app',
databases: ['app'],
}),
mysql: () => ({
driver: 'mysql',
host: 'localhost',
@ -96,6 +101,7 @@ describe('driverRegistrations', () => {
expect(listSupportedDrivers()).toEqual([
'bigquery',
'clickhouse',
'mongodb',
'mysql',
'postgres',
'snowflake',

View file

@ -282,6 +282,33 @@ describe('createLocalProjectMcpContextPorts', () => {
expect(createConnector).not.toHaveBeenCalled();
});
it('refuses sql_execution against a non-SQL (MongoDB) connection before SQL analysis', async () => {
const project = await initKtxProject({ projectDir: tempDir });
project.config.connections.mongo = { driver: 'mongodb', url: 'mongodb://localhost:27017/app' };
const createConnector = vi.fn(async () => testConnector());
const sqlAnalysis = {
analyzeForFingerprint: vi.fn(),
analyzeBatch: vi.fn(),
validateReadOnly: vi.fn(async () => ({ ok: true, error: null })),
};
const ports = createLocalProjectMcpContextPorts(project, {
sqlAnalysis,
localScan: { createConnector },
embeddingService: null,
});
const execution = ports.sqlExecution?.execute({
connectionId: 'mongo',
sql: 'select 1',
maxRows: 5,
});
await expect(execution).rejects.toBeInstanceOf(KtxExpectedError);
await expect(execution).rejects.toThrow("non-SQL driver 'mongodb'");
// Refused before the parser dialect is chosen and before any connector is built.
expect(sqlAnalysis.validateReadOnly).not.toHaveBeenCalled();
expect(createConnector).not.toHaveBeenCalled();
});
it('emits sql_execution progress stages from local MCP ports', async () => {
const project = await initKtxProject({ projectDir: tempDir });
project.config.connections.warehouse = {

View file

@ -28,6 +28,28 @@ describe('connectionConfigSchema (driver discriminated union)', () => {
});
});
it('parses a mongodb connection with sampling fields', () => {
const parsed = connectionConfigSchema.parse({
driver: 'mongodb',
url: 'env:MONGO_URL',
databases: ['app'],
enabled_tables: ['app.users'],
sample_size: 500,
order_by: 'createdAt',
});
expect(parsed).toMatchObject({
driver: 'mongodb',
url: 'env:MONGO_URL',
databases: ['app'],
sample_size: 500,
order_by: 'createdAt',
});
});
it('rejects a mongodb connection without a url', () => {
expect(() => connectionConfigSchema.parse({ driver: 'mongodb', databases: ['app'] })).toThrow();
});
it('rejects an unknown driver', () => {
expect(() => connectionConfigSchema.parse({ driver: 'nope', url: 'x' })).toThrow();
});

View file

@ -1,7 +1,7 @@
import Database from 'better-sqlite3';
import { join } from 'node:path';
import { describe, expect, it } from 'vitest';
import { getDialectForDriver } from '../../../src/context/connections/dialects.js';
import { getSqlDialectForDriver } from '../../../src/context/connections/dialects.js';
import { snapshotToKtxEnrichedSchema } from '../../../src/context/scan/local-enrichment.js';
import { loadKtxRelationshipBenchmarkFixture, maskKtxRelationshipBenchmarkSnapshot } from '../../../src/context/scan/relationship-benchmarks.js';
import { discoverKtxCompositeRelationships } from '../../../src/context/scan/relationship-composite-candidates.js';
@ -42,7 +42,8 @@ describe('composite relationship discovery detector', () => {
const executor = new TestSqliteExecutor(fixture.dataPath ?? '');
const profiles = await profileKtxRelationshipSchema({
connectionId: snapshot.connectionId,
dialect: getDialectForDriver(snapshot.driver),
driver: snapshot.driver,
dialect: getSqlDialectForDriver(snapshot.driver),
schema,
executor,
ctx: { runId: 'test:composite-profile' },
@ -50,7 +51,7 @@ describe('composite relationship discovery detector', () => {
const result = await discoverKtxCompositeRelationships({
connectionId: snapshot.connectionId,
dialect: getDialectForDriver(snapshot.driver),
dialect: getSqlDialectForDriver(snapshot.driver),
schema,
profiles,
executor,

View file

@ -1,7 +1,7 @@
import Database from 'better-sqlite3';
import { afterEach, describe, expect, it, vi } from 'vitest';
import type { KtxLlmRuntimePort } from '../../../src/context/llm/runtime-port.js';
import { getDialectForDriver } from '../../../src/context/connections/dialects.js';
import { getSqlDialectForDriver } from '../../../src/context/connections/dialects.js';
import { buildDefaultKtxProjectConfig } from '../../../src/context/project/config.js';
import { snapshotToKtxEnrichedSchema } from '../../../src/context/scan/local-enrichment.js';
import {
@ -311,7 +311,7 @@ describe('production relationship discovery', () => {
const result = await discoverKtxRelationships({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
connector: connector(executor),
schema: snapshotToKtxEnrichedSchema(snapshot()),
context: { runId: 'relationship-run-1' },
@ -350,7 +350,7 @@ describe('production relationship discovery', () => {
const schema = naturalKeySnapshot();
const result = await discoverKtxRelationships({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
connector: {
...connector(executor),
introspect: async () => schema,
@ -400,7 +400,7 @@ describe('production relationship discovery', () => {
const result = await discoverKtxRelationships({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
connector: {
...connector(executor),
introspect: async () => sourceSnapshot,
@ -433,7 +433,7 @@ describe('production relationship discovery', () => {
it('keeps candidates review-only when read-only SQL is unavailable', async () => {
const result = await discoverKtxRelationships({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
connector: connector(null),
schema: snapshotToKtxEnrichedSchema(snapshot()),
context: { runId: 'relationship-run-no-sql' },
@ -459,7 +459,7 @@ describe('production relationship discovery', () => {
const sourceSnapshot = declaredForeignKeySnapshot();
const result = await discoverKtxRelationships({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
connector: connector(null),
schema: snapshotToKtxEnrichedSchema(sourceSnapshot),
context: { runId: 'formal-metadata-no-sql' },
@ -506,7 +506,7 @@ describe('production relationship discovery', () => {
const result = await discoverKtxRelationships({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
connector: connector(executor),
schema: snapshotToKtxEnrichedSchema(llmOnlyRelationshipSnapshot()),
context: { runId: 'llm-relationship-orchestrator' },
@ -546,7 +546,7 @@ describe('production relationship discovery', () => {
const result = await discoverKtxRelationships({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
connector: connector(executor),
schema: snapshotToKtxEnrichedSchema(snapshot()),
context: { runId: 'configured-thresholds' },
@ -607,7 +607,7 @@ describe('production relationship discovery', () => {
const result = await discoverKtxRelationships({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
connector: {
...connector(executor),
introspect: async () => richSnapshot,
@ -663,7 +663,7 @@ describe('production relationship discovery', () => {
const result = await discoverKtxRelationships({
connectionId: maskedSnapshot.connectionId,
dialect: getDialectForDriver(maskedSnapshot.driver),
dialect: getSqlDialectForDriver(maskedSnapshot.driver),
connector: testConnector,
schema: snapshotToKtxEnrichedSchema(maskedSnapshot, new Map()),
context: { runId: 'test:production-composite' },

View file

@ -2,7 +2,7 @@ import { readFile } from 'node:fs/promises';
import { join } from 'node:path';
import Database from 'better-sqlite3';
import { afterEach, describe, expect, it, vi } from 'vitest';
import { getDialectForDriver } from '../../../src/context/connections/dialects.js';
import { getSqlDialectForDriver } from '../../../src/context/connections/dialects.js';
import type { KtxEnrichedColumn, KtxEnrichedSchema, KtxEnrichedTable } from '../../../src/context/scan/enrichment-types.js';
import { snapshotToKtxEnrichedSchema } from '../../../src/context/scan/local-enrichment.js';
import { loadKtxRelationshipBenchmarkFixture, maskKtxRelationshipBenchmarkSnapshot } from '../../../src/context/scan/relationship-benchmarks.js';
@ -121,7 +121,8 @@ describe('relationship profiling', () => {
const result = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: schema([
table('accounts', [
column('accounts', 'id', { primaryKey: false, nullable: false }),
@ -183,7 +184,8 @@ describe('relationship profiling', () => {
const result = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: schema([
table('accounts', [
column('accounts', 'id', { nullable: false }),
@ -226,7 +228,8 @@ describe('relationship profiling', () => {
const profiles = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: schema([
table('accounts', [
column('accounts', 'id', { nullable: false }),
@ -277,7 +280,8 @@ describe('relationship profiling', () => {
const first = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: relationshipSchema,
executor,
ctx: { runId: 'profile-cache-run' },
@ -285,7 +289,8 @@ describe('relationship profiling', () => {
});
const second = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: relationshipSchema,
executor,
ctx: { runId: 'profile-cache-run' },
@ -293,7 +298,8 @@ describe('relationship profiling', () => {
});
const third = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: relationshipSchema,
executor,
ctx: { runId: 'profile-cache-fresh-run' },
@ -322,7 +328,8 @@ describe('relationship profiling', () => {
try {
const result = await profileKtxRelationshipSchema({
connectionId: fixture.snapshot.connectionId,
dialect: getDialectForDriver(fixture.snapshot.driver),
driver: fixture.snapshot.driver,
dialect: getSqlDialectForDriver(fixture.snapshot.driver),
schema: snapshotToKtxEnrichedSchema(maskedSnapshot, new Map()),
executor: scaleExecutor,
ctx: { runId: 'scale-stress-profile-query-count' },
@ -367,7 +374,8 @@ describe('relationship profiling', () => {
await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: schemaWithTables(['accounts', 'orders', 'payments', 'refunds']),
executor,
ctx: { runId: 'profile-concurrency' },
@ -403,7 +411,8 @@ describe('relationship profiling', () => {
const result = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: schemaWithTables(['accounts', 'orders']),
executor,
ctx: { runId: 'profile-error-isolated' },

View file

@ -1,6 +1,6 @@
import Database from 'better-sqlite3';
import { afterEach, describe, expect, it } from 'vitest';
import { getDialectForDriver } from '../../../src/context/connections/dialects.js';
import { getSqlDialectForDriver } from '../../../src/context/connections/dialects.js';
import type { KtxEnrichedColumn, KtxEnrichedSchema, KtxEnrichedTable } from '../../../src/context/scan/enrichment-types.js';
import { generateKtxRelationshipDiscoveryCandidates } from '../../../src/context/scan/relationship-candidates.js';
import type { KtxRelationshipProfileArtifact } from '../../../src/context/scan/relationship-profiling.js';
@ -102,7 +102,8 @@ describe('relationship validation', () => {
const testSchema = schema();
const profiles = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: testSchema,
executor,
ctx: { runId: 'validate-test' },
@ -113,7 +114,7 @@ describe('relationship validation', () => {
const validated = await validateKtxRelationshipDiscoveryCandidates({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
candidates,
profiles,
executor,
@ -151,7 +152,8 @@ describe('relationship validation', () => {
const testSchema = schema();
const profiles = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: testSchema,
executor,
ctx: { runId: 'validate-test' },
@ -162,7 +164,7 @@ describe('relationship validation', () => {
const validated = await validateKtxRelationshipDiscoveryCandidates({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
candidates,
profiles,
executor,
@ -201,7 +203,8 @@ describe('relationship validation', () => {
const testSchema = schema();
const profiles = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: testSchema,
executor,
ctx: { runId: 'validate-budget-profile' },
@ -214,7 +217,7 @@ describe('relationship validation', () => {
const validated = await validateKtxRelationshipDiscoveryCandidates({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
candidates,
profiles,
executor,
@ -256,7 +259,8 @@ describe('relationship validation', () => {
]);
const profiles = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: testSchema,
executor,
ctx: { runId: 'validate-zero-budget-profile' },
@ -266,7 +270,7 @@ describe('relationship validation', () => {
const validated = await validateKtxRelationshipDiscoveryCandidates({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
candidates,
profiles,
executor,
@ -303,7 +307,8 @@ describe('relationship validation', () => {
]);
const profiles = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: testSchema,
executor,
ctx: { runId: 'llm-rejected-validation' },
@ -332,7 +337,7 @@ describe('relationship validation', () => {
const [validated] = await validateKtxRelationshipDiscoveryCandidates({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
candidates: [llmCandidate],
profiles,
executor,
@ -377,7 +382,8 @@ describe('relationship validation', () => {
]);
const profiles = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: testSchema,
executor,
ctx: { runId: 'validation-concurrency-profile' },
@ -386,7 +392,7 @@ describe('relationship validation', () => {
await validateKtxRelationshipDiscoveryCandidates({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
candidates,
profiles,
executor: throttled,
@ -478,7 +484,7 @@ describe('relationship validation', () => {
const [validated] = await validateKtxRelationshipDiscoveryCandidates({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
candidates: [candidate],
profiles,
executor,

View file

@ -67,6 +67,18 @@ grain: []
await rm(tempDir, { recursive: true, force: true });
});
it('refuses a non-SQL (context-only) connection instead of compiling it as Postgres', async () => {
project.config.connections['mongo-prod'] = { driver: 'mongodb', url: 'mongodb://localhost:27017/app' };
await expect(
compileLocalSlQuery(project, {
connectionId: 'mongo-prod',
query: { measures: ['orders.order_count'], dimensions: ['orders.status'], limit: 25 },
compute,
}),
).rejects.toThrow(/non-SQL driver 'mongodb'|require a SQL warehouse connection/);
expect(compute.query).not.toHaveBeenCalled();
});
it('compiles a local semantic-layer query with computable sources only', async () => {
const result = await compileLocalSlQuery(project, {
connectionId: 'warehouse',

View file

@ -243,6 +243,7 @@ describe('setup databases step', () => {
{ value: 'mysql', label: 'MySQL' },
{ value: 'clickhouse', label: 'ClickHouse' },
{ value: 'sqlserver', label: 'SQL Server' },
{ value: 'mongodb', label: 'MongoDB' },
{ value: 'sqlite', label: 'SQLite' },
],
required: true,

View file

@ -348,6 +348,40 @@ describe('runKtxSql', () => {
expect(io.stderr()).toContain('does not support read-only SQL execution.');
});
it('refuses a non-SQL (MongoDB) connection before invoking SQL analysis', async () => {
const projectDir = join(tempDir, 'project');
await initKtxProject({ projectDir });
await writeConnections(projectDir, { mongo: { driver: 'mongodb', url: 'mongodb://localhost:27017/app' } });
const sqlAnalysis = makeSqlAnalysis({ ok: true, error: null });
const createSqlAnalysis = vi.fn(() => sqlAnalysis);
const createScanConnector = vi.fn(async () => makeConnector());
const io = makeIo();
await expect(
runKtxSql(
{
command: 'execute',
projectDir,
connectionId: 'mongo',
sql: 'select 1',
maxRows: 1000,
output: 'pretty',
json: false,
cliVersion: '0.0.0-test',
},
io.io,
{ createSqlAnalysis, createScanConnector },
),
).resolves.toBe(1);
// The non-SQL boundary is enforced before any SQL parser/daemon work, so a
// MongoDB connection never reaches dialect selection or read-only validation.
expect(createSqlAnalysis).not.toHaveBeenCalled();
expect(sqlAnalysis.validateReadOnly).not.toHaveBeenCalled();
expect(createScanConnector).not.toHaveBeenCalled();
expect(io.stderr()).toContain("non-SQL driver 'mongodb'");
});
it('routes _ktx_federated through the shared federated executor', async () => {
const projectDir = join(tempDir, 'project');
await initKtxProject({ projectDir });