feat(connectors): add MongoDB connector (#305) (#310)

* refactor(connectors): split KtxDialect into core and KtxSqlDialect

Separate the dialect contract into a driver-agnostic core (display/ref
formatting and type mapping) and a SQL-only extension (query generators).
The catalog and entity-details paths resolve the core dialect for any
snapshot driver, so it must stay free of SQL generation; this is the
prerequisite refactor for adding non-SQL primary sources.

- KtxDialect keeps type, formatDisplayRef, parseDisplayRef,
  columnDisplayTablePartCount, mapDataType, mapToDimensionType
- KtxSqlDialect extends it with quoteIdentifier, formatTableName, and the
  query/sample/statistics generators; the 7 SQL dialects implement it
- add getSqlDialectForDriver for SQL drivers; the 7 connectors and the
  relationship-benchmark harness consume it
- thread the relationship pipeline (profiling/validation/composite/
  discovery) as KtxSqlDialect | null so a non-SQL source skips coverage SQL
  and its candidates stay in review; local-enrichment builds the SQL
  dialect only when the connector advertises readOnlySql

Pure extraction: no behavior change for the existing 7 drivers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(connectors): add MongoDB connector for issue #305

Add a read-only MongoDB connector that treats a database as a primary
context source: collections map to tables and inferred top-level fields to
columns. MongoDB is the first non-SQL source (readOnlySql: false), so
ktx sql and metric compilation do not apply, but its collections flow
through ingest, descriptions, and relationship discovery.

- schema-inference: infer a flat column schema from the most recent
  sample_size documents (by _id desc, or order_by for non-ObjectId keys).
  Union BSON types per field, mark multi-type fields mixed (string), keep
  sub-documents/arrays as a single opaque json column, derive nullability
  from presence, treat _id as the primary key
- connector: KtxMongoDbScanConnector behind an injectable client seam;
  strictly read-only (find/listCollections/estimatedDocumentCount only),
  no executeReadOnly; resolves env:/file: via resolveKtxConfigReference
- core-only KtxMongoDbDialect and a live-database introspection adapter
- wire the mongodb driver: driver union, dialect registry, driver
  registration (scopeConfigKey databases), mongodbConnectionSchema,
  connection-drivers, normalizeDriver, the live-database route, and the
  ktx setup picker. ktx sql is refused by the read-only SQL capability gate
- tests: schema inference, connector snapshot via a fake client, dialect,
  driver-schema parsing, and the ktx sql rejection

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(integrations): document the MongoDB primary source

Add a MongoDB section to the primary-sources reference: connection config
(url, databases, enabled_tables, sample_size, order_by), mongodb+srv/TLS/
Atlas notes, the schema-inference explainer, a features matrix, and the
non-SQL caveat. Update the frontmatter and connection field reference.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(connectors): address review blockers on the MongoDB connector

- introspect: skip estimatedDocumentCount for views. The count command is
  rejected on a MongoDB view (CommandNotSupportedOnView), so counting a view
  aborted introspect for the whole connection; compute estimatedRows only for
  real collections, as ClickHouse does.
- sl: refuse a semantic-layer query against a non-SQL connection instead of
  defaulting it to the Postgres dialect. compileLocalSlQuery (the shared CLI +
  MCP path) now rejects a driver with no SQL dialect via the new
  isSqlQueryableDriver authority, keeping MongoDB context-only per issue #305.
- tests: cover input.tableScope and the empty-scope skip for the Mongo
  connector (the scan layer does not post-filter), the view no-count path, and
  the ktx sl query refusal for a mongodb connection.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* polish(mongodb): compute sampled nullCount and document sampling caveats

Address the non-blocking review notes:

- sampleColumn now counts null/absent values over the sampled window instead of
  returning nullCount: null, since the documents are already in hand
- warn that a custom order_by must be indexed (an unindexed sort hits MongoDB's
  in-memory sort limit on large collections) in the connection schema and docs
- note that sampled values for nested fields are stringified, not faithfully
  serialized, so the json opacity is deliberate

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(examples): add a MongoDB connector example

A manual, container-backed example mirroring examples/postgres-historic:

- docker-compose.yml + init/seed.js seed a representative dataset (nested
  documents, arrays, a Decimal128, a mixed-type field, a nullable field, an
  ObjectId reference, and a view) on first container start
- scripts/smoke.sh + introspect-smoke.mjs assert the connector's inferred
  schema with no LLM credentials — the same introspection entry point ktx
  ingest's database-schema stage uses, including the view-no-count path
- README.md documents the smoke and a full keyless ktx ingest run
  (claude-code LLM + managed sentence-transformers embeddings)

Works with Docker Compose or podman compose. Verified end to end.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: ignore examples/** in knip to fix dead-code false positives

The MongoDB connector example files (examples/mongodb/init/seed.js and
examples/mongodb/scripts/introspect-smoke.mjs) are used at runtime but were
flagged as unused by knip. Add examples/** to the ignore array, matching the
existing .context/** entry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0114qQV8fJ5a5ME3XbMVRzbL

* fix(mongodb): refuse non-SQL connections before SQL analysis

`ktx sql` and the MCP sql_execution tool resolved a SQL-analysis dialect
(falling back to Postgres for a non-SQL driver) and ran read-only
validation before the connector capability gate refused the connection.
For a MongoDB connection that spun up the parser/daemon and produced
Postgres parser diagnostics instead of a clean non-SQL refusal.

Route both entry points through a shared assertSqlQueryableConnection
guard before dialect selection, mirroring compileLocalSlQuery. The
federated duckdb path has no driver and is exempted at each call site.
Add CLI and MCP regression tests asserting validation/connector work
never starts for a MongoDB connection.

* fix(mongodb): pass CI gates (dialect boundary, secrets, setup test)

Three latent failures in the connector surfaced once CI ran on the branch:

- connector.ts imported the concrete KtxMongoDbDialect, which the connector
  dialect-import boundary forbids. Route it through getDialectForDriver('mongodb')
  and widen inferKtxMongoCollectionColumns to the base KtxDialect (it only uses
  mapDataType/mapToDimensionType).
- detect-secrets flagged a test ObjectId hex and the mongodb+srv example URL;
  annotate both with allowlist pragmas.
- the "shows every supported database" setup test omitted the new MongoDB option.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Luca Martial <48870843+luca-martial@users.noreply.github.com>
Co-authored-by: Luca Martial <lucamrtl@gmail.com>
Co-authored-by: Andrey Avtomonov <andreybavt@gmail.com>
This commit is contained in:
Pintouch 2026-06-29 15:17:56 +02:00 committed by GitHub
parent 4f084186f1
commit 2afab61417
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
59 changed files with 1971 additions and 129 deletions

View file

@ -1,7 +1,7 @@
import Database from 'better-sqlite3';
import { join } from 'node:path';
import { describe, expect, it } from 'vitest';
import { getDialectForDriver } from '../../../src/context/connections/dialects.js';
import { getSqlDialectForDriver } from '../../../src/context/connections/dialects.js';
import { snapshotToKtxEnrichedSchema } from '../../../src/context/scan/local-enrichment.js';
import { loadKtxRelationshipBenchmarkFixture, maskKtxRelationshipBenchmarkSnapshot } from '../../../src/context/scan/relationship-benchmarks.js';
import { discoverKtxCompositeRelationships } from '../../../src/context/scan/relationship-composite-candidates.js';
@ -42,7 +42,8 @@ describe('composite relationship discovery detector', () => {
const executor = new TestSqliteExecutor(fixture.dataPath ?? '');
const profiles = await profileKtxRelationshipSchema({
connectionId: snapshot.connectionId,
dialect: getDialectForDriver(snapshot.driver),
driver: snapshot.driver,
dialect: getSqlDialectForDriver(snapshot.driver),
schema,
executor,
ctx: { runId: 'test:composite-profile' },
@ -50,7 +51,7 @@ describe('composite relationship discovery detector', () => {
const result = await discoverKtxCompositeRelationships({
connectionId: snapshot.connectionId,
dialect: getDialectForDriver(snapshot.driver),
dialect: getSqlDialectForDriver(snapshot.driver),
schema,
profiles,
executor,

View file

@ -1,7 +1,7 @@
import Database from 'better-sqlite3';
import { afterEach, describe, expect, it, vi } from 'vitest';
import type { KtxLlmRuntimePort } from '../../../src/context/llm/runtime-port.js';
import { getDialectForDriver } from '../../../src/context/connections/dialects.js';
import { getSqlDialectForDriver } from '../../../src/context/connections/dialects.js';
import { buildDefaultKtxProjectConfig } from '../../../src/context/project/config.js';
import { snapshotToKtxEnrichedSchema } from '../../../src/context/scan/local-enrichment.js';
import {
@ -311,7 +311,7 @@ describe('production relationship discovery', () => {
const result = await discoverKtxRelationships({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
connector: connector(executor),
schema: snapshotToKtxEnrichedSchema(snapshot()),
context: { runId: 'relationship-run-1' },
@ -350,7 +350,7 @@ describe('production relationship discovery', () => {
const schema = naturalKeySnapshot();
const result = await discoverKtxRelationships({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
connector: {
...connector(executor),
introspect: async () => schema,
@ -400,7 +400,7 @@ describe('production relationship discovery', () => {
const result = await discoverKtxRelationships({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
connector: {
...connector(executor),
introspect: async () => sourceSnapshot,
@ -433,7 +433,7 @@ describe('production relationship discovery', () => {
it('keeps candidates review-only when read-only SQL is unavailable', async () => {
const result = await discoverKtxRelationships({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
connector: connector(null),
schema: snapshotToKtxEnrichedSchema(snapshot()),
context: { runId: 'relationship-run-no-sql' },
@ -459,7 +459,7 @@ describe('production relationship discovery', () => {
const sourceSnapshot = declaredForeignKeySnapshot();
const result = await discoverKtxRelationships({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
connector: connector(null),
schema: snapshotToKtxEnrichedSchema(sourceSnapshot),
context: { runId: 'formal-metadata-no-sql' },
@ -506,7 +506,7 @@ describe('production relationship discovery', () => {
const result = await discoverKtxRelationships({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
connector: connector(executor),
schema: snapshotToKtxEnrichedSchema(llmOnlyRelationshipSnapshot()),
context: { runId: 'llm-relationship-orchestrator' },
@ -546,7 +546,7 @@ describe('production relationship discovery', () => {
const result = await discoverKtxRelationships({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
connector: connector(executor),
schema: snapshotToKtxEnrichedSchema(snapshot()),
context: { runId: 'configured-thresholds' },
@ -607,7 +607,7 @@ describe('production relationship discovery', () => {
const result = await discoverKtxRelationships({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
connector: {
...connector(executor),
introspect: async () => richSnapshot,
@ -663,7 +663,7 @@ describe('production relationship discovery', () => {
const result = await discoverKtxRelationships({
connectionId: maskedSnapshot.connectionId,
dialect: getDialectForDriver(maskedSnapshot.driver),
dialect: getSqlDialectForDriver(maskedSnapshot.driver),
connector: testConnector,
schema: snapshotToKtxEnrichedSchema(maskedSnapshot, new Map()),
context: { runId: 'test:production-composite' },

View file

@ -2,7 +2,7 @@ import { readFile } from 'node:fs/promises';
import { join } from 'node:path';
import Database from 'better-sqlite3';
import { afterEach, describe, expect, it, vi } from 'vitest';
import { getDialectForDriver } from '../../../src/context/connections/dialects.js';
import { getSqlDialectForDriver } from '../../../src/context/connections/dialects.js';
import type { KtxEnrichedColumn, KtxEnrichedSchema, KtxEnrichedTable } from '../../../src/context/scan/enrichment-types.js';
import { snapshotToKtxEnrichedSchema } from '../../../src/context/scan/local-enrichment.js';
import { loadKtxRelationshipBenchmarkFixture, maskKtxRelationshipBenchmarkSnapshot } from '../../../src/context/scan/relationship-benchmarks.js';
@ -121,7 +121,8 @@ describe('relationship profiling', () => {
const result = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: schema([
table('accounts', [
column('accounts', 'id', { primaryKey: false, nullable: false }),
@ -183,7 +184,8 @@ describe('relationship profiling', () => {
const result = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: schema([
table('accounts', [
column('accounts', 'id', { nullable: false }),
@ -226,7 +228,8 @@ describe('relationship profiling', () => {
const profiles = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: schema([
table('accounts', [
column('accounts', 'id', { nullable: false }),
@ -277,7 +280,8 @@ describe('relationship profiling', () => {
const first = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: relationshipSchema,
executor,
ctx: { runId: 'profile-cache-run' },
@ -285,7 +289,8 @@ describe('relationship profiling', () => {
});
const second = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: relationshipSchema,
executor,
ctx: { runId: 'profile-cache-run' },
@ -293,7 +298,8 @@ describe('relationship profiling', () => {
});
const third = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: relationshipSchema,
executor,
ctx: { runId: 'profile-cache-fresh-run' },
@ -322,7 +328,8 @@ describe('relationship profiling', () => {
try {
const result = await profileKtxRelationshipSchema({
connectionId: fixture.snapshot.connectionId,
dialect: getDialectForDriver(fixture.snapshot.driver),
driver: fixture.snapshot.driver,
dialect: getSqlDialectForDriver(fixture.snapshot.driver),
schema: snapshotToKtxEnrichedSchema(maskedSnapshot, new Map()),
executor: scaleExecutor,
ctx: { runId: 'scale-stress-profile-query-count' },
@ -367,7 +374,8 @@ describe('relationship profiling', () => {
await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: schemaWithTables(['accounts', 'orders', 'payments', 'refunds']),
executor,
ctx: { runId: 'profile-concurrency' },
@ -403,7 +411,8 @@ describe('relationship profiling', () => {
const result = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: schemaWithTables(['accounts', 'orders']),
executor,
ctx: { runId: 'profile-error-isolated' },

View file

@ -1,6 +1,6 @@
import Database from 'better-sqlite3';
import { afterEach, describe, expect, it } from 'vitest';
import { getDialectForDriver } from '../../../src/context/connections/dialects.js';
import { getSqlDialectForDriver } from '../../../src/context/connections/dialects.js';
import type { KtxEnrichedColumn, KtxEnrichedSchema, KtxEnrichedTable } from '../../../src/context/scan/enrichment-types.js';
import { generateKtxRelationshipDiscoveryCandidates } from '../../../src/context/scan/relationship-candidates.js';
import type { KtxRelationshipProfileArtifact } from '../../../src/context/scan/relationship-profiling.js';
@ -102,7 +102,8 @@ describe('relationship validation', () => {
const testSchema = schema();
const profiles = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: testSchema,
executor,
ctx: { runId: 'validate-test' },
@ -113,7 +114,7 @@ describe('relationship validation', () => {
const validated = await validateKtxRelationshipDiscoveryCandidates({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
candidates,
profiles,
executor,
@ -151,7 +152,8 @@ describe('relationship validation', () => {
const testSchema = schema();
const profiles = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: testSchema,
executor,
ctx: { runId: 'validate-test' },
@ -162,7 +164,7 @@ describe('relationship validation', () => {
const validated = await validateKtxRelationshipDiscoveryCandidates({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
candidates,
profiles,
executor,
@ -201,7 +203,8 @@ describe('relationship validation', () => {
const testSchema = schema();
const profiles = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: testSchema,
executor,
ctx: { runId: 'validate-budget-profile' },
@ -214,7 +217,7 @@ describe('relationship validation', () => {
const validated = await validateKtxRelationshipDiscoveryCandidates({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
candidates,
profiles,
executor,
@ -256,7 +259,8 @@ describe('relationship validation', () => {
]);
const profiles = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: testSchema,
executor,
ctx: { runId: 'validate-zero-budget-profile' },
@ -266,7 +270,7 @@ describe('relationship validation', () => {
const validated = await validateKtxRelationshipDiscoveryCandidates({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
candidates,
profiles,
executor,
@ -303,7 +307,8 @@ describe('relationship validation', () => {
]);
const profiles = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: testSchema,
executor,
ctx: { runId: 'llm-rejected-validation' },
@ -332,7 +337,7 @@ describe('relationship validation', () => {
const [validated] = await validateKtxRelationshipDiscoveryCandidates({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
candidates: [llmCandidate],
profiles,
executor,
@ -377,7 +382,8 @@ describe('relationship validation', () => {
]);
const profiles = await profileKtxRelationshipSchema({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
driver: 'sqlite',
dialect: getSqlDialectForDriver('sqlite'),
schema: testSchema,
executor,
ctx: { runId: 'validation-concurrency-profile' },
@ -386,7 +392,7 @@ describe('relationship validation', () => {
await validateKtxRelationshipDiscoveryCandidates({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
candidates,
profiles,
executor: throttled,
@ -478,7 +484,7 @@ describe('relationship validation', () => {
const [validated] = await validateKtxRelationshipDiscoveryCandidates({
connectionId: 'warehouse',
dialect: getDialectForDriver('sqlite'),
dialect: getSqlDialectForDriver('sqlite'),
candidates: [candidate],
profiles,
executor,