feat(connectors): add MongoDB connector (#305) (#310)

* refactor(connectors): split KtxDialect into core and KtxSqlDialect

Separate the dialect contract into a driver-agnostic core (display/ref
formatting and type mapping) and a SQL-only extension (query generators).
The catalog and entity-details paths resolve the core dialect for any
snapshot driver, so it must stay free of SQL generation; this is the
prerequisite refactor for adding non-SQL primary sources.

- KtxDialect keeps type, formatDisplayRef, parseDisplayRef,
  columnDisplayTablePartCount, mapDataType, mapToDimensionType
- KtxSqlDialect extends it with quoteIdentifier, formatTableName, and the
  query/sample/statistics generators; the 7 SQL dialects implement it
- add getSqlDialectForDriver for SQL drivers; the 7 connectors and the
  relationship-benchmark harness consume it
- thread the relationship pipeline (profiling/validation/composite/
  discovery) as KtxSqlDialect | null so a non-SQL source skips coverage SQL
  and its candidates stay in review; local-enrichment builds the SQL
  dialect only when the connector advertises readOnlySql

Pure extraction: no behavior change for the existing 7 drivers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(connectors): add MongoDB connector for issue #305

Add a read-only MongoDB connector that treats a database as a primary
context source: collections map to tables and inferred top-level fields to
columns. MongoDB is the first non-SQL source (readOnlySql: false), so
ktx sql and metric compilation do not apply, but its collections flow
through ingest, descriptions, and relationship discovery.

- schema-inference: infer a flat column schema from the most recent
  sample_size documents (by _id desc, or order_by for non-ObjectId keys).
  Union BSON types per field, mark multi-type fields mixed (string), keep
  sub-documents/arrays as a single opaque json column, derive nullability
  from presence, treat _id as the primary key
- connector: KtxMongoDbScanConnector behind an injectable client seam;
  strictly read-only (find/listCollections/estimatedDocumentCount only),
  no executeReadOnly; resolves env:/file: via resolveKtxConfigReference
- core-only KtxMongoDbDialect and a live-database introspection adapter
- wire the mongodb driver: driver union, dialect registry, driver
  registration (scopeConfigKey databases), mongodbConnectionSchema,
  connection-drivers, normalizeDriver, the live-database route, and the
  ktx setup picker. ktx sql is refused by the read-only SQL capability gate
- tests: schema inference, connector snapshot via a fake client, dialect,
  driver-schema parsing, and the ktx sql rejection

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(integrations): document the MongoDB primary source

Add a MongoDB section to the primary-sources reference: connection config
(url, databases, enabled_tables, sample_size, order_by), mongodb+srv/TLS/
Atlas notes, the schema-inference explainer, a features matrix, and the
non-SQL caveat. Update the frontmatter and connection field reference.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(connectors): address review blockers on the MongoDB connector

- introspect: skip estimatedDocumentCount for views. The count command is
  rejected on a MongoDB view (CommandNotSupportedOnView), so counting a view
  aborted introspect for the whole connection; compute estimatedRows only for
  real collections, as ClickHouse does.
- sl: refuse a semantic-layer query against a non-SQL connection instead of
  defaulting it to the Postgres dialect. compileLocalSlQuery (the shared CLI +
  MCP path) now rejects a driver with no SQL dialect via the new
  isSqlQueryableDriver authority, keeping MongoDB context-only per issue #305.
- tests: cover input.tableScope and the empty-scope skip for the Mongo
  connector (the scan layer does not post-filter), the view no-count path, and
  the ktx sl query refusal for a mongodb connection.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* polish(mongodb): compute sampled nullCount and document sampling caveats

Address the non-blocking review notes:

- sampleColumn now counts null/absent values over the sampled window instead of
  returning nullCount: null, since the documents are already in hand
- warn that a custom order_by must be indexed (an unindexed sort hits MongoDB's
  in-memory sort limit on large collections) in the connection schema and docs
- note that sampled values for nested fields are stringified, not faithfully
  serialized, so the json opacity is deliberate

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(examples): add a MongoDB connector example

A manual, container-backed example mirroring examples/postgres-historic:

- docker-compose.yml + init/seed.js seed a representative dataset (nested
  documents, arrays, a Decimal128, a mixed-type field, a nullable field, an
  ObjectId reference, and a view) on first container start
- scripts/smoke.sh + introspect-smoke.mjs assert the connector's inferred
  schema with no LLM credentials — the same introspection entry point ktx
  ingest's database-schema stage uses, including the view-no-count path
- README.md documents the smoke and a full keyless ktx ingest run
  (claude-code LLM + managed sentence-transformers embeddings)

Works with Docker Compose or podman compose. Verified end to end.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: ignore examples/** in knip to fix dead-code false positives

The MongoDB connector example files (examples/mongodb/init/seed.js and
examples/mongodb/scripts/introspect-smoke.mjs) are used at runtime but were
flagged as unused by knip. Add examples/** to the ignore array, matching the
existing .context/** entry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0114qQV8fJ5a5ME3XbMVRzbL

* fix(mongodb): refuse non-SQL connections before SQL analysis

`ktx sql` and the MCP sql_execution tool resolved a SQL-analysis dialect
(falling back to Postgres for a non-SQL driver) and ran read-only
validation before the connector capability gate refused the connection.
For a MongoDB connection that spun up the parser/daemon and produced
Postgres parser diagnostics instead of a clean non-SQL refusal.

Route both entry points through a shared assertSqlQueryableConnection
guard before dialect selection, mirroring compileLocalSlQuery. The
federated duckdb path has no driver and is exempted at each call site.
Add CLI and MCP regression tests asserting validation/connector work
never starts for a MongoDB connection.

* fix(mongodb): pass CI gates (dialect boundary, secrets, setup test)

Three latent failures in the connector surfaced once CI ran on the branch:

- connector.ts imported the concrete KtxMongoDbDialect, which the connector
  dialect-import boundary forbids. Route it through getDialectForDriver('mongodb')
  and widen inferKtxMongoCollectionColumns to the base KtxDialect (it only uses
  mapDataType/mapToDimensionType).
- detect-secrets flagged a test ObjectId hex and the mongodb+srv example URL;
  annotate both with allowlist pragmas.
- the "shows every supported database" setup test omitted the new MongoDB option.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Luca Martial <48870843+luca-martial@users.noreply.github.com>
Co-authored-by: Luca Martial <lucamrtl@gmail.com>
Co-authored-by: Andrey Avtomonov <andreybavt@gmail.com>

docs-site/content/docs/integrations/primary-sources.mdx

@@ -1,6 +1,6 @@
 ---
 title: Primary Sources
-description: Connect ktx to PostgreSQL, Snowflake, BigQuery, MySQL, ClickHouse, SQL Server, or SQLite.
+description: Connect ktx to PostgreSQL, Snowflake, BigQuery, MySQL, ClickHouse, SQL Server, SQLite, or MongoDB.
 ---
 
 **ktx** connects to your data warehouse or database to build schema context,
@@ -26,10 +26,12 @@ Agents should prefer environment or file references over literal secrets.
 
 | Field | Required | Applies to | Description |
 |-------|----------|------------|-------------|
-| `driver` | Yes | all connections | Connector driver such as `postgres`, `snowflake`, `bigquery`, `mysql`, `clickhouse`, `sqlserver`, or `sqlite` |
+| `driver` | Yes | all connections | Connector driver such as `postgres`, `snowflake`, `bigquery`, `mysql`, `clickhouse`, `sqlserver`, `sqlite`, or `mongodb` |
 | `url` | One of the connection methods | URL-style connectors | Database URL, `env:NAME`, or `file:/path/to/secret` |
 | `host`, `port`, `database`, `username`, `password` | One of the connection methods | PostgreSQL, MySQL, SQL Server | Field-by-field connection values |
 | `schema` or `schemas` | No | schema-aware warehouses | Single schema or list of schemas to scan |
+| `databases` | No | ClickHouse, MongoDB | List of databases to scan |
+| `sample_size`, `order_by` | No | MongoDB | Schema-inference sampling controls (recent documents, sort field) |
 | `context.queryHistory` | No | PostgreSQL, Snowflake, BigQuery | Enables query-history ingestion when the warehouse supports it |
 | `path` | Yes for path-style SQLite | SQLite | Local SQLite database path or `env:NAME` reference |
 | `max_bytes_billed` | No | BigQuery | Maximum bytes billed per query job |
@@ -510,6 +512,87 @@ No authentication required - SQLite is file-based. The file must be readable by
 - Foreign key enforcement requires explicit `PRAGMA foreign_keys = ON`
 - Database file must exist before `ktx connection test` or ingest runs
 
+---
+
+## MongoDB
+
+Connects to MongoDB as a primary context source. **ktx** treats each collection
+as a table and each inferred top-level field as a column. MongoDB is a non-SQL
+source: `ktx sql` and semantic-layer metric compilation do not apply to a MongoDB
+connection, but its collections still flow through `ktx ingest`, descriptions, and
+relationship discovery.
+
+### Connection config
+
+```yaml title="ktx.yaml"
+connections:
+  mongo-prod:
+    driver: mongodb
+    url: env:MONGO_URL
+    databases: [app]
+    enabled_tables: [app.users, app.orders] # optional collection allowlist
+    sample_size: 1000
+    # order_by: createdAt                    # only when _id is not an ObjectId
+```
+
+Standard `mongodb://` and `mongodb+srv://` connection strings are supported,
+including TLS and MongoDB Atlas — pass the full connection string (with its
+query parameters) as `url`. The `databases` list selects which databases to
+introspect; if omitted, **ktx** uses the database in the URL path. `ktx setup`
+also offers MongoDB and stores the selected databases under
+`connections.<id>.databases`.
+
+### Authentication
+
+| Method | Config |
+|--------|--------|
+| Connection string | `url: env:MONGO_URL` or `url: file:/path/to/secret` |
+| Atlas / TLS | Use a `mongodb+srv://` URL with the credentials and TLS options Atlas provides |
+
+### Schema inference
+
+MongoDB has no fixed schema, so **ktx** infers one by sampling the most recent
+`sample_size` documents per collection (default 1000), sorted by `_id`
+descending. Because an ObjectId embeds its creation time, this captures the
+collection's current shape with zero configuration. When `_id` is not an
+ObjectId (custom string or UUID keys), set `order_by` to a timestamp field such
+as `createdAt` so "most recent" is well-defined. A custom `order_by` field
+should be indexed — an unindexed sort hits MongoDB's in-memory sort limit and
+fails on large collections (`_id`, the default, is always indexed).
+
+For each top-level field, **ktx** unions the BSON types seen and derives
+nullability from how often the field is present:
+
+- Scalar BSON types map to `string`, `number`, `time`, or `boolean`
+- A field seen with more than one type is recorded as `mixed` and treated as a string
+- Sub-documents and arrays become a single opaque `json` column (no dotted-path
+  columns); their sampled values are stringified, not faithfully serialized
+- `_id` is the primary key
+
+### Features
+
+| Feature | Supported | Notes |
+|---------|-----------|-------|
+| Collections (as tables) | Yes | Via `listCollections`; `system.*` collections are excluded |
+| Primary keys | Yes | `_id` |
+| Foreign keys | No | MongoDB has no formal foreign keys |
+| Row count estimates | Yes | Via `estimatedDocumentCount` |
+| Column statistics | No | - |
+| Query history | No | - |
+| Table sampling | Yes | Reads the most recent documents |
+| Nested analysis | Yes | Sub-documents and arrays modeled as opaque `json` |
+| Read-only SQL (`ktx sql`) | No | MongoDB is not a SQL source |
+
+### Dialect notes
+
+- Strictly read-only: the connector only issues `find`, `listCollections`,
+  `estimatedDocumentCount`, and read aggregations
+- Sampling rides the `_id` index and uses a server-side time limit so large
+  collections do not stall a run; a custom `order_by` must be indexed for the
+  same guarantee
+- `sample_size` trades inference coverage for speed; raise it for collections
+  with highly variable documents
+
 ## Common errors
 
 | Error or symptom | Likely cause | Recovery |