fix(snowflake): unblock multi-schema ingest and relationship discovery (#204)

* feat(setup): drop redundant Snowflake schema prompt; fall back to free-text on listSchemas failure

Snowflake setup previously asked for a single schema as free text, then
ran a multiselect against the discovered schemas — two schema questions
back-to-back, with the first being only a session bootstrap. The SDK's
`schema` is optional, so the bootstrap step is unnecessary.

- Remove the free-text Snowflake schema prompt; only pass `schema` to
  snowflake-sdk when one is configured.
- When `listSchemas()` fails (e.g. role lacks SHOW SCHEMAS), prompt the
  user for a comma-separated list, persist it as `schema_names`, and use
  it as both the table-list filter and the multiselect default. Applies
  to every driver with a scope-discovery spec, not just Snowflake.
- Update docs to lead with `schema_names`; keep `schema_name` as a
  documented single-schema shorthand.

* fix(snowflake): keep introspecting when primary-key discovery is denied

The PK query joins INFORMATION_SCHEMA.TABLE_CONSTRAINTS and
INFORMATION_SCHEMA.KEY_COLUMN_USAGE, which require grants the
connection role may not have. Previously a 'SQL compilation error:
Object ANALYTICS.INFORMATION_SCHEMA.KEY_COLUMN_USAGE does not exist
or not authorized' aborted the entire introspect — schemas, columns,
and row counts were all discarded over a missing nice-to-have.

Wrap the constraint query in try/catch, log a one-line warning per
schema, and return an empty PK map. Columns end up with
primaryKey=false; relationship inference still has FK and profiling
to fall back on.

* fix(scan): unblock relationship discovery on Snowflake

Two adjacent bugs prevented the scan's relationship pipeline from producing
any joins on a Snowflake warehouse:

- relationship-profiling.ts fell through to a default `GROUP_CONCAT` branch
  for unknown drivers. Snowflake has no GROUP_CONCAT, so every per-table
  profile query failed with "Unknown function GROUP_CONCAT". Add an explicit
  Snowflake branch that uses LISTAGG with a literal '\x1f' delimiter
  (Snowflake requires the delimiter to be a constant, so CHR(31) is rejected).
- description-generation.ts destructured `connector.sampleTable` and
  `connector.sampleColumn` into bare locals, losing the `this` binding when
  the class-method connectors (Snowflake, Postgres, MySQL) were invoked.
  Every sample call threw "Cannot read properties of undefined (reading
  'assertConnection')" and degraded LLM descriptions to metadata-only
  prompts. Call the methods through the connector instead.

Without these, even after the primary-key probe is allowed to fail softly,
the scan ends up with 0 validated relationships and an empty `joins:` block
in every shard YAML.

* test(scan): cover table-ref helpers

* feat(scan): plumb tableScope through live-database introspection port

* feat(scan): apply tableScope during metadata fetch

* feat(scan): enforce table scope at fetch boundary

* feat(scan): pool Snowflake sessions and batch enrichment for faster ingest (#206)

* feat(cli): add RSA key-pair auth option to Snowflake setup wizard

Extends the interactive Snowflake setup flow with an authentication-method
prompt (password vs RSA/JWT key-pair). The RSA branch collects a private-key
path (env/file/absolute) and an optional passphrase; the resulting connection
config records `authMethod: 'rsa'` with `privateKey` and `passphrase` instead
of `password`.

* feat(scan): pool Snowflake sessions

* fix(scan): reuse structural snapshots and cleanup connectors

* feat(scan): parallelize relationship profiling

* feat(scan): batch table description generation

* docs: document Snowflake ingest concurrency knobs

* fix(scan): close Snowflake ingest perf verification gaps

* fix(scan): keep batched description failure bounded

* feat(scan): dispatch query-history probes by connection driver

Extract historic-sql dialect resolution into a shared helper so the
status-project readiness check and the local ingest factory agree on
which connections enable query history and which probe to run. The
status command now picks the postgres/snowflake/bigquery probe based on
the connection's driver instead of always reporting against postgres,
which previously caused snowflake connections with queryHistory.enabled
to surface a misleading "driver is snowflake" failure.

Also drops a noisy console.warn from Snowflake primary-key discovery —
INFORMATION_SCHEMA.KEY_COLUMN_USAGE is commonly ungranted for read-only
roles and the FK + profiling paths handle the empty PK map already.

* fix(llm): allow StructuredOutput tool and raise maxTurns for generateObject

The Claude Code agent SDK announces an internal pseudo-tool named
StructuredOutput in the system/init message whenever outputFormat is set
to { type: 'json_schema' }. The runtime's isolation check built its
allowedToolIds set only from MCP tool ids and treated StructuredOutput
as an unexpected host-injected tool, so every generateObject call threw
"Claude Code runtime isolation failed: tools=StructuredOutput ..." and
the table-descriptions and relationship-LLM-proposal enrichment stages
recorded null output across the board.

Whitelist StructuredOutput specifically in generateObject's
allowedToolIds — the check also enforces missing_tools symmetry, so
generateText and runAgentLoop, which do not see StructuredOutput, must
not require it.

generateObject also ran with maxTurns: 1, which the model intermittently
breached when it emitted thinking text before the structured response.
Raised to 5 to give the schema-bound call enough headroom without
allowing unbounded loops. The existing tests now exercise the path with
an init message that announces StructuredOutput so the regression cannot
slip back in.

* chore(scripts): add ktx-reset.sh project-cleanup helper

Convenience script for repeatable ingest testing: takes a project
directory and prunes everything except ktx.yaml and .ktx/secrets/, so
the next ktx setup or ktx ingest run starts from a known-clean state.

python/ktx-daemon/src/ktx_daemon/database_introspection.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import json
 from collections.abc import Callable, Mapping, Sequence
 from dataclasses import dataclass
 from datetime import datetime, timezone
@@ -24,6 +25,16 @@ join pg_catalog.pg_class c
   and c.relname = t.table_name
 where t.table_schema = any(%s)
   and t.table_type = 'BASE TABLE'
+  and (
+    %s::jsonb is null
+    or exists (
+      select 1
+      from jsonb_to_recordset(%s::jsonb) as scope(catalog text, db text, name text)
+      where (scope.catalog is null or scope.catalog = current_database())
+        and (scope.db is null or scope.db = t.table_schema)
+        and scope.name = t.table_name
+    )
+  )
 order by t.table_schema, t.table_name
 """
 
@@ -52,6 +63,16 @@ where n.nspname = any(%s)
   and c.relkind in ('r', 'p')
   and a.attnum > 0
   and not a.attisdropped
+  and (
+    %s::jsonb is null
+    or exists (
+      select 1
+      from jsonb_to_recordset(%s::jsonb) as scope(catalog text, db text, name text)
+      where (scope.catalog is null or scope.catalog = current_database())
+        and (scope.db is null or scope.db = n.nspname)
+        and scope.name = c.relname
+    )
+  )
 order by n.nspname, c.relname, a.attnum
 """
 
@@ -80,6 +101,16 @@ join information_schema.key_column_usage target_key
   and target_key.ordinal_position = source_key.position_in_unique_constraint
 where source_constraint.constraint_type = 'FOREIGN KEY'
   and source_constraint.table_schema = any(%s)
+  and (
+    %s::jsonb is null
+    or exists (
+      select 1
+      from jsonb_to_recordset(%s::jsonb) as scope(catalog text, db text, name text)
+      where (scope.catalog is null or scope.catalog = current_database())
+        and (scope.db is null or scope.db = source_constraint.table_schema)
+        and scope.name = source_constraint.table_name
+    )
+  )
 order by source_constraint.table_schema, source_constraint.table_name, source_constraint.constraint_name, source_key.ordinal_position
 """
 
@@ -108,6 +139,12 @@ class LiveDatabaseTable(BaseModel):
     foreign_keys: list[LiveDatabaseForeignKey] = Field(default_factory=list)
 
 
+class LiveDatabaseTableScopeRef(BaseModel):
+    catalog: str | None = None
+    db: str | None = None
+    name: str
+
+
 class DatabaseIntrospectionRequest(BaseModel):
     connection_id: str
     driver: str = "postgres"
@@ -115,6 +152,7 @@ class DatabaseIntrospectionRequest(BaseModel):
     schemas: list[str] = Field(default_factory=lambda: ["public"])
     statement_timeout_ms: int = Field(default=30_000, ge=1)
     connection_timeout_seconds: int = Field(default=5, ge=1)
+    table_scope: list[LiveDatabaseTableScopeRef] | None = None
 
     @field_validator("schemas")
     @classmethod
@@ -169,6 +207,23 @@ def _statement_timeout_config(statement_timeout_ms: int) -> tuple[str, tuple[str
     )
 
 
+def _table_scope_json(
+    table_scope: Sequence[LiveDatabaseTableScopeRef] | None,
+) -> str | None:
+    if table_scope is None:
+        return None
+    return json.dumps(
+        [
+            {
+                "catalog": ref.catalog,
+                "db": ref.db,
+                "name": ref.name,
+            }
+            for ref in table_scope
+        ]
+    )
+
+
 def _load_postgres_rows(
     request: DatabaseIntrospectionRequest,
 ) -> DatabaseIntrospectionRows:
@@ -190,7 +245,8 @@ def _load_postgres_rows(
         connection.execute("BEGIN READ ONLY")
         try:
             connection.execute(*_statement_timeout_config(request.statement_timeout_ms))
-            params = (request.schemas,)
+            scope_json = _table_scope_json(request.table_scope)
+            params = (request.schemas, scope_json, scope_json)
             table_rows = list(connection.execute(TABLES_SQL, params))
             column_rows = list(connection.execute(COLUMNS_SQL, params))
             foreign_key_rows = list(connection.execute(FOREIGN_KEYS_SQL, params))

python/ktx-daemon/tests/test_app.py

@@ -155,6 +155,7 @@ def test_database_introspect_endpoint_returns_snapshot() -> None:
             "driver": "postgres",
             "url": "postgresql://readonly@example.test/warehouse",
             "schemas": ["public"],
+            "table_scope": [{"db": "public", "name": "orders"}],
         },
     )
 
@@ -162,6 +163,8 @@ def test_database_introspect_endpoint_returns_snapshot() -> None:
     assert response.json()["connection_id"] == "warehouse"
     assert response.json()["tables"][0]["name"] == "orders"
     assert calls[0].connection_id == "warehouse"
+    assert calls[0].table_scope[0].db == "public"
+    assert calls[0].table_scope[0].name == "orders"
 
 
 def test_database_introspect_endpoint_maps_value_error_to_400() -> None:

python/ktx-daemon/tests/test_cli.py

@@ -311,6 +311,9 @@ def test_database_introspect_command_reads_stdin_and_writes_json(
         assert request.connection_id == "warehouse"
         assert request.driver == "postgres"
         assert request.schemas == ["public"]
+        assert request.table_scope is not None
+        assert request.table_scope[0].db == "public"
+        assert request.table_scope[0].name == "orders"
         return DatabaseIntrospectionResponse(
             connection_id="warehouse",
             extracted_at="2026-04-28T10:00:00+00:00",
@@ -337,7 +340,7 @@ def test_database_introspect_command_reads_stdin_and_writes_json(
         sys,
         "stdin",
         io.StringIO(
-            '{"connection_id":"warehouse","driver":"postgres","url":"postgresql://readonly@example.test/warehouse","schemas":["public"]}'
+            '{"connection_id":"warehouse","driver":"postgres","url":"postgresql://readonly@example.test/warehouse","schemas":["public"],"table_scope":[{"db":"public","name":"orders"}]}'
         ),
     )
 

python/ktx-daemon/tests/test_database_introspection.py

@@ -5,7 +5,9 @@ import pytest
 from ktx_daemon.database_introspection import (
     DatabaseIntrospectionRequest,
     DatabaseIntrospectionRows,
+    LiveDatabaseTableScopeRef,
     _statement_timeout_config,
+    _table_scope_json,
     introspect_database_response,
 )
 
@@ -146,6 +148,22 @@ def test_database_introspection_request_rejects_empty_schema_list() -> None:
         )
 
 
+def test_table_scope_json_serializes_null_wildcards() -> None:
+    assert _table_scope_json(
+        [
+            LiveDatabaseTableScopeRef(catalog=None, db="public", name="orders"),
+            LiveDatabaseTableScopeRef(
+                catalog="warehouse",
+                db="marts",
+                name="customers",
+            ),
+        ]
+    ) == (
+        '[{"catalog": null, "db": "public", "name": "orders"}, '
+        '{"catalog": "warehouse", "db": "marts", "name": "customers"}]'
+    )
+
+
 def test_statement_timeout_config_uses_parameterized_set_config() -> None:
     assert _statement_timeout_config(30_000) == (
         "SELECT set_config('statement_timeout', %s, true)",