Merge origin/main into merge-scan-into-ingest-v1

This commit is contained in:
Andrey Avtomonov 2026-05-14 00:47:03 +02:00
commit 9131c82724
98 changed files with 3207 additions and 1007 deletions

View file

@ -40,37 +40,37 @@ This plan does not update `examples/postgres-historic/README.md` or `examples/po
Modify:
- `packages/context/src/ingest/adapters/historic-sql/types.ts`
- `packages/context/src/ingest/adapters/historic-sql/types.ts`
Adds optional probe `info` notes and lets injected historic-SQL dependencies use any reader/query client pair while preserving the existing Postgres-specific option.
- `packages/context/src/ingest/adapters/historic-sql/postgres-pgss-reader.ts`
- `packages/context/src/ingest/adapters/historic-sql/postgres-pgss-reader.ts`
Moves low `pg_stat_statements.max` from `warnings` to `info`.
- `packages/context/src/ingest/adapters/historic-sql/postgres-pgss-reader.test.ts`
- `packages/context/src/ingest/adapters/historic-sql/postgres-pgss-reader.test.ts`
Locks `track = none` as warning and low `max` as info.
- `packages/context/src/ingest/adapters/historic-sql/bigquery-query-history-reader.ts`
- `packages/context/src/ingest/adapters/historic-sql/bigquery-query-history-reader.ts`
Returns `{ warnings: [], info: [] }` from `probe()`.
- `packages/context/src/ingest/adapters/historic-sql/bigquery-query-history-reader.test.ts`
- `packages/context/src/ingest/adapters/historic-sql/bigquery-query-history-reader.test.ts`
Locks the BigQuery probe return object.
- `packages/context/src/ingest/adapters/historic-sql/snowflake-query-history-reader.ts`
- `packages/context/src/ingest/adapters/historic-sql/snowflake-query-history-reader.ts`
Returns `{ warnings: [], info: [] }` from `probe()`.
- `packages/context/src/ingest/adapters/historic-sql/snowflake-query-history-reader.test.ts`
- `packages/context/src/ingest/adapters/historic-sql/snowflake-query-history-reader.test.ts`
Locks the Snowflake probe return object.
- `packages/context/src/ingest/adapters/historic-sql/stage-unified.test.ts`
- `packages/context/src/ingest/adapters/historic-sql/stage-unified.test.ts`
Updates test readers to return the normalized probe shape.
- `packages/context/src/ingest/adapters/historic-sql/historic-sql.adapter.test.ts`
- `packages/context/src/ingest/adapters/historic-sql/historic-sql.adapter.test.ts`
Updates test readers to return the normalized probe shape.
- `packages/context/src/ingest/local-adapters.ts`
- `packages/context/src/ingest/local-adapters.ts`
Accepts generic historic-SQL reader/query-client dependencies while keeping `postgresQueryClient` as the compatibility input used by current callers.
- `packages/context/src/ingest/local-adapters.test.ts`
- `packages/context/src/ingest/local-adapters.test.ts`
Verifies generic reader/query-client injection and the existing Postgres compatibility path.
- `packages/cli/src/local-adapters.ts`
- `packages/cli/src/local-adapters.ts`
Chooses Postgres, BigQuery, or Snowflake historic-SQL readers/query clients from the configured connection.
- `packages/cli/src/local-adapters.test.ts`
- `packages/cli/src/local-adapters.test.ts`
Adds direct tests for CLI local adapter registration for Postgres, BigQuery, and Snowflake.
- `packages/cli/src/historic-sql-doctor.ts`
- `packages/cli/src/historic-sql-doctor.ts`
Treats info-only Postgres probe notes as a passing doctor check, and warnings as warnings.
- `packages/cli/src/historic-sql-doctor.test.ts`
- `packages/cli/src/historic-sql-doctor.test.ts`
Verifies low `pg_stat_statements.max` is pass/detail, while `track = none` remains warn.
- `packages/cli/src/doctor.test.ts`
- `packages/cli/src/doctor.test.ts`
Updates the project doctor integration expectation for the new info-only behavior.
## Task 1: Normalize Historic-SQL Probe Results

View file

@ -44,7 +44,7 @@ Remaining acceptance gap this plan covers:
Create:
- `packages/context/src/ingest/adapters/historic-sql/local-ingest-acceptance.test.ts`
- `packages/context/src/ingest/adapters/historic-sql/local-ingest-acceptance.test.ts`
Owns the end-to-end local regression for the redesigned historic-SQL pipeline. It uses the real adapter and local ingest runner, with fake deterministic reader/analysis/agent components so the test does not need a live database or LLM provider.
## Task 1: Add Real-Adapter Local Ingest Acceptance Coverage

View file

@ -41,50 +41,50 @@ The next plan after this one should cover search enrichment from spec §6.2.3-§
Create:
- `packages/context/src/ingest/adapters/historic-sql/skill-schemas.ts`
- `packages/context/src/ingest/adapters/historic-sql/skill-schemas.ts`
Owns the shared zod schemas for historic-SQL LLM outputs.
- `packages/context/src/ingest/adapters/historic-sql/skill-schemas.test.ts`
- `packages/context/src/ingest/adapters/historic-sql/skill-schemas.test.ts`
Locks schema acceptance, JSON schema generation, and future-key tolerance.
- `python/ktx-daemon/src/ktx_daemon/sql_analysis.py`
- `python/ktx-daemon/src/ktx_daemon/sql_analysis.py`
Implements batch sqlglot parsing for table and clause-level column extraction.
- `python/ktx-daemon/tests/test_sql_analysis.py`
- `python/ktx-daemon/tests/test_sql_analysis.py`
Tests batch parser behavior without FastAPI.
Modify:
- `packages/context/src/ingest/index.ts`
- `packages/context/src/ingest/index.ts`
Exports the new historic-SQL skill schemas.
- `packages/context/src/sl/types.ts`
- `packages/context/src/sl/types.ts`
Adds `usage?: TableUsageOutput` to `SemanticLayerSource`.
- `packages/context/src/sl/schemas.ts`
- `packages/context/src/sl/schemas.ts`
Accepts `usage` in standalone and overlay semantic-layer source validation.
- `packages/context/src/sl/semantic-layer.service.ts`
- `packages/context/src/sl/semantic-layer.service.ts`
Projects manifest `usage` onto `SemanticLayerSource` and composes overlay usage intentionally.
- `packages/context/src/sl/semantic-layer.service.test.ts`
- `packages/context/src/sl/semantic-layer.service.test.ts`
Tests source schema acceptance, manifest projection, and overlay composition.
- `packages/context/src/ingest/adapters/live-database/manifest.ts`
- `packages/context/src/ingest/adapters/live-database/manifest.ts`
Adds `LiveDatabaseManifestTableEntry.usage`, existing-usage inputs, and `mergeUsagePreservingExternal()`.
- `packages/context/src/ingest/adapters/live-database/manifest.test.ts`
- `packages/context/src/ingest/adapters/live-database/manifest.test.ts`
Tests scan-managed usage replacement while preserving external keys.
- `packages/context/src/scan/local-enrichment-artifacts.ts`
- `packages/context/src/scan/local-enrichment-artifacts.ts`
Loads existing manifest usage and passes it through scan manifest rebuilds.
- `packages/context/src/scan/local-enrichment-artifacts.test.ts`
- `packages/context/src/scan/local-enrichment-artifacts.test.ts`
Tests that structural scan rewrites preserve existing usage.
- `python/ktx-daemon/src/ktx_daemon/app.py`
- `python/ktx-daemon/src/ktx_daemon/app.py`
Registers `/sql/analyze-batch`.
- `python/ktx-daemon/tests/test_app.py`
- `python/ktx-daemon/tests/test_app.py`
Tests the FastAPI endpoint.
- `packages/context/src/sql-analysis/ports.ts`
- `packages/context/src/sql-analysis/ports.ts`
Adds batch analysis types and `SqlAnalysisPort.analyzeBatch()`.
- `packages/context/src/sql-analysis/index.ts`
- `packages/context/src/sql-analysis/index.ts`
Exports the new batch analysis types.
- `packages/context/src/sql-analysis/http-sql-analysis-port.ts`
- `packages/context/src/sql-analysis/http-sql-analysis-port.ts`
Maps `/sql/analyze-batch` request and response payloads.
- `packages/context/src/sql-analysis/http-sql-analysis-port.test.ts`
- `packages/context/src/sql-analysis/http-sql-analysis-port.test.ts`
Tests HTTP mapping and malformed response rejection.
- `packages/cli/src/managed-python-http.test.ts`
- `packages/cli/src/managed-python-http.test.ts`
Verifies the managed daemon wrapper routes `analyzeBatch()`.
- Existing test files with `SqlAnalysisPort` object literals
- Existing test files with `SqlAnalysisPort` object literals
Add a no-op `analyzeBatch: async () => new Map()` while legacy paths still use `analyzeForFingerprint()`.
## Task 1: Add Historic SQL Skill Schemas

View file

@ -39,13 +39,13 @@ Remaining gap this plan fixes:
## File Structure
- Modify `scripts/examples-docs.test.mjs`
- Modify `scripts/examples-docs.test.mjs`
Pins docs and smoke script to the sharded pattern WorkUnit contract.
- Modify `examples/postgres-historic/scripts/smoke.sh`
- Modify `examples/postgres-historic/scripts/smoke.sh`
Validates `patterns-input/part-*.json` shard files and `historic-sql-patterns-part-*` stage-only WorkUnits.
- Modify `examples/postgres-historic/README.md`
- Modify `examples/postgres-historic/README.md`
Documents `patterns-input.json` as the full audit artifact and `patterns-input/part-*.json` as bounded pattern WorkUnit input.
- Modify `examples/README.md`
- Modify `examples/README.md`
Updates the short example catalog entry with the same audit-vs-shard wording.
### Task 1: Pin Example Tests To Pattern Shards

View file

@ -30,23 +30,23 @@ No existing spec-derived plan is currently unimplemented in this worktree. This
## File Structure
- Create `packages/context/src/ingest/adapters/historic-sql/pattern-inputs.ts`
- Create `packages/context/src/ingest/adapters/historic-sql/pattern-inputs.ts`
Owns deterministic pattern audit ordering, cross-table candidate filtering, byte-bounded shard creation, shard path constants, and shard path detection.
- Create `packages/context/src/ingest/adapters/historic-sql/pattern-inputs.test.ts`
- Create `packages/context/src/ingest/adapters/historic-sql/pattern-inputs.test.ts`
Covers deterministic shard ordering, single-table exclusion from WorkUnit shards, byte limits, and oversize-template manifest warnings.
- Modify `packages/context/src/ingest/adapters/historic-sql/stage-unified.ts`
- Modify `packages/context/src/ingest/adapters/historic-sql/stage-unified.ts`
Writes full `patterns-input.json` plus bounded `patterns-input/part-0001.json` shard files, and appends shard warnings to `manifest.json`.
- Modify `packages/context/src/ingest/adapters/historic-sql/stage-unified.test.ts`
- Modify `packages/context/src/ingest/adapters/historic-sql/stage-unified.test.ts`
Adds a regression for audit file preservation and sharded WorkUnit input creation.
- Modify `packages/context/src/ingest/adapters/historic-sql/chunk-unified.ts`
- Modify `packages/context/src/ingest/adapters/historic-sql/chunk-unified.ts`
Emits one patterns WorkUnit per changed shard path, treats root `patterns-input.json` as audit-only, and includes shard paths in the scope descriptor and eviction calculation.
- Modify `packages/context/src/ingest/adapters/historic-sql/chunk-unified.test.ts`
- Modify `packages/context/src/ingest/adapters/historic-sql/chunk-unified.test.ts`
Updates root-file expectations and adds multi-shard diff behavior.
- Modify `packages/context/skills/historic_sql_patterns/SKILL.md`
- Modify `packages/context/skills/historic_sql_patterns/SKILL.md`
Tells the skill to read the exact pattern shard in `rawFiles` and emit evidence with that shard as `rawPath`.
- Modify `packages/context/src/ingest/adapters/historic-sql/local-ingest-acceptance.test.ts`
- Modify `packages/context/src/ingest/adapters/historic-sql/local-ingest-acceptance.test.ts`
Updates the fake agent to emit pattern evidence for `historic-sql-patterns-part-0001`.
- Modify `packages/context/src/ingest/ingest-runtime-assets.test.ts`
- Modify `packages/context/src/ingest/ingest-runtime-assets.test.ts`
Keeps packaged skill assertions aligned with sharded pattern file guidance.
## Task 1: Add Pattern Input Sharding Helper

View file

@ -55,16 +55,16 @@ Remaining spec gap this plan covers:
Create:
- `packages/context/src/ingest/adapters/historic-sql/redaction.ts`
- `packages/context/src/ingest/adapters/historic-sql/redaction.ts`
Owns compilation and application of historic-SQL SQL-text redaction patterns. Supports JavaScript regex strings and the documented `(?i)` case-insensitive prefix used by setup tests/docs.
- `packages/context/src/ingest/adapters/historic-sql/redaction.test.ts`
- `packages/context/src/ingest/adapters/historic-sql/redaction.test.ts`
Tests raw regex replacement, `(?i)` compatibility, empty config behavior, and invalid-pattern diagnostics.
Modify:
- `packages/context/src/ingest/adapters/historic-sql/stage-unified.ts`
- `packages/context/src/ingest/adapters/historic-sql/stage-unified.ts`
Compiles `config.redactionPatterns` once per fetch. Keeps original SQL for filtering and `SqlAnalysisPort.analyzeBatch()`, then stores redacted SQL in `ParsedTemplate.template.canonicalSql` before `toStagedTable()` and `toPatternsInput()` serialize files.
- `packages/context/src/ingest/adapters/historic-sql/stage-unified.test.ts`
- `packages/context/src/ingest/adapters/historic-sql/stage-unified.test.ts`
Adds a regression proving raw secrets are absent from staged artifacts while `analyzeBatch()` still receives the original SQL.
## Task 1: Add Historic SQL Redaction Helper
@ -89,7 +89,7 @@ describe('historic-SQL redaction', () => {
]);
const sql =
"select * from public.api_events where api_key = 'sk_live_abc123' and note = 'Secret_Token_9f'";
"select * from public.api_events where api_key = 'sk_live_abc123' and note = 'Secret_Token_9f'"; // pragma: allowlist secret
expect(redactHistoricSqlText(sql, redactors)).toBe(
"select * from public.api_events where api_key = '[REDACTED]' and note = '[REDACTED]'",
@ -202,7 +202,7 @@ Append this test inside the existing `describe('stageHistoricSqlAggregatedSnapsh
it('redacts configured SQL substrings in staged artifacts while analyzing original SQL', async () => {
const stagedDir = await tempDir();
const originalSql =
"select * from public.api_events where api_key = 'sk_live_abc123' and note = 'Secret_Token_9f'";
"select * from public.api_events where api_key = 'sk_live_abc123' and note = 'Secret_Token_9f'"; // pragma: allowlist secret
const reader: HistoricSqlReader = {
async probe() {
return { warnings: [], info: [] };

View file

@ -37,27 +37,27 @@ This plan does not rewrite the historic-SQL adapter, readers, skills, projection
Modify:
- `packages/context/src/sl/sl-search.service.ts`
- `packages/context/src/sl/sl-search.service.ts`
Adds usage narrative, frequency, filters, group-bys, joins, and stale marker to the canonical SL search text. Preserves snippets returned by repository search for direct `SlSearchService.search()` callers.
- `packages/context/src/sl/sl-search.service.test.ts`
- `packages/context/src/sl/sl-search.service.test.ts`
Tests usage search-text content and direct service snippet pass-through.
- `packages/context/src/sl/ports.ts`
- `packages/context/src/sl/ports.ts`
Extends `SlSourcesIndexPort.search()` rows with optional `snippet`.
- `packages/context/src/sl/sqlite-sl-sources-index.ts`
- `packages/context/src/sl/sqlite-sl-sources-index.ts`
Adds FTS5 `snippet()` selection to lexical candidate search and direct index search.
- `packages/context/src/sl/sqlite-sl-sources-index.test.ts`
- `packages/context/src/sl/sqlite-sl-sources-index.test.ts`
Locks snippet behavior for both direct search and lexical lane candidates.
- `packages/context/src/sl/local-sl.ts`
- `packages/context/src/sl/local-sl.ts`
Adds `frequencyTier` and `snippet` to query-mode `LocalSlSourceSearchResult`; collects snippets from the lexical lane and hydrates frequency from `SemanticLayerSource.usage`.
- `packages/context/src/sl/local-sl.test.ts`
- `packages/context/src/sl/local-sl.test.ts`
Tests that usage-only terms can find a source and that results include `frequencyTier` and FTS snippet.
- `packages/context/src/sl/pglite-sl-search-prototype.ts`
- `packages/context/src/sl/pglite-sl-search-prototype.ts`
Propagates `frequencyTier` for the prototype backend so the shared result type stays truthful.
- `packages/context/src/mcp/types.ts`
- `packages/context/src/mcp/types.ts`
Adds `frequencyTier` and `snippet` to `KtxSemanticLayerSourceSummary`.
- `packages/context/src/mcp/local-project-ports.ts`
- `packages/context/src/mcp/local-project-ports.ts`
Includes `frequencyTier` and `snippet` in `semanticLayer.listSources()` output.
- `packages/context/src/mcp/local-project-ports.test.ts`
- `packages/context/src/mcp/local-project-ports.test.ts`
Tests the agent/MCP-facing list response.
## Task 1: Index Historic SQL Usage In SL Search Text

View file

@ -52,58 +52,58 @@ Still not implemented:
Create:
- `packages/context/src/ingest/adapters/historic-sql/evidence.ts`
- `packages/context/src/ingest/adapters/historic-sql/evidence.ts`
Owns typed evidence envelopes, ignored evidence path helpers, and load/write helpers for table usage and pattern evidence.
- `packages/context/src/ingest/adapters/historic-sql/evidence.test.ts`
- `packages/context/src/ingest/adapters/historic-sql/evidence.test.ts`
Tests evidence schema validation, path normalization, and loader rejection of malformed evidence.
- `packages/context/src/ingest/adapters/historic-sql/evidence-tool.ts`
- `packages/context/src/ingest/adapters/historic-sql/evidence-tool.ts`
Adds `emit_historic_sql_evidence`, the only write tool the two new historic-SQL skills use.
- `packages/context/src/ingest/adapters/historic-sql/evidence-tool.test.ts`
- `packages/context/src/ingest/adapters/historic-sql/evidence-tool.test.ts`
Tests the tool writes ignored run-local JSON with `skipLock: true` and rejects non-historic ingest sessions.
- `packages/context/src/ingest/adapters/historic-sql/projection.ts`
- `packages/context/src/ingest/adapters/historic-sql/projection.ts`
Projects table usage evidence into manifest shards, writes pattern wiki pages, marks stale usage/pages, and deletes legacy query pages.
- `packages/context/src/ingest/adapters/historic-sql/projection.test.ts`
- `packages/context/src/ingest/adapters/historic-sql/projection.test.ts`
Tests `_schema` merge, stale usage, pattern slug reuse, stale page tagging, archive movement, and legacy page cleanup.
- `packages/context/src/ingest/adapters/historic-sql/post-processor.ts`
- `packages/context/src/ingest/adapters/historic-sql/post-processor.ts`
Implements `IngestBundlePostProcessorPort` for the deterministic projection phase.
- `packages/context/src/ingest/adapters/historic-sql/post-processor.test.ts`
- `packages/context/src/ingest/adapters/historic-sql/post-processor.test.ts`
Tests post-processor path resolution from `workdir`, `connectionId`, `sourceKey`, and `syncId`.
- `packages/context/skills/historic_sql_table_digest/SKILL.md`
- `packages/context/skills/historic_sql_table_digest/SKILL.md`
Skill for one changed `tables/*.json` WorkUnit; emits one table usage evidence object.
- `packages/context/skills/historic_sql_patterns/SKILL.md`
- `packages/context/skills/historic_sql_patterns/SKILL.md`
Skill for `patterns-input.json`; emits one pattern evidence object per recurring cross-table intent.
Modify:
- `packages/context/src/ingest/adapters/historic-sql/types.ts`
- `packages/context/src/ingest/adapters/historic-sql/types.ts`
Keep only unified config/staged schemas and reader contracts; extend config preprocessing for existing `serviceAccountUserPatterns` and `minCalls` aliases.
- `packages/context/src/ingest/adapters/historic-sql/stage-unified.ts`
- `packages/context/src/ingest/adapters/historic-sql/stage-unified.ts`
Add `staleArchiveAfterDays` to `manifest.json` so projection can archive stale pattern pages deterministically.
- `packages/context/src/ingest/adapters/historic-sql/chunk-unified.ts`
- `packages/context/src/ingest/adapters/historic-sql/chunk-unified.ts`
Keep the same WorkUnits, but mention `emit_historic_sql_evidence` in `notes`.
- `packages/context/src/ingest/adapters/historic-sql/historic-sql.adapter.ts`
- `packages/context/src/ingest/adapters/historic-sql/historic-sql.adapter.ts`
Switch production fetch/chunk/scope to the unified hot path, replace skills, remove legacy triage support, and run legacy PGSS baseline cache cleanup.
- `packages/context/src/ingest/adapters/historic-sql/historic-sql.adapter.test.ts`
- `packages/context/src/ingest/adapters/historic-sql/historic-sql.adapter.test.ts`
Rewrite around unified staging and new skills.
- `packages/context/src/ingest/adapters/historic-sql/postgres-pgss-reader.ts`
- `packages/context/src/ingest/adapters/historic-sql/postgres-pgss-reader.ts`
Inline the PGSS probe logic so `postgres-pgss-query-history-reader.ts` can be deleted.
- `packages/context/src/ingest/local-adapters.ts`
- `packages/context/src/ingest/local-adapters.ts`
Use `PostgresPgssReader` for local Postgres historic SQL and return unified pull config.
- `packages/context/src/ingest/local-bundle-runtime.ts`
- `packages/context/src/ingest/local-bundle-runtime.ts`
Add the source-specific evidence tool to historic-SQL WorkUnits and register the historic-SQL post-processor.
- `packages/context/src/ingest/ingest-runtime-assets.test.ts`
- `packages/context/src/ingest/ingest-runtime-assets.test.ts`
Replace old skill asset assertions with the two new skills.
- `packages/context/src/memory/memory-runtime-assets.test.ts`
- `packages/context/src/memory/memory-runtime-assets.test.ts`
Replace old historic-SQL skill heading with the two new skill headings.
- `packages/context/src/package-exports.test.ts`
- `packages/context/src/package-exports.test.ts`
Remove legacy export assertions and add evidence/projection export assertions.
- `packages/context/src/ingest/index.ts`
- `packages/context/src/ingest/index.ts`
Export new evidence/projection/post-processor helpers and remove legacy historic-SQL exports.
- `packages/cli/src/setup-databases.ts` and `packages/cli/src/historic-sql-doctor.ts`
- `packages/cli/src/setup-databases.ts` and `packages/cli/src/historic-sql-doctor.ts`
Import `PostgresPgssReader` instead of `PostgresPgssQueryHistoryReader`.
- `packages/cli/src/commands/setup-commands.ts`, `packages/cli/src/index.test.ts`, `packages/cli/src/setup-databases.test.ts`
- `packages/cli/src/commands/setup-commands.ts`, `packages/cli/src/index.test.ts`, `packages/cli/src/setup-databases.test.ts`
Rename generated config to `minExecutions` while accepting the old `--historic-sql-min-calls` flag for one release.
- `packages/context/prompts/skills/page_triage_classifier.md`, `packages/context/src/ingest/page-triage/page-triage.service.test.ts`, `packages/context/src/ingest/ingest-prompts.test.ts`
- `packages/context/prompts/skills/page_triage_classifier.md`, `packages/context/src/ingest/page-triage/page-triage.service.test.ts`, `packages/context/src/ingest/ingest-prompts.test.ts`
Remove historic-SQL template triage examples because the new adapter no longer uses page triage.
Delete: