mirror of
https://github.com/Kaelio/ktx.git
synced 2026-06-07 07:55:13 +02:00
2366b00301
* refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
129 lines
4.2 KiB
Markdown
129 lines
4.2 KiB
Markdown
# Postgres Query History Example
|
|
|
|
This example is a manual smoke for Postgres query-history ingest through
|
|
`pg_stat_statements`. It starts Postgres 14 with the extension preloaded,
|
|
generates query workload under separate users, runs `ktx setup` with
|
|
`--enable-query-history`, and verifies the staged query-history artifacts:
|
|
|
|
- `manifest.json`
|
|
- `tables/*.json`
|
|
- `patterns-input.json` as the full audit input
|
|
- `patterns-input/part-*.json` as bounded pattern WorkUnit shards
|
|
|
|
The smoke also runs the same workload twice and verifies the second stage-only
|
|
run has `workUnitCount: 0`, which proves unchanged bucketed table inputs and
|
|
unchanged bounded pattern shards do not schedule LLM work.
|
|
|
|
## Prerequisites
|
|
|
|
- Docker with Compose v2
|
|
- Node and pnpm matching the KTX workspace
|
|
- `uv` on `PATH` so the KTX-managed Python runtime can install the bundled
|
|
runtime wheel
|
|
|
|
## Run
|
|
|
|
From the KTX repository root:
|
|
|
|
```bash
|
|
examples/postgres-historic/scripts/smoke.sh
|
|
```
|
|
|
|
The smoke creates a temporary KTX project, isolates the managed Python runtime
|
|
under the temporary project parent, starts Postgres on `127.0.0.1:55432`, and
|
|
uses this connection URL:
|
|
|
|
```bash
|
|
postgresql://ktx_reader:ktx_reader@127.0.0.1:55432/analytics # pragma: allowlist secret
|
|
```
|
|
|
|
Set `KTX_POSTGRES_HISTORIC_KEEP_DOCKER=1` to leave the container running after
|
|
the script exits.
|
|
|
|
The smoke validates the query-history raw snapshot path without requiring LLM
|
|
credentials. It uses KTX's local stage-only ingest API after `ktx setup`, so the
|
|
deterministic reader, batch SQL parser, stable artifact writer, and diff-based
|
|
WorkUnit planning are checked independently from curation.
|
|
|
|
## Manual Commands
|
|
|
|
Start Postgres and generate the base workload:
|
|
|
|
```bash
|
|
docker compose -f examples/postgres-historic/docker-compose.yml up -d --wait
|
|
examples/postgres-historic/scripts/generate-workload.sh base
|
|
```
|
|
|
|
Create a project and enable query history:
|
|
|
|
```bash
|
|
export WAREHOUSE_DATABASE_URL=postgresql://ktx_reader:ktx_reader@127.0.0.1:55432/analytics # pragma: allowlist secret
|
|
pnpm --filter @kaelio/ktx run build
|
|
node packages/cli/dist/bin.js --project-dir /tmp/ktx-postgres-historic setup \
|
|
--new \
|
|
--skip-agents \
|
|
--skip-llm \
|
|
--skip-embeddings \
|
|
--skip-sources \
|
|
--database postgres \
|
|
--new-database-connection-id warehouse \
|
|
--database-url env:WAREHOUSE_DATABASE_URL \
|
|
--database-schema public \
|
|
--enable-query-history \
|
|
--query-history-min-executions 2 \
|
|
--yes \
|
|
--no-input
|
|
```
|
|
|
|
### Readiness check
|
|
|
|
```bash
|
|
pnpm run ktx -- status --project-dir /tmp/ktx-postgres-historic --no-input
|
|
```
|
|
|
|
The installed CLI form is:
|
|
|
|
```bash
|
|
ktx status --project-dir /tmp/ktx-postgres-historic --no-input
|
|
```
|
|
|
|
Expected output includes `PASS Postgres query history (warehouse)` when
|
|
`pg_stat_statements` is installed, `pg_read_all_stats` is granted, and tracking
|
|
is enabled. A low `pg_stat_statements.max` value is reported as an informational
|
|
note, not a warning.
|
|
|
|
Run query-history ingest:
|
|
|
|
```bash
|
|
pnpm run ktx -- ingest warehouse --project-dir /tmp/ktx-postgres-historic \
|
|
--query-history \
|
|
--no-input
|
|
```
|
|
|
|
The public query-history ingest path also runs curation WorkUnits, so it
|
|
requires a configured LLM provider.
|
|
|
|
Inspect the latest manifest:
|
|
|
|
```bash
|
|
find /tmp/ktx-postgres-historic/raw-sources/warehouse/historic-sql -name manifest.json | sort | tail -n 1
|
|
```
|
|
|
|
The manifest should have `source: "historic-sql"`, `dialect: "postgres"`,
|
|
positive `snapshotRowCount`, positive `touchedTableCount`, numeric
|
|
`parseFailures`, `warnings`, and `probeWarnings`. The same directory should
|
|
contain `patterns-input.json`, at least one `patterns-input/part-*.json` pattern
|
|
shard for cross-table candidates, and one `tables/*.json` file per touched
|
|
table.
|
|
|
|
## Troubleshooting
|
|
|
|
- Missing extension: confirm `shared_preload_libraries=pg_stat_statements` and
|
|
`CREATE EXTENSION pg_stat_statements;` both happened in the `analytics`
|
|
database.
|
|
- Missing grants: confirm `GRANT pg_read_all_stats TO ktx_reader;`.
|
|
- Empty snapshot: rerun `scripts/generate-workload.sh base` and keep
|
|
`--query-history-min-executions 2` for the smoke.
|
|
- SQL-analysis failures: run `pnpm run ktx -- dev runtime status` from the KTX
|
|
repository root and confirm `uv`, the bundled Python wheel, and the managed
|
|
runtime all pass.
|