mirror of
https://github.com/Kaelio/ktx.git
synced 2026-06-07 07:55:13 +02:00
9dad936ac7
* docs: add npm managed python runtime design * build: add bundled python runtime wheel builder * build: make local embedding dependencies optional * build: bundle python runtime wheel in cli artifacts * build: track bundled python runtime release artifact * test: verify bundled python runtime wheel * docs: add plan for bundled python runtime wheel * test: cover managed python runtime lifecycle * feat: add managed python runtime installer * feat: add runtime command runner * feat: expose runtime management commands * test: verify managed python runtime commands * docs: add plan for managed python runtime installer * feat: add managed python command helper * feat: use managed runtime for sl query compute * feat: route sl query managed runtime policy * docs: add plan for managed runtime sl query integration * feat: add managed runtime daemon metadata * feat: manage python daemon lifecycle * feat: add runtime daemon start stop commands * fix: verify managed runtime daemon lifecycle * docs: add plan for managed runtime daemon lifecycle * feat: add managed local embeddings config marker * feat: add managed local embeddings daemon helper * feat: use managed runtime for local embedding setup * feat: pass managed runtime policy through setup * docs: add plan for managed local embeddings runtime * feat: read CLI package metadata dynamically * feat: assemble public kaelio ktx npm package * feat: release one public kaelio ktx npm artifact * test: cover public kaelio ktx package invocations * chore: verify public kaelio ktx package artifacts * docs: add plan for public kaelio ktx npm package * test: verify managed runtime in public package smoke * test: finalize managed runtime release smoke * docs: add plan for managed runtime release smoke * test: specify local embeddings release smoke * feat: add local embeddings runtime smoke * chore: register local embeddings smoke * fix: verify local embeddings smoke * fix: restore artifact smoke python env helper * docs: add plan for managed local embeddings release smoke * refactor: share managed runtime install policy parsing * feat: use managed runtime for agent semantic queries * feat: use managed runtime for MCP semantic compute * docs: add plan for managed agent and MCP semantic runtime * feat(cli): add managed daemon HTTP helpers * feat(cli): route local adapters through managed daemon * feat(cli): use managed daemon for ingest helpers * feat(cli): pass managed daemon options to scan * feat(context): pass MCP ingest pull config options * feat(cli): pass managed daemon options to serve ingest * test: verify managed local ingest daemon runtime * docs: add plan for managed local ingest daemon runtime * docs: align managed runtime examples * docs: add plan for managed runtime docs cleanup * test: cover published package runtime smoke commands * test: validate published package smoke outputs * docs: add plan for published package runtime smoke * build: stamp public npm package version * release: add npm public release policy * release: add guarded npm publish script * release: document public npm release handoff * docs: add plan for public npm release handoff * test: cover managed runtime prune in package smoke * docs: document managed runtime prune * docs: add plan for managed runtime prune smoke and docs * chore: encode uv runtime prerequisite policy * fix: clarify missing uv runtime error * docs: document uv runtime prerequisite * docs: add plan for uv runtime prerequisite contract * refactor: limit release artifacts to public package runtime * chore: align release policy with bundled runtime wheel * docs: describe single public runtime artifact surface * test: verify single public runtime artifact contract * docs: add plan for single public runtime artifact cleanup * fix: align local embeddings smoke with public version * docs: add plan for local embeddings smoke public version * release: soft-launch as @kaelio/ktx@0.1.0-rc.0 on next tag Publish target moves to the pre-release version 0.1.0-rc.0 under the next dist-tag so npm install @kaelio/ktx (which resolves to latest) does not pick up the soft-launch build. Users opt in via @kaelio/ktx@next. * Fix release script boundary checks * Remove PostHog from public package bundle
118 lines
3.7 KiB
Markdown
118 lines
3.7 KiB
Markdown
# Postgres Historic SQL Example
|
|
|
|
This example is a manual smoke for Postgres historic-SQL ingest through
|
|
`pg_stat_statements`. It starts Postgres 14 with the extension preloaded,
|
|
generates query workload under separate users, runs `ktx setup` with
|
|
`--enable-historic-sql`, and verifies three local ingest runs:
|
|
|
|
- first run creates a fresh PGSS baseline
|
|
- second run emits only positive deltas
|
|
- reset run treats `pg_stat_statements_reset()` as a fresh baseline
|
|
|
|
## Prerequisites
|
|
|
|
- Docker with Compose v2
|
|
- Node and pnpm matching the KTX workspace
|
|
- `uv` on `PATH` so the KTX-managed Python runtime can install the bundled
|
|
runtime wheel
|
|
|
|
## Run
|
|
|
|
From the KTX repository root:
|
|
|
|
```bash
|
|
examples/postgres-historic/scripts/smoke.sh
|
|
```
|
|
|
|
The smoke creates a temporary KTX project, isolates the managed Python runtime
|
|
under the temporary project parent, starts Postgres on `127.0.0.1:55432`, and
|
|
uses this connection URL:
|
|
|
|
```bash
|
|
postgresql://ktx_reader:ktx_reader@127.0.0.1:55432/analytics # pragma: allowlist secret
|
|
```
|
|
|
|
Set `KTX_POSTGRES_HISTORIC_KEEP_DOCKER=1` to leave the container running after
|
|
the script exits.
|
|
|
|
The smoke validates the historic-SQL raw snapshot path without requiring LLM
|
|
credentials. It uses KTX's local stage-only ingest API after `ktx setup` so the
|
|
PGSS baseline and delta behavior can be checked independently from curation.
|
|
|
|
## Manual Commands
|
|
|
|
Start Postgres and generate the base workload:
|
|
|
|
```bash
|
|
docker compose -f examples/postgres-historic/docker-compose.yml up -d --wait
|
|
examples/postgres-historic/scripts/generate-workload.sh base
|
|
```
|
|
|
|
Create a project and enable historic SQL:
|
|
|
|
```bash
|
|
export WAREHOUSE_DATABASE_URL=postgresql://ktx_reader:ktx_reader@127.0.0.1:55432/analytics # pragma: allowlist secret
|
|
pnpm --filter @ktx/cli run build
|
|
node packages/cli/dist/bin.js --project-dir /tmp/ktx-postgres-historic setup \
|
|
--new \
|
|
--skip-agents \
|
|
--skip-llm \
|
|
--skip-embeddings \
|
|
--skip-sources \
|
|
--database postgres \
|
|
--new-database-connection-id warehouse \
|
|
--database-url env:WAREHOUSE_DATABASE_URL \
|
|
--database-schema public \
|
|
--enable-historic-sql \
|
|
--historic-sql-min-calls 2 \
|
|
--yes \
|
|
--no-input
|
|
```
|
|
|
|
### Readiness check
|
|
|
|
```bash
|
|
pnpm run ktx -- dev doctor --project-dir /tmp/ktx-postgres-historic --no-input
|
|
```
|
|
|
|
The installed CLI form is `ktx dev doctor --project-dir
|
|
/tmp/ktx-postgres-historic --no-input`. Expected output includes `PASS Postgres
|
|
Historic SQL (warehouse)` when `pg_stat_statements` is installed,
|
|
`pg_read_all_stats` is granted, tracking is enabled, and
|
|
`pg_stat_statements.max` is at least 5000.
|
|
|
|
Run local historic-SQL ingest:
|
|
|
|
```bash
|
|
pnpm run ktx -- dev ingest run --project-dir /tmp/ktx-postgres-historic \
|
|
--connection-id warehouse \
|
|
--adapter historic-sql \
|
|
--plain \
|
|
--yes \
|
|
--no-input
|
|
```
|
|
|
|
The full `dev ingest run` path also runs curation work units, so it requires a
|
|
configured LLM provider.
|
|
|
|
Inspect the latest manifest:
|
|
|
|
```bash
|
|
find /tmp/ktx-postgres-historic/raw-sources/warehouse/historic-sql -name manifest.json | sort | tail -n 1
|
|
```
|
|
|
|
The manifest should have `dialect: "postgres"`, `degraded: true`,
|
|
`baselineFirstRun: true` on the first run, and populated `pgServerVersion` and
|
|
`statsResetAt`.
|
|
|
|
## Troubleshooting
|
|
|
|
- Missing extension: confirm `shared_preload_libraries=pg_stat_statements` and
|
|
`CREATE EXTENSION pg_stat_statements;` both happened in the `analytics`
|
|
database.
|
|
- Missing grants: confirm `GRANT pg_read_all_stats TO ktx_reader;`.
|
|
- Empty templates: rerun `scripts/generate-workload.sh base` and keep
|
|
`--historic-sql-min-calls 2` for the smoke.
|
|
- SQL-analysis failures: run `pnpm run ktx -- runtime doctor` from the KTX
|
|
repository root and confirm `uv`, the bundled Python wheel, and the managed
|
|
runtime all pass.
|