apunkt/ktx
1
0
Fork 0
mirror of https://github.com/Kaelio/ktx.git synced 2026-06-25 08:48:08 +02:00
Code Issues Projects Releases Packages Wiki Activity

rename klo to ktx

Browse source
This commit is contained in:
Andrey Avtomonov 2026-05-10 23:51:24 +02:00
parent 1a42152e6f
commit 3ce510b55b
704 changed files with 10205 additions and 10255 deletions
Download patch file Download diff file Expand all files Collapse all files

36
examples/postgres-historic/README.md
View file

@ -2,7 +2,7 @@
This example is a manual smoke for Postgres historic-SQL ingest through
`pg_stat_statements`. It starts Postgres 14 with the extension preloaded,
generates query workload under separate users, runs `klo setup` with
generates query workload under separate users, runs `ktx setup` with
`--enable-historic-sql`, and verifies three local ingest runs:
- first run creates a fresh PGSS baseline
@ -12,30 +12,30 @@ generates query workload under separate users, runs `klo setup` with
## Prerequisites
- Docker with Compose v2
- Node and pnpm matching the KLO workspace
- `python-service/.venv` already created, or `KLO_SQL_ANALYSIS_URL` pointing at
- Node and pnpm matching the KTX workspace
- `python-service/.venv` already created, or `KTX_SQL_ANALYSIS_URL` pointing at
a running service that exposes `/api/sql/analyze-for-fingerprint`
## Run
From the KLO repository root:
From the KTX repository root:
```bash
examples/postgres-historic/scripts/smoke.sh
```
The smoke creates a temporary KLO project, starts Postgres on
The smoke creates a temporary KTX project, starts Postgres on
`127.0.0.1:55432`, and uses this connection URL:
```bash
postgresql://klo_reader:klo_reader@127.0.0.1:55432/analytics # pragma: allowlist secret
postgresql://ktx_reader:ktx_reader@127.0.0.1:55432/analytics # pragma: allowlist secret
```
Set `KLO_POSTGRES_HISTORIC_KEEP_DOCKER=1` to leave the container running after
Set `KTX_POSTGRES_HISTORIC_KEEP_DOCKER=1` to leave the container running after
the script exits.
The smoke validates the historic-SQL raw snapshot path without requiring LLM
credentials. It uses KLO's local stage-only ingest API after `klo setup` so the
credentials. It uses KTX's local stage-only ingest API after `ktx setup` so the
PGSS baseline and delta behavior can be checked independently from curation.
## Manual Commands
@ -50,9 +50,9 @@ examples/postgres-historic/scripts/generate-workload.sh base
Create a project and enable historic SQL:
```bash
export WAREHOUSE_DATABASE_URL=postgresql://klo_reader:klo_reader@127.0.0.1:55432/analytics # pragma: allowlist secret
pnpm --filter @klo/cli run build
node packages/cli/dist/bin.js --project-dir /tmp/klo-postgres-historic setup \
export WAREHOUSE_DATABASE_URL=postgresql://ktx_reader:ktx_reader@127.0.0.1:55432/analytics # pragma: allowlist secret
pnpm --filter @ktx/cli run build
node packages/cli/dist/bin.js --project-dir /tmp/ktx-postgres-historic setup \
--new \
--skip-agents \
--skip-llm \
@ -71,11 +71,11 @@ node packages/cli/dist/bin.js --project-dir /tmp/klo-postgres-historic setup \
### Readiness check
```bash
pnpm run klo -- dev doctor --project-dir /tmp/klo-postgres-historic --no-input
pnpm run ktx -- dev doctor --project-dir /tmp/ktx-postgres-historic --no-input
```
The installed CLI form is `klo dev doctor --project-dir
/tmp/klo-postgres-historic --no-input`. Expected output includes `PASS Postgres
The installed CLI form is `ktx dev doctor --project-dir
/tmp/ktx-postgres-historic --no-input`. Expected output includes `PASS Postgres
Historic SQL (warehouse)` when `pg_stat_statements` is installed,
`pg_read_all_stats` is granted, tracking is enabled, and
`pg_stat_statements.max` is at least 5000.
@ -83,7 +83,7 @@ Historic SQL (warehouse)` when `pg_stat_statements` is installed,
Run local historic-SQL ingest:
```bash
node packages/cli/dist/bin.js --project-dir /tmp/klo-postgres-historic dev ingest run \
node packages/cli/dist/bin.js --project-dir /tmp/ktx-postgres-historic dev ingest run \
--connection-id warehouse \
--adapter historic-sql \
--plain \
@ -96,7 +96,7 @@ configured LLM provider.
Inspect the latest manifest:
```bash
find /tmp/klo-postgres-historic/raw-sources/warehouse/historic-sql -name manifest.json | sort | tail -n 1
find /tmp/ktx-postgres-historic/raw-sources/warehouse/historic-sql -name manifest.json | sort | tail -n 1
```
The manifest should have `dialect: "postgres"`, `degraded: true`,
@ -108,8 +108,8 @@ The manifest should have `dialect: "postgres"`, `degraded: true`,
- Missing extension: confirm `shared_preload_libraries=pg_stat_statements` and
`CREATE EXTENSION pg_stat_statements;` both happened in the `analytics`
database.
- Missing grants: confirm `GRANT pg_read_all_stats TO klo_reader;`.
- Missing grants: confirm `GRANT pg_read_all_stats TO ktx_reader;`.
- Empty templates: rerun `scripts/generate-workload.sh base` and keep
`--historic-sql-min-calls 2` for the smoke.
- SQL-analysis failures: set `KLO_SQL_ANALYSIS_URL` to the running service URL
- SQL-analysis failures: set `KTX_SQL_ANALYSIS_URL` to the running service URL
or create `python-service/.venv` before running `scripts/smoke.sh`.

8
examples/postgres-historic/init/001-schema.sql
View file

@ -2,9 +2,9 @@ CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
CREATE ROLE app_user LOGIN PASSWORD 'app_pass';
CREATE ROLE etl_user LOGIN PASSWORD 'etl_pass';
CREATE ROLE klo_reader LOGIN PASSWORD 'klo_reader';
CREATE ROLE ktx_reader LOGIN PASSWORD 'ktx_reader';
GRANT pg_read_all_stats TO klo_reader;
GRANT pg_read_all_stats TO ktx_reader;
CREATE TABLE customers (
id integer PRIMARY KEY,
@ -47,5 +47,5 @@ INSERT INTO events (id, customer_id, event_name, occurred_at) VALUES
(4, 3, 'sync_completed', now() - interval '6 hours'),
(5, 4, 'dashboard_viewed', now() - interval '5 hours');
GRANT USAGE ON SCHEMA public TO app_user, etl_user, klo_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO app_user, etl_user, klo_reader;
GRANT USAGE ON SCHEMA public TO app_user, etl_user, ktx_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO app_user, etl_user, ktx_reader;

46
examples/postgres-historic/scripts/smoke.sh
View file

@ -3,12 +3,12 @@ set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
EXAMPLE_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
KLO_ROOT="$(cd "$EXAMPLE_DIR/../.." && pwd)"
REPO_ROOT="$(cd "$KLO_ROOT/.." && pwd)"
KTX_ROOT="$(cd "$EXAMPLE_DIR/../.." && pwd)"
REPO_ROOT="$(cd "$KTX_ROOT/.." && pwd)"
COMPOSE_FILE="$EXAMPLE_DIR/docker-compose.yml"
PROJECT_PARENT="${KLO_POSTGRES_HISTORIC_PROJECT_PARENT:-$(mktemp -d)}"
PROJECT_DIR="$PROJECT_PARENT/postgres-historic-klo"
KLO_BIN="$KLO_ROOT/packages/cli/dist/bin.js"
PROJECT_PARENT="${KTX_POSTGRES_HISTORIC_PROJECT_PARENT:-$(mktemp -d)}"
PROJECT_DIR="$PROJECT_PARENT/postgres-historic-ktx"
KTX_BIN="$KTX_ROOT/packages/cli/dist/bin.js"
PYTHON_SERVICE_LOG="$PROJECT_PARENT/python-service.log"
PYTHON_SERVICE_PID=""
@ -16,18 +16,18 @@ cleanup() {
if [[ -n "$PYTHON_SERVICE_PID" ]]; then
kill "$PYTHON_SERVICE_PID" >/dev/null 2>&1 || true
fi
if [[ "${KLO_POSTGRES_HISTORIC_KEEP_DOCKER:-0}" != "1" ]]; then
if [[ "${KTX_POSTGRES_HISTORIC_KEEP_DOCKER:-0}" != "1" ]]; then
docker compose -f "$COMPOSE_FILE" down -v >/dev/null 2>&1 || true
fi
}
trap cleanup EXIT
start_sql_analysis_if_needed() {
if [[ -n "${KLO_SQL_ANALYSIS_URL:-}" ]]; then
if [[ -n "${KTX_SQL_ANALYSIS_URL:-}" ]]; then
return
fi
if [[ ! -d "$REPO_ROOT/python-service/.venv" ]]; then
echo "Set KLO_SQL_ANALYSIS_URL or create python-service/.venv before running this smoke." >&2
echo "Set KTX_SQL_ANALYSIS_URL or create python-service/.venv before running this smoke." >&2
exit 1
fi
(
@ -36,9 +36,9 @@ start_sql_analysis_if_needed() {
uvicorn app.main:app --host 127.0.0.1 --port 18081 >"$PYTHON_SERVICE_LOG" 2>&1
) &
PYTHON_SERVICE_PID="$!"
export KLO_SQL_ANALYSIS_URL="http://127.0.0.1:18081"
export KTX_SQL_ANALYSIS_URL="http://127.0.0.1:18081"
for _ in $(seq 1 60); do
if curl -fsS "$KLO_SQL_ANALYSIS_URL/health" >/dev/null 2>&1; then
if curl -fsS "$KTX_SQL_ANALYSIS_URL/health" >/dev/null 2>&1; then
return
fi
sleep 1
@ -74,18 +74,18 @@ NODE
run_historic_stage_only() {
local job_id="$1"
node - "$KLO_ROOT" "$PROJECT_DIR" "$job_id" <<'NODE'
node - "$KTX_ROOT" "$PROJECT_DIR" "$job_id" <<'NODE'
const { join } = await import('node:path');
const kloRoot = process.argv[2];
const ktxRoot = process.argv[2];
const projectDir = process.argv[3];
const jobId = process.argv[4];
const { loadKloProject } = await import(join(kloRoot, 'packages/context/dist/project/index.js'));
const { runLocalStageOnlyIngest } = await import(join(kloRoot, 'packages/context/dist/ingest/index.js'));
const { createKloCliLocalIngestAdapters } = await import(join(kloRoot, 'packages/cli/dist/local-adapters.js'));
const { loadKtxProject } = await import(join(ktxRoot, 'packages/context/dist/project/index.js'));
const { runLocalStageOnlyIngest } = await import(join(ktxRoot, 'packages/context/dist/ingest/index.js'));
const { createKtxCliLocalIngestAdapters } = await import(join(ktxRoot, 'packages/cli/dist/local-adapters.js'));
const project = await loadKloProject({ projectDir });
const adapters = createKloCliLocalIngestAdapters(project, { historicSqlConnectionId: 'warehouse' });
const project = await loadKtxProject({ projectDir });
const adapters = createKtxCliLocalIngestAdapters(project, { historicSqlConnectionId: 'warehouse' });
const adapter = adapters.find((candidate) => candidate.source === 'historic-sql');
if (!adapter) throw new Error('historic-sql adapter was not registered for local run');
const record = await runLocalStageOnlyIngest({
@ -102,22 +102,22 @@ await adapter.onPullSucceeded?.({
syncId: record.syncId,
trigger: 'manual_resync',
completedAt: new Date(record.completedAt),
stagedDir: join(project.projectDir, '.klo/cache/local-ingest', jobId, 'staged'),
stagedDir: join(project.projectDir, '.ktx/cache/local-ingest', jobId, 'staged'),
});
console.log(record.syncId);
NODE
}
cd "$KLO_ROOT"
pnpm --filter @klo/context run build
pnpm --filter @klo/cli run build
cd "$KTX_ROOT"
pnpm --filter @ktx/context run build
pnpm --filter @ktx/cli run build
start_sql_analysis_if_needed
docker compose -f "$COMPOSE_FILE" up -d --wait
"$EXAMPLE_DIR/scripts/generate-workload.sh" base
export WAREHOUSE_DATABASE_URL="${WAREHOUSE_DATABASE_URL:-postgresql://klo_reader:klo_reader@127.0.0.1:55432/analytics}" # pragma: allowlist secret
node "$KLO_BIN" --project-dir "$PROJECT_DIR" setup \
export WAREHOUSE_DATABASE_URL="${WAREHOUSE_DATABASE_URL:-postgresql://ktx_reader:ktx_reader@127.0.0.1:55432/analytics}" # pragma: allowlist secret
node "$KTX_BIN" --project-dir "$PROJECT_DIR" setup \
--new \
--skip-agents \
--skip-llm \