chore(workspace): gate dead-code with knip production mode (#196)

* refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm

* refactor(workspace): rewrite @ktx/llm imports to relative paths

* refactor(workspace): fold internal packages into cli

* chore(workspace): gate dead-code with knip production mode

Turn on production-mode knip plus an autofix run in pre-commit and the
`pnpm dead-code` script, document the `/** @internal */` convention for
test-only exports in AGENTS.md, annotate test-only exports across the
CLI with that JSDoc, and drop dead exports/wrappers the new gate
surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`,
`createLocalScanEnrichmentProvidersFromConfig`,
`PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports).
Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit
production entries so cross-package barrel leaks are caught.

* refactor(cli): delete internal barrel index.ts files

The 34 `index.ts` re-export barrels inside `packages/cli/src/` were
holdovers from the pre-fold multi-workspace structure. Post-fold-in they
served no production purpose: external consumers go through the single
package main entry, and in-repo callers mostly imported through them
only because the path was short. Internally, knip flagged most barrel
re-exports as production-dead (only reached via tests).

This change:
- Deletes every internal barrel except `packages/cli/src/index.ts`
  (the published package entry).
- Rewrites ~270 source/test files to import each name directly from
  the file that defines it.
- Moves `tools/warehouse-verification/index.ts` to
  `create-warehouse-verification-tools.ts` (the function it defined
  locally) and updates its single consumer.
- Renames `search/backend-conformance.ts` → `.test-utils.ts` to match
  the existing test-helper file convention.
- Deletes 13 dead test-only chains (dbt-descriptions/*,
  live-database/extracted-schema, live-database/structural-sync,
  relationship-* feedback/review chain) plus their tests and a
  cascading orphan integration test.
- Updates test mocks that pointed at deleted barrel paths
  (notion-client, connector barrels in scan/local-scan-connectors
  tests) to mock the source files instead.
- Points the maintainer benchmark script
  (`scripts/relationship-benchmark-report.mjs`) at source files
  instead of `dist/context/scan/index.js`.
- Drops the barrel `!` entries from `knip.json`; adds explicit
  production entries only for the benchmark code reached via dist by
  the maintainer script.

Net: 413 files changed, ~1.2k insertions, ~9.4k deletions.

`pnpm run dead-code` (Biome + knip default + knip production) and
`pnpm run type-check` are clean; 2277 tests pass.

* refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly

Promote the CLI workspace package to the public name `@kaelio/ktx` and
drop the separate `scripts/build-public-npm-package.mjs` wrapper. The
CLI package is now publishable in place (`publishConfig.access: public`,
`provenance: true`), so artifact packing uses `pnpm pack` against
`packages/cli/` instead of assembling a parallel package tree.

Updates all workspace filter invocations, docs, tests, and release
readiness checks to reference the new package name, and folds the
tarball-name helper into `scripts/public-npm-release-metadata.mjs`.

* docs: align "agent clients" and "data agents" terminology

Replace "client agents" with "agent clients" and "database agents" with
"data agents" across AGENTS.md, README.md, the docs-site copy, and the
matching setup-agents test description, matching the canonical
vocabulary in docs/terminology.md.

Also moves packages/cli/tsconfig.json's tsBuildInfoFile from
node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive
node_modules reinstalls.

* refactor(release): single source of truth for package version

Make packages/cli/package.json the single source of truth for the
@kaelio/ktx version. publicNpmPackageVersion() now reads it directly,
so artifact filenames, release-readiness checks, and the Python wheel
version all derive from one field. The duplicate
release-policy.json.publicNpmPackageVersion is removed.

Previously the two fields could drift: tarballs were named
kaelio-ktx-0.4.1.tgz while internally containing
@kaelio/ktx@0.0.0-private.

- update-public-release-version.mjs rewrites both Python pyproject.toml
  files (ktx-daemon, ktx-sl) alongside the npm package.jsons,
  normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2).
- semantic-release-config.cjs adds the two pyproject.toml files to
  @semantic-release/git assets so the release commit back to main
  carries every version source in lockstep.
- The six "?? '0.0.0-private'" fallback literals across the CLI are
  replaced with "?? getKtxCliPackageInfo().version", and
  createDefaultKtxMcpServer makes its version arg required.
- docs/release.md describes the actual commit-back model: the dev tree
  always reflects the most recent release; no sentinel pin to
  maintain.

Verified: pnpm run artifacts:build now produces
kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with
@kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and
2287 vitests + 173 script tests pass.

* refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime

Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and
scan command entrypoints so tests can stub them, and teach
resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime
feature when ktx.yaml selects sentence-transformers.

* chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal

Both symbols are consumed only by status-project.test.ts. Annotating with
/** @internal */ keeps knip's production-mode check clean without changing
runtime behavior.

* fix(cli): use real package metadata in print-command-tree

The stubbed package name embedded a forbidden product identifier that
tripped the boundary check in CI. Read the metadata from package.json
instead — keeps the rendered tree unchanged and removes a duplicate
source of truth.

* feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts

Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer
source counts, computed with `SUM(embedding_json IS NOT NULL)` over
`knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to
"Wiki" (canonical per `docs/terminology.md`) and rename the matching
`localStats.knowledgePages` field to `localStats.wikiPages`.

Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those
duplicated the per-surface rows above. Disk now reports only actual byte
usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` /
`semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry`
helpers, and the `filter` arg on `summarizeDir` are removed.
This commit is contained in:
Andrey Avtomonov 2026-05-21 15:28:58 +02:00 committed by GitHub
parent a1cfb03d73
commit 2366b00301
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1002 changed files with 2286 additions and 12051 deletions

View file

@ -83,36 +83,8 @@ jobs:
- name: Run TypeScript checks
run: pnpm run check
slow-context-tests:
name: Slow context tests
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Setup pnpm
uses: pnpm/action-setup@0e279bb959325dab635dd2c09392533439d90093 # v6.0.8
with:
run_install: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: "24"
cache: "pnpm"
cache-dependency-path: "pnpm-lock.yaml"
- name: Install TypeScript dependencies
run: pnpm install --frozen-lockfile
- name: Build TypeScript packages
run: pnpm run build
- name: Run slow context tests
run: pnpm --filter @ktx/context run test:slow
slow-cli-tests:
name: Slow CLI tests
name: Slow TypeScript tests
runs-on: ubuntu-latest
steps:
- name: Checkout repository
@ -137,7 +109,7 @@ jobs:
run: pnpm run build
- name: Run slow CLI tests
run: pnpm --filter @ktx/cli run test:slow
run: pnpm --filter @kaelio/ktx run test:slow
cli-smoke-tests:
name: CLI smoke tests
@ -236,7 +208,7 @@ jobs:
with:
token: ${{ secrets.CODECOV_TOKEN }}
slug: Kaelio/ktx
files: ./packages/cli/coverage/lcov.info,./packages/connector-bigquery/coverage/lcov.info,./packages/connector-clickhouse/coverage/lcov.info,./packages/connector-mysql/coverage/lcov.info,./packages/connector-postgres/coverage/lcov.info,./packages/connector-snowflake/coverage/lcov.info,./packages/connector-sqlite/coverage/lcov.info,./packages/connector-sqlserver/coverage/lcov.info,./packages/context/coverage/lcov.info,./packages/llm/coverage/lcov.info
files: ./packages/cli/coverage/lcov.info
flags: typescript
name: typescript
disable_search: true

View file

@ -41,8 +41,13 @@ repos:
language: system
pass_filenames: false
- id: knip-dead-code
name: knip dead-code check
entry: pnpm exec knip --reporter compact
name: knip dead-code (auto-fix)
entry: pnpm exec knip --fix --reporter compact
language: system
pass_filenames: false
- id: knip-dead-code-production
name: knip dead-code (production mode)
entry: pnpm exec knip --production --reporter compact
language: system
pass_filenames: false

View file

@ -1,6 +1,6 @@
# ktx Development Notes
**ktx** is a standalone open-source context layer for database agents. These
**ktx** is a standalone open-source context layer for data agents. These
instructions apply to all agents working in this repository (Codex, Claude,
Gemini, and similar tools). Do not assume an external app server, frontend,
database migrations, ORPC contracts, or `python-service/` layout exist here.
@ -63,11 +63,10 @@ When rules conflict, follow this order:
**ktx** is a pnpm + uv workspace.
- TypeScript packages: `packages/*`
- CLI package: `packages/cli`
- Core context package: `packages/context`
- LLM package: `packages/llm`
- Database connectors: `packages/connector-*`
- TypeScript package: `packages/cli` (the sole npm-published package source)
- Core context modules: `packages/cli/src/context/`
- LLM provider modules: `packages/cli/src/llm/`
- Database connector modules: `packages/cli/src/connectors/<driver>/`
- Python semantic layer: `python/ktx-sl`
- **ktx** daemon: `python/ktx-daemon`
- Examples and fixtures: `examples/`
@ -76,9 +75,8 @@ When rules conflict, follow this order:
commit `.agents/`, `.claude/`, or `docs/superpowers/` to this public
repository.
Some package names still contain `ktx` during the split. Do not mass-rename
symbols, package names, paths, or docs to `ktx` unless the task asks for that
rename.
Some source identifiers still contain historical package-oriented names. Do not
mass-rename symbols, paths, or docs unless the task asks for that rename.
## Quick Commands
@ -91,7 +89,7 @@ pnpm run type-check
pnpm run test
pnpm run check
pnpm run dead-code
pnpm --filter @ktx/cli run smoke
pnpm --filter @kaelio/ktx run smoke
pnpm --filter './packages/*' run build
pnpm --filter './packages/*' run test
pnpm --filter './packages/*' run type-check
@ -165,11 +163,11 @@ rules there with the same weight as the ones in this file.
- Keep package exports, `types`, and built `dist` expectations aligned when
changing public APIs.
- Use `zod` schemas for runtime validation at CLI/config/API boundaries.
- Keep connector packages thin: connector-specific scanning/auth behavior
belongs in `packages/connector-*`; shared types and orchestration belong in
`packages/context`.
- Avoid circular package dependencies. Shared code should move to the lowest
sensible package, not be duplicated across connectors.
- Keep connector modules thin: connector-specific scanning/auth behavior
belongs in `packages/cli/src/connectors/<driver>/`; shared types and
orchestration belong in `packages/cli/src/context/`.
- Avoid circular module dependencies. Shared code should move to the lowest
sensible module, not be duplicated across connectors.
- Do not manually edit generated or built output under `dist/`; edit source and
rebuild.
@ -179,7 +177,16 @@ rules there with the same weight as the ones in this file.
analysis. These checks are intentionally part of CI and pre-commit because the
normal development workflow is agent-based.
- Run `pnpm run dead-code` after TypeScript changes.
- `pnpm run dead-code` runs three checks: Biome (`dead-code:biome`), Knip
default-mode (`dead-code:knip`), and Knip production-mode
(`dead-code:knip:production`). All three must pass.
- Default-mode Knip catches dead code reachable from no entry at all (broken
graph). Production-mode Knip catches code reachable only via tests —
i.e. code that's tested but doesn't ship.
- Pre-commit runs `knip --fix` (auto-removes the `export` keyword from
symbols that are exported but unused) plus `knip --production` (alerts on
test-only paths). CI runs the same checks without `--fix` and fails on any
finding.
- Treat Knip findings as investigation prompts, not automatic deletion orders.
- Remove private dead code when you confirm there are no imports, dynamic
references, generated references, or tests that still need it.
@ -190,6 +197,26 @@ normal development workflow is agent-based.
- Update `knip.json` when adding dynamic entrypoints, generated files, package
exports, CLI bins, or framework files that Knip cannot infer.
#### Internal exports for testability
When a function, type, or constant must be exported solely so a unit test can
import it (i.e. it has no production cross-file consumer), annotate the
declaration with `/** @internal */` JSDoc. Knip's production-mode check
ignores `@internal` exports, so the convention keeps the gate clean without
silencing the rest of the file.
```typescript
/** @internal */
export function reindexHasErrors(result: ReindexResult): boolean { ... }
```
Do NOT use Vitest in-source testing (`if (import.meta.vitest)` blocks). Keep
tests in separate `*.test.ts(x)` files.
If the only consumer of an export is its own test and the underlying behavior
isn't used in production, delete both the export AND the test — testing dead
code is still dead code.
### CLI Standards
- Use Commander for CLI command trees, arguments, options, help text, custom

View file

@ -153,7 +153,7 @@ ktx wiki "refund policy" --json
ktx sl query --connection-id warehouse --measure orders.revenue --format sql
```
During setup, choose **Ask data questions with ktx MCP** for client agents.
During setup, choose **Ask data questions with ktx MCP** for agent clients.
Choose **Ask data questions + manage ktx with CLI commands** when an operator
agent also needs pinned `ktx` admin commands.
@ -162,20 +162,14 @@ commands to run. If the output includes `ktx mcp start --project-dir ...`, run
it before opening your agent. Claude Desktop uses its own launcher and prints
separate skill upload steps under `.ktx/agents/claude/`.
## Workspace packages
## Workspace layout
| Package | Purpose |
|---------|---------|
| `packages/cli` | CLI entry point |
| `packages/context` | Core context engine |
| `packages/llm` | LLM and embedding providers |
| `packages/connector-bigquery` | BigQuery scan connector |
| `packages/connector-clickhouse` | ClickHouse scan connector |
| `packages/connector-mysql` | MySQL scan connector |
| `packages/connector-postgres` | Postgres scan connector |
| `packages/connector-snowflake` | Snowflake scan connector |
| `packages/connector-sqlite` | SQLite scan connector |
| `packages/connector-sqlserver` | SQL Server scan connector |
| Path | Purpose |
|------|---------|
| `packages/cli` | TypeScript CLI package and published npm package source |
| `packages/cli/src/context` | Core context engine |
| `packages/cli/src/llm` | LLM and embedding providers |
| `packages/cli/src/connectors` | Database scan connectors |
| `python/ktx-sl` | Semantic-layer query planning |
| `python/ktx-daemon` | Portable compute service |

View file

@ -124,7 +124,7 @@ const agent: AgentNode = {
position: { x: AGENT_X, y: AGENT_Y },
data: {
variant: "single",
title: "Analytics agent",
title: "Data agent",
subtitle:
"Asks: monthly net revenue and open tickets per segment, high-value orders only, no test customers",
},

View file

@ -22,8 +22,8 @@ documentation, connector coverage, and examples.
| Area | Good first context |
|------|--------------------|
| CLI and setup | `packages/cli`, especially setup steps, command definitions, status checks, and smoke tests |
| Context engine | `packages/context`, including project config, ingest orchestration, and semantic search |
| Connectors | `packages/connector-*`, plus connector-specific tests and integration docs |
| Context engine | `packages/cli/src/context`, including project config, ingest orchestration, and semantic search |
| Connectors | `packages/cli/src/connectors/<driver>`, plus connector-specific tests and integration docs |
| Python semantic layer | `python/ktx-sl` for planning and SQL compilation |
| **ktx** daemon | `python/ktx-daemon` for the portable runtime API |
| Documentation | `docs-site/content/docs` for public docs and `docs-site/tests` for docs behavior |
@ -50,7 +50,7 @@ pnpm install
uv sync --all-groups
```
`pnpm install` sets up all TypeScript packages in the workspace.
`pnpm install` sets up the TypeScript workspace.
`uv sync --all-groups` installs Python dependencies for the semantic layer and
daemon, including dev and test groups.
@ -60,11 +60,10 @@ daemon, including dev and test groups.
pnpm run build
```
This builds all TypeScript packages. You can also build individual packages:
This builds the TypeScript package. You can also build the package directly:
```bash
pnpm --filter @ktx/cli run build
pnpm --filter @ktx/context run build
pnpm --filter @kaelio/ktx run build
```
### Link the CLI for local testing
@ -80,21 +79,15 @@ changes.
## Repository structure
**ktx** is a pnpm + uv workspace. TypeScript packages live in `packages/`, Python
projects in `python/`.
**ktx** is a pnpm + uv workspace. TypeScript source lives in `packages/cli`,
and Python projects live in `python/`.
```text
packages/
cli/ # CLI entry point and commands
context/ # Core context engine (scan, ingest, MCP, semantic layer)
llm/ # LLM client abstraction
connector-postgres/ # PostgreSQL connector
connector-snowflake/ # Snowflake connector
connector-bigquery/ # BigQuery connector
connector-mysql/ # MySQL connector
connector-sqlserver/ # SQL Server connector
connector-sqlite/ # SQLite connector
connector-posthog/ # PostHog connector
cli/ # CLI package and published npm package source
src/context/ # Core context engine (scan, ingest, MCP, semantic layer)
src/llm/ # LLM client abstraction
src/connectors/ # Database connectors
python/
ktx-sl/ # Semantic layer - grain-aware query planning and SQL compilation
@ -105,7 +98,7 @@ scripts/ # Workspace scripts (benchmarks, verification, release)
docs-site/ # Documentation site (Fumadocs)
```
All TypeScript packages are ESM (`"type": "module"`) and use `NodeNext` module
The TypeScript package is ESM (`"type": "module"`) and uses `NodeNext` module
resolution. The Python projects use `pyproject.toml` for dependency management.
## Running tests
@ -116,18 +109,17 @@ resolution. The Python projects use `pyproject.toml` for dependency management.
# Run all tests
pnpm run test
# Run tests for a specific package
pnpm --filter @ktx/cli run test
pnpm --filter @ktx/context run test
# Run tests for the TypeScript package
pnpm --filter @kaelio/ktx run test
# Type-check all packages
pnpm run type-check
# Type-check a specific package
pnpm --filter @ktx/context run type-check
# Type-check the TypeScript package
pnpm --filter @kaelio/ktx run type-check
# CLI smoke test
pnpm --filter @ktx/cli run smoke
pnpm --filter @kaelio/ktx run smoke
```
### Python
@ -164,43 +156,22 @@ uv run pytest -q
## Adding a connector
Database connectors live in `packages/connector-<name>/`. Each connector
implements the `KtxScanConnector` interface from `@ktx/context`.
Database connectors live in `packages/cli/src/connectors/<driver>/`. Each
connector implements the `KtxScanConnector` interface from the internal context
modules.
### Step 1: Scaffold the package
### Step 1: Scaffold the connector
Create a new directory at `packages/connector-<name>/` with:
Create a new directory at `packages/cli/src/connectors/<driver>/` with:
```text
packages/connector-<name>/
package.json
tsconfig.json
src/
index.ts # Public exports
connector.ts # KtxScanConnector implementation
dialect.ts # SQL dialect handling
packages/cli/src/connectors/<driver>/
index.ts # Internal connector exports
connector.ts # KtxScanConnector implementation
dialect.ts # SQL dialect handling
```
The `package.json` should follow the pattern of existing connectors:
```json
{
"name": "@ktx/connector-<name>",
"private": true,
"type": "module",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"exports": {
".": {
"types": "./dist/index.d.ts",
"import": "./dist/index.js"
}
},
"dependencies": {
"@ktx/context": "workspace:*"
}
}
```
Add any connector-specific npm dependency to `packages/cli/package.json`.
### Step 2: Implement the connector
@ -226,20 +197,20 @@ and statistics.
### Step 4: Wire it up
Register the new connector in `packages/context` so the CLI and scan
engine can instantiate it. Look at how existing connectors are registered for
the pattern.
Register the new connector in `packages/cli/src/local-scan-connectors.ts` and
`packages/cli/src/local-adapters.ts` so the CLI and scan engine can instantiate
it. Keep runtime loading dynamic when the connector is optional.
### Step 5: Test
```bash
pnpm --filter @ktx/connector-<name> run build
pnpm --filter @ktx/connector-<name> run type-check
pnpm --filter @ktx/connector-<name> run test
pnpm --filter @kaelio/ktx run build
pnpm --filter @kaelio/ktx run type-check
pnpm --filter @kaelio/ktx run test
```
Use `packages/connector-sqlite/` as a minimal reference and
`packages/connector-postgres/` as a full-featured one.
Use `packages/cli/src/connectors/sqlite/` as a minimal reference and
`packages/cli/src/connectors/postgres/` as a full-featured one.
## Code conventions

View file

@ -6,9 +6,9 @@ description: Set up ktx with Claude Code, Claude Desktop, Cursor, Codex, and Ope
**ktx** exposes context to end-user agents through MCP tools. The CLI remains the
admin surface for setup, ingest, status, daemon lifecycle, and debugging.
Run `ktx setup` and select your client agent targets, or configure manually
using the snippets below. Choose **Ask data questions with ktx MCP** for client
agents. Choose **Ask data questions + manage ktx with CLI commands** only when
Run `ktx setup` and select your agent client targets, or configure manually
using the snippets below. Choose **Ask data questions with ktx MCP** for agent
clients. Choose **Ask data questions + manage ktx with CLI commands** only when
a developer or operator agent also needs pinned `ktx` admin commands.
## Install with setup

View file

@ -105,27 +105,36 @@ prior rc lineage is consumed cleanly.
## Release metadata
semantic-release calls `scripts/update-public-release-version.mjs` during the
prepare step before the exec publish command runs. That script updates the
following files **inside the CI runner only**:
prepare step before the exec publish command runs. That script rewrites the
following files in lockstep:
- `package.json` with the semantic-release version.
- `release-policy.json` with `publicNpmPackageVersion`, npm publish settings,
and the published package smoke-test version.
- `package.json` and `packages/cli/package.json` with the semantic-release
version.
- `python/ktx-daemon/pyproject.toml` and `python/ktx-sl/pyproject.toml` with
the same version, normalized for PEP 440 (e.g. `0.1.0-rc.2`
`0.1.0rc2`). Branch-prefixed npm releases skip this step because they are
not published to PyPI.
- `release-policy.json` with the npm publish settings, release mode, and
published package smoke-test version. The package version itself is not
stored here; it is derived from `packages/cli/package.json.version` by
`scripts/public-npm-release-metadata.mjs`, so there is only one place that
owns the version.
The artifact packaging, readiness, and smoke-test scripts read
`publicNpmPackageVersion` from `release-policy.json` within the same CI run.
Nothing reads these files at runtime — the daemon and CLI rely on the
published `package.json` (for the installed `@kaelio/ktx` package) or
`packages/cli/package.json` (for dev-tree runs from this repo, which always
report `0.0.0-private`). Because the metadata mutation never has to survive
the run, no commit is pushed back to `main`. The git tag plus the published
npm artifact carry the version forward.
The `@semantic-release/git` plugin then commits these files back to `main`
with the release tag (see `scripts/semantic-release-config.cjs`). As a
result, the dev tree always reflects the most recently published version —
there is no sentinel pin. `ktx --version` and the bundled Python wheel both
report whatever was last released.
The bundled Python runtime wheel also derives its version from
`publicNpmPackageVersion`. Stable npm versions are reused as-is, and rc
versions are normalized to Python's version format. For example,
`0.1.0-rc.2` becomes `0.1.0rc2` in the `kaelio-ktx` wheel filename and wheel
metadata.
At runtime the CLI reads its version from its own `package.json` via
`getKtxCliPackageInfo()` (`packages/cli/src/cli-runtime.ts`); the Python
daemon reads its version from installed-package metadata via
`importlib.metadata.version()` (`python/ktx-daemon/src/ktx_daemon/__init__.py`).
The bundled Python runtime wheel also derives its version from the same
single source. Stable npm versions are reused as-is, and rc versions are
normalized to Python's version format. For example, `0.1.0-rc.2` becomes
`0.1.0rc2` in the `kaelio-ktx` wheel filename and wheel metadata.
## npm authentication

View file

@ -58,7 +58,7 @@ Create a project and enable query history:
```bash
export WAREHOUSE_DATABASE_URL=postgresql://ktx_reader:ktx_reader@127.0.0.1:55432/analytics # pragma: allowlist secret
pnpm --filter @ktx/cli run build
pnpm --filter @kaelio/ktx run build
node packages/cli/dist/bin.js --project-dir /tmp/ktx-postgres-historic setup \
--new \
--skip-agents \

View file

@ -198,8 +198,7 @@ NODE
}
cd "$KTX_ROOT"
pnpm --filter @ktx/context run build
pnpm --filter @ktx/cli run build
pnpm --filter @kaelio/ktx run build
docker compose -f "$COMPOSE_FILE" up -d --wait
"$EXAMPLE_DIR/scripts/generate-workload.sh" base

132
knip.json
View file

@ -2,121 +2,41 @@
"$schema": "https://unpkg.com/knip@6/schema.json",
"workspaces": {
".": {
"entry": ["scripts/**/*.mjs"],
"project": ["scripts/**/*.mjs"],
"ignoreDependencies": [
"@semantic-release/commit-analyzer",
"@semantic-release/github",
"@semantic-release/npm",
"@semantic-release/release-notes-generator",
"conventional-changelog-conventionalcommits"
"entry": [
"scripts/**/*.mjs",
"scripts/**/*.cjs",
".releaserc.cjs!"
]
},
"packages/cli": {
"entry": [
"src/index.ts",
"src/bin.ts",
"src/**/*.test.ts",
"src/**/*.test.tsx",
"scripts/**/*.mjs"
],
"project": ["src/**/*.{ts,tsx}", "scripts/**/*.mjs", "vitest.config.ts"]
},
"packages/context": {
"entry": [
"src/index.ts",
"src/agent/index.ts",
"src/core/index.ts",
"src/connections/index.ts",
"src/daemon/index.ts",
"src/ingest/index.ts",
"src/ingest/memory-flow/index.ts",
"src/ingest/metabase-mapping.ts",
"src/scan/index.ts",
"src/search/index.ts",
"src/sql-analysis/index.ts",
"src/memory/index.ts",
"src/mcp/index.ts",
"src/project/index.ts",
"src/prompts/index.ts",
"src/skills/index.ts",
"src/sl/index.ts",
"src/sl/descriptions.ts",
"src/tools/index.ts",
"src/wiki/index.ts",
"src/**/*.test.ts",
"scripts/**/*.mjs"
],
"project": ["src/**/*.ts", "scripts/**/*.mjs", "vitest.config.ts"]
},
"packages/llm": {
"entry": ["src/index.ts", "src/**/*.test.ts"],
"project": ["src/**/*.ts", "vitest.config.ts"]
},
"packages/connector-*": {
"entry": ["src/index.ts", "src/**/*.test.ts"],
"project": ["src/**/*.ts"]
"src/print-command-tree.ts!",
"scripts/**/*.mjs",
"src/**/*.test-utils.ts",
"src/**/acceptance-fixtures.ts",
"src/context/scan/relationship-benchmarks.ts!",
"src/context/scan/relationship-benchmark-report.ts!"
]
},
"docs-site": {
"entry": [
"app/**/*.{ts,tsx}",
"components/**/*.{ts,tsx}",
"lib/**/*.{ts,tsx}",
"middleware.ts",
"next.config.mjs",
"source.config.ts",
"tests/**/*.mjs"
"components/**/*.{ts,tsx}!",
"source.config.ts!"
],
"project": [
"app/**/*.{ts,tsx}",
"components/**/*.{ts,tsx}",
"lib/**/*.{ts,tsx}",
"*.ts",
"*.mjs",
"tests/**/*.mjs"
],
"ignoreDependencies": ["tailwindcss"]
"ignoreDependencies": [
"tailwindcss"
]
}
},
"ignore": [
"**/dist/**",
"**/coverage/**",
"**/.next/**",
"**/node_modules/**",
"**/*.gen.ts",
"**/*.generated.ts"
"ignoreDependencies": [
"@semantic-release/commit-analyzer",
"@semantic-release/github",
"@semantic-release/npm",
"@semantic-release/release-notes-generator",
"conventional-changelog-conventionalcommits"
],
"ignoreBinaries": ["uv", "lsof"],
"ignoreIssues": {
"packages/cli/src/clack.ts": ["exports"],
"packages/cli/src/commands/connection-metabase-setup.ts": ["exports", "types"],
"packages/cli/src/ingest.test-utils.ts": ["exports"],
"packages/cli/src/io/symbols.ts": ["exports"],
"packages/cli/src/managed-python-command.ts": ["types"],
"packages/cli/src/managed-python-daemon.ts": ["types"],
"packages/cli/src/managed-python-http.ts": ["exports", "types"],
"packages/cli/src/managed-python-runtime.ts": ["types"],
"packages/cli/src/memory-flow-tui.tsx": ["types"],
"packages/cli/src/next-steps.ts": ["exports"],
"packages/cli/src/print-command-tree.ts": ["exports"],
"packages/cli/src/setup-agents.ts": ["exports", "types"],
"packages/cli/src/setup-context.ts": ["types"],
"packages/cli/src/setup-demo-tour.ts": ["exports"],
"packages/cli/src/setup-models.ts": ["exports"],
"packages/cli/src/setup-project.ts": ["types"],
"packages/cli/src/setup-ready-menu.ts": ["types"],
"packages/cli/src/setup-sources.ts": ["types"],
"packages/context/src/ingest/adapters/historic-sql/pattern-inputs.ts": ["exports", "types"],
"packages/context/src/ingest/adapters/lookml/pull-config.ts": ["exports"],
"packages/context/src/ingest/adapters/metabase/serialize-card.ts": ["types"],
"packages/context/src/ingest/adapters/metabase/types.ts": ["exports"],
"packages/context/src/ingest/adapters/metricflow/parse.ts": ["types"],
"packages/context/src/ingest/ports.ts": ["types"],
"packages/context/src/ingest/stages/stage-3-work-units.ts": ["types"],
"packages/context/src/ingest/stages/stage-index.types.ts": ["types"],
"packages/context/src/project/config.ts": ["types"],
"packages/context/src/scan/relationship-candidates.ts": ["types"],
"packages/context/src/scan/relationship-diagnostics.ts": ["types"],
"packages/context/src/tools/context-evidence-tool-store.ts": ["types"]
}
"ignoreBinaries": [
"uv",
"lsof"
]
}

View file

@ -19,10 +19,11 @@
"artifacts:verify-manifest": "node scripts/package-artifacts.mjs verify-manifest",
"build": "pnpm --filter './packages/*' run build",
"check": "node scripts/check-boundaries.mjs && node --test scripts/*.test.mjs && pnpm --filter './packages/*' run build && pnpm --filter './packages/*' run test",
"dead-code": "pnpm run dead-code:biome && pnpm run dead-code:knip",
"dead-code": "pnpm run dead-code:biome && pnpm run dead-code:knip && pnpm run dead-code:knip:production",
"dead-code:biome": "biome ci . --formatter-enabled=false --assist-enabled=false",
"dead-code:fix": "biome check . --formatter-enabled=false --assist-enabled=false --write && knip --fix --format",
"dead-code:knip": "knip --reporter compact",
"dead-code:knip:production": "knip --production --reporter compact",
"docs": "kill $(lsof -ti:3000) 2>/dev/null; pnpm --filter ktx-docs run dev",
"ktx": "node scripts/run-ktx.mjs",
"link:dev": "node scripts/link-dev-cli.mjs",
@ -39,16 +40,18 @@
"semantic-release": "semantic-release",
"semantic-release:debug": "semantic-release --dry-run --debug",
"semantic-release:dry-run": "semantic-release --dry-run --no-ci",
"smoke": "pnpm run build && pnpm --filter @ktx/cli run smoke",
"smoke": "pnpm run build && pnpm --filter @kaelio/ktx run smoke",
"test": "node --test scripts/*.test.mjs && pnpm --filter './packages/*' run test",
"test:coverage": "pnpm run test:coverage:ts && pnpm run test:coverage:py",
"test:coverage:py": "uv run pytest --cov=python/ktx-sl/semantic_layer --cov=python/ktx-daemon/src/ktx_daemon --cov-report=xml:coverage/python.xml --cov-report=term",
"test:coverage:ts": "pnpm --filter './packages/*' run build && pnpm --filter './packages/*' run test --coverage --coverage.reporter=lcov --coverage.exclude='dist/**' && node scripts/normalize-lcov-paths.mjs packages/*/coverage/lcov.info",
"test:slow": "pnpm --filter @ktx/context run test:slow && pnpm --filter @ktx/cli run test:slow",
"test:slow": "pnpm --filter @kaelio/ktx run test:slow",
"type-check": "pnpm --filter './packages/*' run type-check"
},
"devDependencies": {
"@biomejs/biome": "^2.4.15",
"@electric-sql/pglite": "^0.4.5",
"@electric-sql/pglite-socket": "^0.1.5",
"@semantic-release/commit-analyzer": "^13.0.1",
"@semantic-release/exec": "^7.1.0",
"@semantic-release/git": "^10.0.1",
@ -59,6 +62,7 @@
"better-sqlite3": "^12.10.0",
"conventional-changelog-conventionalcommits": "^9.3.1",
"knip": "^6.12.2",
"pg": "^8.20.0",
"semantic-release": "^25.0.3",
"typescript": "^6.0.3",
"yaml": "^2.9.0"

View file

@ -1,8 +1,7 @@
{
"name": "@ktx/cli",
"name": "@kaelio/ktx",
"version": "0.4.1",
"description": "CLI wrapper for ktx context packages",
"private": true,
"description": "Standalone ktx context layer for data agents",
"type": "module",
"engines": {
"node": ">=22.0.0"
@ -24,41 +23,68 @@
"dist",
"assets"
],
"publishConfig": {
"access": "public",
"provenance": true
},
"scripts": {
"assets:demo": "node scripts/build-demo-assets.mjs",
"build": "tsc -p tsconfig.json && node scripts/copy-runtime-assets.mjs && node ../../scripts/prepare-cli-bin.mjs",
"clean": "node -e \"fs.rmSync('dist', { recursive: true, force: true }); fs.rmSync('node_modules/.cache/tsc.tsbuildinfo', { force: true })\"",
"clean": "node -e \"fs.rmSync('dist', { recursive: true, force: true })\"",
"docs:commands": "pnpm run build && node dist/print-command-tree.js",
"smoke": "vitest run src/standalone-smoke.test.ts src/example-smoke.test.ts --testTimeout 30000",
"test": "vitest run --exclude src/standalone-smoke.test.ts --exclude src/example-smoke.test.ts --exclude src/setup-databases.test.ts --exclude src/scan.test.ts --exclude src/commands/connection-metabase-setup.test.ts --exclude src/setup-models.test.ts --exclude src/setup-sources.test.ts --exclude src/setup.test.ts --exclude src/connection.test.ts --exclude src/setup-embeddings.test.ts --exclude src/ingest.test.ts --exclude src/commands/connection-mapping.test.ts --exclude src/ingest-viz.test.ts --exclude src/demo.test.ts --exclude src/setup-project.test.ts --exclude src/sl.test.ts --exclude src/local-scan-connectors.test.ts --exclude src/commands/connection-notion.test.ts",
"test:slow": "vitest run src/setup-databases.test.ts src/scan.test.ts src/commands/connection-metabase-setup.test.ts src/setup-models.test.ts src/setup-sources.test.ts src/setup.test.ts src/connection.test.ts src/setup-embeddings.test.ts src/ingest.test.ts src/commands/connection-mapping.test.ts src/ingest-viz.test.ts src/demo.test.ts src/setup-project.test.ts src/sl.test.ts src/local-scan-connectors.test.ts src/commands/connection-notion.test.ts --testTimeout 30000",
"type-check": "tsc -p tsconfig.json --noEmit"
"test": "vitest run --exclude src/standalone-smoke.test.ts --exclude src/example-smoke.test.ts --exclude src/setup-databases.test.ts --exclude src/scan.test.ts --exclude src/commands/connection-metabase-setup.test.ts --exclude src/setup-models.test.ts --exclude src/setup-sources.test.ts --exclude src/setup.test.ts --exclude src/connection.test.ts --exclude src/setup-embeddings.test.ts --exclude src/ingest.test.ts --exclude src/commands/connection-mapping.test.ts --exclude src/ingest-viz.test.ts --exclude src/demo.test.ts --exclude src/setup-project.test.ts --exclude src/sl.test.ts --exclude src/local-scan-connectors.test.ts --exclude src/commands/connection-notion.test.ts --exclude src/context/scan/local-scan.test.ts --exclude src/context/mcp/local-project-ports.test.ts --exclude src/context/ingest/local-stage-ingest.test.ts --exclude src/context/sl/pglite-sl-search-prototype.test.ts --exclude src/context/core/git.service.test.ts --exclude src/context/ingest/local-adapters.test.ts --exclude src/context/ingest/local-bundle-ingest.test.ts --exclude src/context/ingest/local-metabase-ingest.test.ts --exclude src/context/sl/local-sl.test.ts --exclude src/context/search/pglite-owner-process.test.ts --exclude src/context/scan/local-enrichment-artifacts.test.ts --exclude src/context/search/pglite-spike.test.ts --exclude src/context/wiki/local-knowledge.test.ts --exclude src/context/sl/local-query.test.ts --exclude src/context/scan/relationship-review-decisions.test.ts --exclude src/context/scan/relationship-profiling.test.ts",
"test:slow": "vitest run src/setup-databases.test.ts src/scan.test.ts src/commands/connection-metabase-setup.test.ts src/setup-models.test.ts src/setup-sources.test.ts src/setup.test.ts src/connection.test.ts src/setup-embeddings.test.ts src/ingest.test.ts src/commands/connection-mapping.test.ts src/ingest-viz.test.ts src/demo.test.ts src/setup-project.test.ts src/sl.test.ts src/local-scan-connectors.test.ts src/commands/connection-notion.test.ts src/context/scan/local-scan.test.ts src/context/mcp/local-project-ports.test.ts src/context/ingest/local-stage-ingest.test.ts src/context/sl/pglite-sl-search-prototype.test.ts src/context/core/git.service.test.ts src/context/ingest/local-adapters.test.ts src/context/ingest/local-bundle-ingest.test.ts src/context/ingest/local-metabase-ingest.test.ts src/context/sl/local-sl.test.ts src/context/search/pglite-owner-process.test.ts src/context/scan/local-enrichment-artifacts.test.ts src/context/search/pglite-spike.test.ts src/context/wiki/local-knowledge.test.ts src/context/sl/local-query.test.ts src/context/scan/relationship-review-decisions.test.ts src/context/scan/relationship-profiling.test.ts --testTimeout 30000",
"type-check": "tsc -p tsconfig.json --noEmit",
"relationships:benchmarks": "pnpm --silent run build && node ../../scripts/relationship-benchmark-report.mjs",
"relationships:benchmarks:test": "KTX_RUN_RELATIONSHIP_BENCHMARKS=1 vitest run src/context/scan/relationship-benchmarks.test.ts",
"search:pglite-spike": "node ../../scripts/pglite-hybrid-search-spike.mjs",
"search:pglite-owner-prototype": "node ../../scripts/pglite-owner-process-prototype.mjs",
"search:pglite-sl-prototype": "node ../../scripts/pglite-sl-search-prototype.mjs"
},
"dependencies": {
"@ai-sdk/anthropic": "3.0.77",
"@ai-sdk/devtools": "0.0.17",
"@ai-sdk/google-vertex": "^4.0.128",
"@anthropic-ai/claude-agent-sdk": "0.3.142",
"@clack/prompts": "1.4.0",
"@clickhouse/client": "^1.18.4",
"@commander-js/extra-typings": "14.0.0",
"@ktx/connector-bigquery": "workspace:*",
"@ktx/connector-clickhouse": "workspace:*",
"@ktx/connector-mysql": "workspace:*",
"@ktx/connector-postgres": "workspace:*",
"@ktx/connector-snowflake": "workspace:*",
"@ktx/connector-sqlite": "workspace:*",
"@ktx/connector-sqlserver": "workspace:*",
"@ktx/context": "workspace:*",
"@ktx/llm": "workspace:*",
"@google-cloud/bigquery": "^8.3.1",
"@looker/sdk": "^26.8.0",
"@looker/sdk-node": "^26.8.0",
"@looker/sdk-rtl": "^21.6.5",
"@modelcontextprotocol/sdk": "^1.29.0",
"@notionhq/client": "^5.21.0",
"ai": "^6.0.180",
"better-sqlite3": "^12.10.0",
"commander": "14.0.3",
"fflate": "^0.8.2",
"handlebars": "^4.7.9",
"ink": "^7.0.2",
"lookml-parser": "7.1.0",
"minimatch": "^10.2.5",
"mssql": "^12.5.2",
"mysql2": "^3.22.3",
"openai": "^6.37.0",
"p-limit": "^7.3.0",
"pg": "^8.20.0",
"react": "^19.2.6",
"simple-git": "3.36.0",
"snowflake-sdk": "^2.4.1",
"yaml": "^2.9.0",
"zod": "^4.4.3"
},
"devDependencies": {
"@electric-sql/pglite": "^0.4.5",
"@electric-sql/pglite-socket": "^0.1.5",
"@types/better-sqlite3": "^7.6.13",
"@types/mssql": "^12.3.0",
"@types/node": "^25.7.0",
"@types/pg": "^8.20.0",
"@types/react": "^19.2.14",
"@vitest/coverage-v8": "^4.1.6",
"better-sqlite3": "^12.10.0",
"ajv": "8.20.0",
"ink-testing-library": "^4.0.0",
"typescript": "^6.0.3",
"vitest": "^4.1.6"
@ -66,11 +92,11 @@
"license": "Apache-2.0",
"repository": {
"type": "git",
"url": "git+https://github.com/kaelio/ktx.git",
"url": "https://github.com/Kaelio/ktx",
"directory": "packages/cli"
},
"bugs": {
"url": "https://github.com/kaelio/ktx/issues"
"url": "https://github.com/Kaelio/ktx/issues"
},
"homepage": "https://github.com/kaelio/ktx#readme"
"homepage": "https://github.com/Kaelio/ktx#readme"
}

View file

@ -3,9 +3,14 @@ import { dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
const packageRoot = fileURLToPath(new URL('..', import.meta.url));
const promptsSource = join(packageRoot, 'src', 'prompts');
const promptsTarget = join(packageRoot, 'dist', 'prompts');
const skillsSource = join(packageRoot, 'src', 'skills');
const skillsTarget = join(packageRoot, 'dist', 'skills');
await rm(promptsTarget, { recursive: true, force: true });
await rm(skillsTarget, { recursive: true, force: true });
await mkdir(dirname(promptsTarget), { recursive: true });
await mkdir(dirname(skillsTarget), { recursive: true });
await cp(promptsSource, promptsTarget, { recursive: true });
await cp(skillsSource, skillsTarget, { recursive: true });

View file

@ -1,11 +1,11 @@
import { createRequire } from 'node:module';
import type { ReindexSummary } from '@ktx/context/index-sync';
import type { ReindexSummary } from './context/index-sync/types.js';
import { describe, expect, it, vi } from 'vitest';
import { renderReindexJson, renderReindexPlain, reindexHasErrors } from './admin-reindex.js';
import { runKtxCli } from './index.js';
const cliVersion = (createRequire(import.meta.url)('@ktx/cli/package.json') as { version: string })
const cliVersion = (createRequire(import.meta.url)('@kaelio/ktx/package.json') as { version: string })
.version;
function makeIo(options: { stdoutIsTTY?: boolean } = {}) {

View file

@ -1,6 +1,8 @@
import { KtxIngestEmbeddingPortAdapter, type KtxEmbeddingPort } from '@ktx/context';
import { reindexLocalIndexes, type ReindexScopeResult, type ReindexSummary } from '@ktx/context/index-sync';
import { loadKtxProject } from '@ktx/context/project';
import { KtxIngestEmbeddingPortAdapter } from './context/llm/embedding-port.js';
import type { KtxEmbeddingPort } from './context/core/embedding.js';
import { reindexLocalIndexes } from './context/index-sync/reindex.js';
import type { ReindexScopeResult, ReindexSummary } from './context/index-sync/types.js';
import { loadKtxProject } from './context/project/project.js';
import { Option, type Command } from '@commander-js/extra-typings';
import { cancel, intro, log, note, outro } from '@clack/prompts';
import type { KtxCliCommandContext } from './cli-program.js';
@ -55,10 +57,12 @@ function quotePlainValue(value: string): string {
return `"${value.replaceAll('\\', '\\\\').replaceAll('"', '\\"')}"`;
}
/** @internal */
export function reindexHasErrors(summary: ReindexSummary): boolean {
return summary.scopes.some((scope) => scope.error);
}
/** @internal */
export function renderReindexPlain(summary: ReindexSummary, io: KtxCliIo): void {
const updateKey = summary.force ? 'rebuilt' : 'updated';
for (const scope of summary.scopes) {
@ -88,6 +92,7 @@ export function renderReindexPlain(summary: ReindexSummary, io: KtxCliIo): void
);
}
/** @internal */
export function renderReindexJson(summary: ReindexSummary, io: KtxCliIo): void {
io.stdout.write(`${JSON.stringify({ kind: 'reindex', data: summary, meta: { command: 'admin reindex' } }, null, 2)}\n`);
}

View file

@ -50,7 +50,7 @@ export function registerAdminCommands(program: Command, context: KtxCliCommandCo
.description('Print a JSON Schema describing ktx.yaml (for editors and LLM agents)')
.option('--output <file>', 'Write the schema to a file instead of stdout')
.action(async (options: { output?: string }) => {
const { generateKtxProjectConfigJsonSchema } = await import('@ktx/context/project');
const { generateKtxProjectConfigJsonSchema } = await import('./context/project/config.js');;
const json = `${JSON.stringify(generateKtxProjectConfigJsonSchema(), null, 2)}\n`;
if (options.output) {
const { writeFile } = await import('node:fs/promises');

View file

@ -26,7 +26,7 @@ export interface KtxCliPromptAdapter {
spinner(): KtxCliSpinner;
}
export class KtxCliPromptCancelledError extends Error {
class KtxCliPromptCancelledError extends Error {
constructor(message = 'Operation cancelled.') {
super(message);
this.name = 'KtxCliPromptCancelledError';

View file

@ -1,4 +1,4 @@
import type { KtxProjectLlmConfig } from '@ktx/context/project';
import type { KtxProjectLlmConfig } from './context/project/config.js';
const CLAUDE_CODE_IGNORED_PROMPT_CACHING_FIELDS = [
'systemTtl',

View file

@ -12,9 +12,8 @@ function stubIo(): KtxCliIo {
function stubPackageInfo(): KtxCliPackageInfo {
return {
name: '@ktx/cli',
name: '@kaelio/ktx',
version: '0.0.0-test',
contextPackageName: '@ktx/context',
};
}

View file

@ -1,26 +0,0 @@
import { describe, expect, it, vi } from 'vitest';
import { buildDefaultKtxProjectConfig, type KtxLocalProject, type KtxProjectConfig } from '@ktx/context/project';
import { loadKtxCliProject } from './cli-project.js';
function projectWithConfig(config: KtxProjectConfig): KtxLocalProject {
return {
projectDir: '/work/proj',
configPath: '/work/proj/ktx.yaml',
config,
coreConfig: {} as KtxLocalProject['coreConfig'],
git: {} as KtxLocalProject['git'],
fileStore: {} as KtxLocalProject['fileStore'],
};
}
describe('loadKtxCliProject', () => {
it('delegates to loadKtxProject and returns the project unchanged', async () => {
const project = projectWithConfig(buildDefaultKtxProjectConfig());
const loadProject = vi.fn(async () => project);
const result = await loadKtxCliProject({ projectDir: '/work/proj' }, { loadProject });
expect(result).toBe(project);
expect(loadProject).toHaveBeenCalledWith({ projectDir: '/work/proj' });
});
});

View file

@ -1,20 +0,0 @@
import { loadKtxProject, type KtxLocalProject } from '@ktx/context/project';
export interface LoadKtxCliProjectOptions {
projectDir: string;
}
export interface LoadKtxCliProjectDeps {
loadProject?: typeof loadKtxProject;
}
/**
* Thin wrapper around `loadKtxProject`. Kept as a single entrypoint so the CLI can grow shared
* pre-load behavior later (telemetry, project lock, etc.). Today it does no extra work.
*/
export async function loadKtxCliProject(
options: LoadKtxCliProjectOptions,
deps: LoadKtxCliProjectDeps = {},
): Promise<KtxLocalProject> {
return (deps.loadProject ?? loadKtxProject)({ projectDir: options.projectDir });
}

View file

@ -20,7 +20,6 @@ const requirePackageJson = createRequire(import.meta.url);
export interface KtxCliPackageInfo {
name: string;
version: string;
contextPackageName: '@ktx/context';
}
export interface KtxCliIo {
@ -67,12 +66,11 @@ export function packageInfoFromJson(packageJson: unknown): KtxCliPackageInfo {
return {
name: packageJson.name,
version: assertCliVersion(packageJson.version, `${packageJson.name}/package.json`),
contextPackageName: '@ktx/context',
};
}
async function runInit(args: { projectDir: string; force: boolean }, io: KtxCliIo): Promise<number> {
const { initKtxProject } = await import('@ktx/context/project');
const { initKtxProject } = await import('./context/project/project.js');;
const result = await initKtxProject({
projectDir: args.projectDir,
force: args.force,

View file

@ -11,7 +11,7 @@ function makeContext(overrides: Partial<KtxCliCommandContext> = {}): KtxCliComma
stderr: { write: vi.fn() },
},
deps: {},
packageInfo: { name: '@ktx/cli', version: '0.0.0-test', contextPackageName: '@ktx/context' },
packageInfo: { name: '@kaelio/ktx', version: '0.0.0-test' },
setExitCode: (code) => {
exitCode = code;
},

View file

@ -11,7 +11,7 @@ function makeContext(overrides: Partial<KtxCliCommandContext> = {}): KtxCliComma
stderr: { write: vi.fn() },
},
deps: {},
packageInfo: { name: '@ktx/cli', version: '0.0.0-test', contextPackageName: '@ktx/context' },
packageInfo: { name: '@kaelio/ktx', version: '0.0.0-test' },
setExitCode: (code) => {
exitCode = code;
},

View file

@ -1,9 +1,12 @@
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import type { LookerClient, MetabaseRuntimeClient, NotionClient } from '@ktx/context/ingest';
import { initKtxProject, parseKtxProjectConfig, serializeKtxProjectConfig } from '@ktx/context/project';
import type { KtxConnectionDriver, KtxScanConnector } from '@ktx/context/scan';
import type { LookerClient } from './context/ingest/adapters/looker/client.js';
import type { MetabaseRuntimeClient } from './context/ingest/adapters/metabase/client-port.js';
import type { NotionClient } from './context/ingest/adapters/notion/notion-client.js';
import { initKtxProject } from './context/project/project.js';
import { parseKtxProjectConfig, serializeKtxProjectConfig } from './context/project/config.js';
import type { KtxConnectionDriver, KtxScanConnector } from './context/scan/types.js';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { runKtxConnection } from './connection.js';

View file

@ -1,19 +1,15 @@
import {
DEFAULT_METABASE_CLIENT_CONFIG,
DefaultLookerConnectionClientFactory,
DefaultMetabaseConnectionClientFactory,
type LookerClient,
type MetabaseRuntimeClient,
type NotionBotInfo,
NotionClient,
createLocalLookerCredentialResolver,
metabaseRuntimeConfigFromLocalConnection,
testRepoConnection,
} from '@ktx/context/ingest';
import { parseNotionConnectionConfig, resolveNotionConnectionAuthToken } from '@ktx/context/connections';
import { resolveKtxConfigReference } from '@ktx/context/core';
import { type KtxLocalProject, loadKtxProject } from '@ktx/context/project';
import type { KtxScanConnector } from '@ktx/context/scan';
import { DEFAULT_METABASE_CLIENT_CONFIG, DefaultMetabaseConnectionClientFactory } from './context/ingest/adapters/metabase/client.js';
import { DefaultLookerConnectionClientFactory } from './context/ingest/adapters/looker/factory.js';
import type { LookerClient } from './context/ingest/adapters/looker/client.js';
import type { MetabaseRuntimeClient } from './context/ingest/adapters/metabase/client-port.js';
import { type NotionBotInfo, NotionClient } from './context/ingest/adapters/notion/notion-client.js';
import { createLocalLookerCredentialResolver } from './context/ingest/adapters/looker/local-looker.adapter.js';
import { metabaseRuntimeConfigFromLocalConnection } from './context/ingest/adapters/metabase/local-metabase.adapter.js';
import { testRepoConnection } from './context/ingest/repo-fetch.js';
import { parseNotionConnectionConfig, resolveNotionConnectionAuthToken } from './context/connections/notion-config.js';
import { resolveKtxConfigReference } from './context/core/config-reference.js';
import { type KtxLocalProject, loadKtxProject } from './context/project/project.js';
import type { KtxScanConnector } from './context/scan/types.js';
import type { KtxCliIo } from './index.js';
import { bold, dim, green, red, SYMBOLS } from './io/symbols.js';
import { createKtxCliScanConnector } from './local-scan-connectors.js';

View file

@ -1,15 +1,6 @@
import { describe, expect, it, vi } from 'vitest';
import {
bigQueryConnectionConfigFromConfig,
createBigQueryLiveDatabaseIntrospection,
isKtxBigQueryConnectionConfig,
type KtxBigQueryClient,
KtxBigQueryScanConnector,
type KtxBigQueryClientFactory,
type KtxBigQueryDataset,
type KtxBigQueryQueryJob,
type KtxBigQueryTableRef,
} from './index.js';
import { bigQueryConnectionConfigFromConfig, isKtxBigQueryConnectionConfig, type KtxBigQueryClient, KtxBigQueryScanConnector, type KtxBigQueryClientFactory, type KtxBigQueryDataset, type KtxBigQueryQueryJob, type KtxBigQueryTableRef } from '../../connectors/bigquery/connector.js';
import { createBigQueryLiveDatabaseIntrospection } from '../../connectors/bigquery/live-database-introspection.js';
function fakeClientFactory(): KtxBigQueryClientFactory {
const queryResults = vi.fn(async (): ReturnType<KtxBigQueryQueryJob['getQueryResults']> => [

View file

@ -1,24 +1,6 @@
import { BigQuery, type TableField } from '@google-cloud/bigquery';
import { assertReadOnlySql, limitSqlForExecution } from '@ktx/context/connections';
import {
createKtxConnectorCapabilities,
type KtxColumnSampleInput,
type KtxColumnSampleResult,
type KtxColumnStatsInput,
type KtxColumnStatsResult,
type KtxQueryResult,
type KtxReadOnlyQueryInput,
type KtxScanConnector,
type KtxScanContext,
type KtxScanInput,
type KtxSchemaColumn,
type KtxSchemaSnapshot,
type KtxSchemaTable,
type KtxTableListEntry,
type KtxTableRef,
type KtxTableSampleInput,
type KtxTableSampleResult,
} from '@ktx/context/scan';
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaColumn, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableListEntry, type KtxTableRef, type KtxTableSampleInput, type KtxTableSampleResult } from '../../context/scan/types.js';
import { readFileSync } from 'node:fs';
import { homedir } from 'node:os';
import { resolve } from 'node:path';
@ -57,12 +39,14 @@ export interface KtxBigQueryColumnDistinctValuesResult {
cardinality: number;
}
/** @internal */
export interface KtxBigQueryQueryJob {
getQueryResults(): Promise<
[Array<Record<string, unknown>>, unknown, { schema?: { fields?: TableField[] } }?, ...unknown[]]
>;
}
/** @internal */
export interface KtxBigQueryTableRef {
id?: string;
metadata?: { type?: string };
@ -81,6 +65,7 @@ export interface KtxBigQueryTableRef {
>;
}
/** @internal */
export interface KtxBigQueryDataset {
get(): Promise<unknown>;
getTables(): Promise<[KtxBigQueryTableRef[], ...unknown[]]>;
@ -223,6 +208,7 @@ export function isKtxBigQueryConnectionConfig(
return String(connection?.driver ?? '').toLowerCase() === 'bigquery';
}
/** @internal */
export function bigQueryConnectionConfigFromConfig(input: {
connectionId: string;
connection: KtxBigQueryConnectionConfig | undefined;

View file

@ -1,4 +1,4 @@
import type { KtxSchemaDimensionType, KtxTableRef } from '@ktx/context/scan';
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
type BigQueryTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;

View file

@ -1,5 +1,5 @@
import type { LiveDatabaseIntrospectionPort } from '@ktx/context/ingest';
import type { KtxProjectConnectionConfig } from '@ktx/context/project';
import type { LiveDatabaseIntrospectionPort } from '../../context/ingest/adapters/live-database/types.js';
import type { KtxProjectConnectionConfig } from '../../context/project/config.js';
import {
KtxBigQueryScanConnector,
type KtxBigQueryClientFactory,

View file

@ -1,11 +1,6 @@
import { describe, expect, it, vi } from 'vitest';
import {
clickHouseClientConfigFromConfig,
createClickHouseLiveDatabaseIntrospection,
isKtxClickHouseConnectionConfig,
KtxClickHouseScanConnector,
type KtxClickHouseClientFactory,
} from './index.js';
import { clickHouseClientConfigFromConfig, isKtxClickHouseConnectionConfig, KtxClickHouseScanConnector, type KtxClickHouseClientFactory } from '../../connectors/clickhouse/connector.js';
import { createClickHouseLiveDatabaseIntrospection } from '../../connectors/clickhouse/live-database-introspection.js';
function result<T>(payload: T) {
return {

View file

@ -1,24 +1,6 @@
import { createClient } from '@clickhouse/client';
import { assertReadOnlySql, limitSqlForExecution } from '@ktx/context/connections';
import {
createKtxConnectorCapabilities,
type KtxColumnSampleInput,
type KtxColumnSampleResult,
type KtxColumnStatsInput,
type KtxColumnStatsResult,
type KtxQueryResult,
type KtxReadOnlyQueryInput,
type KtxScanConnector,
type KtxScanContext,
type KtxScanInput,
type KtxSchemaColumn,
type KtxSchemaSnapshot,
type KtxSchemaTable,
type KtxTableRef,
type KtxTableSampleInput,
type KtxTableListEntry,
type KtxTableSampleResult,
} from '@ktx/context/scan';
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaColumn, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableRef, type KtxTableSampleInput, type KtxTableListEntry, type KtxTableSampleResult } from '../../context/scan/types.js';
import { readFileSync } from 'node:fs';
import { Agent as HttpsAgent } from 'node:https';
import { homedir } from 'node:os';
@ -198,6 +180,7 @@ export function isKtxClickHouseConnectionConfig(
return String(connection?.driver ?? '').toLowerCase() === 'clickhouse';
}
/** @internal */
export function clickHouseClientConfigFromConfig(input: {
connectionId: string;
connection: KtxClickHouseConnectionConfig | undefined;

View file

@ -1,4 +1,4 @@
import type { KtxSchemaDimensionType, KtxTableRef } from '@ktx/context/scan';
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
type ClickHouseTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;

View file

@ -1,5 +1,5 @@
import type { LiveDatabaseIntrospectionPort } from '@ktx/context/ingest';
import type { KtxProjectConnectionConfig } from '@ktx/context/project';
import type { LiveDatabaseIntrospectionPort } from '../../context/ingest/adapters/live-database/types.js';
import type { KtxProjectConnectionConfig } from '../../context/project/config.js';
import {
KtxClickHouseScanConnector,
type KtxClickHouseClientFactory,

View file

@ -1,12 +1,7 @@
import { describe, expect, it, vi } from 'vitest';
import type { FieldPacket, RowDataPacket } from 'mysql2/promise';
import {
createMysqlLiveDatabaseIntrospection,
isKtxMysqlConnectionConfig,
KtxMysqlScanConnector,
mysqlConnectionPoolConfigFromConfig,
type KtxMysqlPoolFactory,
} from './index.js';
import { createMysqlLiveDatabaseIntrospection } from '../../connectors/mysql/live-database-introspection.js';
import { isKtxMysqlConnectionConfig, KtxMysqlScanConnector, mysqlConnectionPoolConfigFromConfig, type KtxMysqlPoolFactory } from '../../connectors/mysql/connector.js';
function mysqlResult(rows: Record<string, unknown>[], fields: Array<{ name: string; type?: number }>): [RowDataPacket[], FieldPacket[]] {
return [rows as RowDataPacket[], fields as FieldPacket[]];

View file

@ -2,27 +2,8 @@ import mysql, { type FieldPacket, type Pool, type RowDataPacket } from 'mysql2/p
import { readFileSync } from 'node:fs';
import { homedir } from 'node:os';
import { resolve } from 'node:path';
import { assertReadOnlySql, limitSqlForExecution } from '@ktx/context/connections';
import {
createKtxConnectorCapabilities,
type KtxColumnSampleInput,
type KtxColumnSampleResult,
type KtxColumnStatsInput,
type KtxColumnStatsResult,
type KtxQueryResult,
type KtxReadOnlyQueryInput,
type KtxScanConnector,
type KtxScanContext,
type KtxScanInput,
type KtxSchemaColumn,
type KtxTableListEntry,
type KtxSchemaForeignKey,
type KtxSchemaSnapshot,
type KtxSchemaTable,
type KtxTableRef,
type KtxTableSampleInput,
type KtxTableSampleResult,
} from '@ktx/context/scan';
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaColumn, type KtxTableListEntry, type KtxSchemaForeignKey, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableRef, type KtxTableSampleInput, type KtxTableSampleResult } from '../../context/scan/types.js';
import { KtxMysqlDialect } from './dialect.js';
export interface KtxMysqlConnectionConfig {
@ -237,6 +218,7 @@ export function isKtxMysqlConnectionConfig(
return String(connection?.driver ?? '').toLowerCase() === 'mysql';
}
/** @internal */
export function mysqlConnectionPoolConfigFromConfig(input: {
connectionId: string;
connection: KtxMysqlConnectionConfig | undefined;

View file

@ -1,4 +1,4 @@
import type { KtxSchemaDimensionType, KtxTableRef } from '@ktx/context/scan';
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
type MysqlTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;

View file

@ -1,5 +1,5 @@
import type { LiveDatabaseIntrospectionPort } from '@ktx/context/ingest';
import type { KtxProjectConnectionConfig } from '@ktx/context/project';
import type { LiveDatabaseIntrospectionPort } from '../../context/ingest/adapters/live-database/types.js';
import type { KtxProjectConnectionConfig } from '../../context/project/config.js';
import {
KtxMysqlScanConnector,
type KtxMysqlConnectionConfig,

View file

@ -1,11 +1,6 @@
import { describe, expect, it, vi } from 'vitest';
import {
createPostgresLiveDatabaseIntrospection,
isKtxPostgresConnectionConfig,
KtxPostgresScanConnector,
postgresPoolConfigFromConfig,
type KtxPostgresPoolFactory,
} from './index.js';
import { createPostgresLiveDatabaseIntrospection } from '../../connectors/postgres/live-database-introspection.js';
import { isKtxPostgresConnectionConfig, KtxPostgresScanConnector, postgresPoolConfigFromConfig, type KtxPostgresPoolFactory } from '../../connectors/postgres/connector.js';
interface FakeQueryResult {
rows: Record<string, unknown>[];

View file

@ -1,27 +1,8 @@
import { readFileSync } from 'node:fs';
import { homedir } from 'node:os';
import { resolve } from 'node:path';
import { assertReadOnlySql, limitSqlForExecution } from '@ktx/context/connections';
import {
createKtxConnectorCapabilities,
type KtxColumnSampleInput,
type KtxColumnSampleResult,
type KtxColumnStatsInput,
type KtxColumnStatsResult,
type KtxQueryResult,
type KtxReadOnlyQueryInput,
type KtxScanConnector,
type KtxScanContext,
type KtxScanInput,
type KtxSchemaColumn,
type KtxSchemaForeignKey,
type KtxSchemaSnapshot,
type KtxSchemaTable,
type KtxTableListEntry,
type KtxTableRef,
type KtxTableSampleInput,
type KtxTableSampleResult,
} from '@ktx/context/scan';
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaColumn, type KtxSchemaForeignKey, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableListEntry, type KtxTableRef, type KtxTableSampleInput, type KtxTableSampleResult } from '../../context/scan/types.js';
import { Pool } from 'pg';
import { KtxPostgresDialect } from './dialect.js';
@ -297,6 +278,7 @@ export function isKtxPostgresConnectionConfig(
return driver === 'postgres' || driver === 'postgresql';
}
/** @internal */
export function postgresPoolConfigFromConfig(input: {
connectionId: string;
connection: KtxPostgresConnectionConfig | undefined;

View file

@ -1,4 +1,4 @@
import type { KtxSchemaDimensionType, KtxTableRef } from '@ktx/context/scan';
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
type PostgresTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;

View file

@ -1,4 +1,4 @@
import type { KtxPostgresQueryClient } from '@ktx/context/ingest';
import type { KtxPostgresQueryClient } from '../../context/ingest/adapters/historic-sql/types.js';
import { KtxPostgresScanConnector, type KtxPostgresScanConnectorOptions } from './connector.js';
export type KtxPostgresHistoricSqlQueryClientOptions = KtxPostgresScanConnectorOptions;

View file

@ -1,5 +1,5 @@
import type { LiveDatabaseIntrospectionPort } from '@ktx/context/ingest';
import type { KtxProjectConnectionConfig } from '@ktx/context/project';
import type { LiveDatabaseIntrospectionPort } from '../../context/ingest/adapters/live-database/types.js';
import type { KtxProjectConnectionConfig } from '../../context/project/config.js';
import {
KtxPostgresScanConnector,
type KtxPostgresConnectionConfig,

View file

@ -1,12 +1,6 @@
import { describe, expect, it, vi } from 'vitest';
import {
createSnowflakeLiveDatabaseIntrospection,
isKtxSnowflakeConnectionConfig,
KtxSnowflakeScanConnector,
snowflakeConnectionConfigFromConfig,
type KtxSnowflakeDriver,
type KtxSnowflakeDriverFactory,
} from './index.js';
import { createSnowflakeLiveDatabaseIntrospection } from '../../connectors/snowflake/live-database-introspection.js';
import { isKtxSnowflakeConnectionConfig, KtxSnowflakeScanConnector, snowflakeConnectionConfigFromConfig, type KtxSnowflakeDriver, type KtxSnowflakeDriverFactory } from '../../connectors/snowflake/connector.js';
function fakeDriverFactory(): KtxSnowflakeDriverFactory {
const driver: KtxSnowflakeDriver = {

View file

@ -2,26 +2,8 @@ import { createPrivateKey } from 'node:crypto';
import { readFileSync } from 'node:fs';
import { homedir } from 'node:os';
import { resolve } from 'node:path';
import { assertReadOnlySql, limitSqlForExecution } from '@ktx/context/connections';
import {
createKtxConnectorCapabilities,
type KtxColumnSampleInput,
type KtxColumnSampleResult,
type KtxColumnStatsInput,
type KtxColumnStatsResult,
type KtxQueryResult,
type KtxReadOnlyQueryInput,
type KtxScanConnector,
type KtxScanContext,
type KtxScanInput,
type KtxSchemaColumn,
type KtxSchemaSnapshot,
type KtxSchemaTable,
type KtxTableRef,
type KtxTableSampleInput,
type KtxTableListEntry,
type KtxTableSampleResult,
} from '@ktx/context/scan';
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaColumn, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableRef, type KtxTableSampleInput, type KtxTableListEntry, type KtxTableSampleResult } from '../../context/scan/types.js';
import * as snowflake from 'snowflake-sdk';
import { KtxSnowflakeDialect } from './dialect.js';
@ -196,6 +178,7 @@ export function isKtxSnowflakeConnectionConfig(
return String(connection?.driver ?? '').toLowerCase() === 'snowflake';
}
/** @internal */
export function snowflakeConnectionConfigFromConfig(input: {
connectionId: string;
connection: KtxSnowflakeConnectionConfig | undefined;

View file

@ -1,4 +1,4 @@
import type { KtxSchemaDimensionType, KtxTableRef } from '@ktx/context/scan';
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
type SnowflakeTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;

View file

@ -1,5 +1,5 @@
import type { LiveDatabaseIntrospectionPort } from '@ktx/context/ingest';
import type { KtxProjectConnectionConfig } from '@ktx/context/project';
import type { LiveDatabaseIntrospectionPort } from '../../context/ingest/adapters/live-database/types.js';
import type { KtxProjectConnectionConfig } from '../../context/project/config.js';
import {
KtxSnowflakeScanConnector,
type KtxSnowflakeConnectionConfig,

View file

@ -4,12 +4,8 @@ import { mkdtemp, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import {
createSqliteLiveDatabaseIntrospection,
isKtxSqliteConnectionConfig,
KtxSqliteScanConnector,
sqliteDatabasePathFromConfig,
} from './index.js';
import { createSqliteLiveDatabaseIntrospection } from '../../connectors/sqlite/live-database-introspection.js';
import { isKtxSqliteConnectionConfig, KtxSqliteScanConnector, sqliteDatabasePathFromConfig } from '../../connectors/sqlite/connector.js';
describe('KtxSqliteScanConnector', () => {
let tempDir: string;

View file

@ -3,25 +3,9 @@ import { existsSync, readFileSync, statSync } from 'node:fs';
import { homedir } from 'node:os';
import { isAbsolute, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import { assertReadOnlySql, limitSqlForExecution, normalizeQueryRows } from '@ktx/context/connections';
import {
createKtxConnectorCapabilities,
type KtxColumnSampleInput,
type KtxColumnSampleResult,
type KtxColumnStatsInput,
type KtxColumnStatsResult,
type KtxQueryResult,
type KtxReadOnlyQueryInput,
type KtxScanConnector,
type KtxScanContext,
type KtxScanInput,
type KtxSchemaForeignKey,
type KtxSchemaSnapshot,
type KtxSchemaTable,
type KtxTableRef,
type KtxTableSampleInput,
type KtxTableSampleResult,
} from '@ktx/context/scan';
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
import { normalizeQueryRows } from '../../context/connections/query-executor.js';
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaForeignKey, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableRef, type KtxTableSampleInput, type KtxTableSampleResult } from '../../context/scan/types.js';
import { KtxSqliteDialect } from './dialect.js';
export interface KtxSqliteConnectionConfig {
@ -31,6 +15,7 @@ export interface KtxSqliteConnectionConfig {
[key: string]: unknown;
}
/** @internal */
export interface SqliteDatabasePathInput {
connectionId: string;
projectDir?: string;
@ -142,6 +127,7 @@ export function isKtxSqliteConnectionConfig(
return driver === 'sqlite' || driver === 'sqlite3';
}
/** @internal */
export function sqliteDatabasePathFromConfig(input: SqliteDatabasePathInput): string {
const inputDriver = input.connection?.driver ?? 'unknown';
if (!isKtxSqliteConnectionConfig(input.connection)) {

View file

@ -1,4 +1,4 @@
import type { KtxSchemaDimensionType, KtxTableRef } from '@ktx/context/scan';
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
type SqliteTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;

View file

@ -1,5 +1,5 @@
import type { LiveDatabaseIntrospectionPort } from '@ktx/context/ingest';
import type { KtxProjectConnectionConfig } from '@ktx/context/project';
import type { LiveDatabaseIntrospectionPort } from '../../context/ingest/adapters/live-database/types.js';
import type { KtxProjectConnectionConfig } from '../../context/project/config.js';
import { KtxSqliteScanConnector, type KtxSqliteConnectionConfig } from './connector.js';
export interface CreateSqliteLiveDatabaseIntrospectionOptions {

View file

@ -1,12 +1,6 @@
import { describe, expect, it, vi } from 'vitest';
import {
createSqlServerLiveDatabaseIntrospection,
isKtxSqlServerConnectionConfig,
KtxSqlServerScanConnector,
sqlServerConnectionPoolConfigFromConfig,
type KtxSqlServerPoolFactory,
type KtxSqlServerQueryResult,
} from './index.js';
import { createSqlServerLiveDatabaseIntrospection } from '../../connectors/sqlserver/live-database-introspection.js';
import { isKtxSqlServerConnectionConfig, KtxSqlServerScanConnector, sqlServerConnectionPoolConfigFromConfig, type KtxSqlServerPoolFactory, type KtxSqlServerQueryResult } from '../../connectors/sqlserver/connector.js';
function recordset<T extends Record<string, unknown>>(
rows: T[],

View file

@ -1,24 +1,5 @@
import { assertReadOnlySql } from '@ktx/context/connections';
import {
createKtxConnectorCapabilities,
type KtxColumnSampleInput,
type KtxColumnSampleResult,
type KtxColumnStatsInput,
type KtxColumnStatsResult,
type KtxQueryResult,
type KtxReadOnlyQueryInput,
type KtxScanConnector,
type KtxScanContext,
type KtxScanInput,
type KtxSchemaColumn,
type KtxSchemaForeignKey,
type KtxSchemaSnapshot,
type KtxSchemaTable,
type KtxTableListEntry,
type KtxTableRef,
type KtxTableSampleInput,
type KtxTableSampleResult,
} from '@ktx/context/scan';
import { assertReadOnlySql } from '../../context/connections/read-only-sql.js';
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaColumn, type KtxSchemaForeignKey, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableListEntry, type KtxTableRef, type KtxTableSampleInput, type KtxTableSampleResult } from '../../context/scan/types.js';
import { readFileSync } from 'node:fs';
import { homedir } from 'node:os';
import { resolve } from 'node:path';
@ -50,6 +31,7 @@ export interface KtxSqlServerPoolConfig {
pool: { max: number; min: number; idleTimeoutMillis: number };
}
/** @internal */
export interface KtxSqlServerQueryResult {
recordset?: Array<Record<string, unknown>> & { columns?: Record<string, { type?: { declaration?: string } }> };
}
@ -239,6 +221,7 @@ export function isKtxSqlServerConnectionConfig(
return String(connection?.driver ?? '').toLowerCase() === 'sqlserver';
}
/** @internal */
export function sqlServerConnectionPoolConfigFromConfig(input: {
connectionId: string;
connection: KtxSqlServerConnectionConfig | undefined;

View file

@ -1,4 +1,4 @@
import type { KtxSchemaDimensionType, KtxTableRef } from '@ktx/context/scan';
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
type SqlServerTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;

View file

@ -1,5 +1,5 @@
import type { LiveDatabaseIntrospectionPort } from '@ktx/context/ingest';
import type { KtxProjectConnectionConfig } from '@ktx/context/project';
import type { LiveDatabaseIntrospectionPort } from '../../context/ingest/adapters/live-database/types.js';
import type { KtxProjectConnectionConfig } from '../../context/project/config.js';
import {
KtxSqlServerScanConnector,
type KtxSqlServerConnectionConfig,

View file

@ -1,4 +1,4 @@
import { buildDefaultKtxProjectConfig, type KtxProjectConfig } from '@ktx/context/project';
import { buildDefaultKtxProjectConfig, type KtxProjectConfig } from './context/project/config.js';
import { describe, expect, it, vi } from 'vitest';
import type { KtxPublicIngestProject, KtxPublicIngestTargetResult } from './public-ingest.js';
import {

View file

@ -1,4 +1,4 @@
import type { KtxProgressPort, KtxProgressUpdateOptions } from '@ktx/context/scan';
import type { KtxProgressPort, KtxProgressUpdateOptions } from './context/scan/types.js';
import type { KtxCliIo } from './index.js';
import type { KtxIngestProgressUpdate } from './ingest.js';
import type { KtxManagedPythonInstallPolicy } from './managed-python-command.js';
@ -444,17 +444,20 @@ export function renderContextBuildView(
const ESC_K_RE = new RegExp(`${ESC.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}\\[K`, 'g');
const ANSI_RE = /\x1b\[[0-9;]*m/g;
/** @internal */
export function extractProgressMessage(chunk: string): string | null {
const cleaned = chunk.replace(/^\r/, '').replace(ESC_K_RE, '').replace(/\n$/, '').trim();
const match = cleaned.match(/^\[(\d+)%\]\s*(.+)$/);
return match ? `[${match[1]}%] ${match[2]}` : null;
}
/** @internal */
export function parseScanSummary(output: string): string | null {
const match = output.match(/(\d+) changes? across (\d+) tables?/);
return match ? `${match[2]} tables` : null;
}
/** @internal */
export function parseIngestSummary(output: string): string | null {
const savedMemory = output.match(/Saved memory: (.+)/);
if (savedMemory) return savedMemory[1];
@ -560,6 +563,7 @@ function collectSourceProgress(targets: ContextBuildTargetState[]): ContextBuild
});
}
/** @internal */
export function viewStateFromSourceProgress(
sources: ContextBuildSourceProgressUpdate[],
now: number,

View file

@ -1,6 +1,6 @@
import type { KtxSchemaDimensionType, KtxTableRef } from '../scan/types.js';
export type SupportedDriver =
type SupportedDriver =
| 'postgres'
| 'postgresql'
| 'mysql'

View file

@ -8,7 +8,7 @@ import {
} from '../ingest/adapters/notion/types.js';
import type { KtxProjectConnectionConfig } from '../project/config.js';
export const KTX_NOTION_ORG_KNOWLEDGE_WARNING =
const KTX_NOTION_ORG_KNOWLEDGE_WARNING =
'Anything accessible to this Notion integration can become organization knowledge.';
type KtxNotionCrawlMode = 'all_accessible' | 'selected_roots';
@ -39,6 +39,7 @@ export type KtxNotionConnectionConfig = Omit<
max_knowledge_updates_per_run: number;
};
/** @internal */
export interface RedactedKtxNotionConnectionConfig {
driver: 'notion';
hasAuthToken: boolean;
@ -152,6 +153,7 @@ export function parseNotionConnectionConfig(raw: unknown): KtxNotionConnectionCo
};
}
/** @internal */
export function redactNotionConnectionConfig(config: KtxNotionConnectionConfig): RedactedKtxNotionConnectionConfig {
return {
driver: 'notion',
@ -171,6 +173,7 @@ function expandHome(path: string): string {
return path === '~' || path.startsWith('~/') ? resolve(homedir(), path.slice(2)) : path;
}
/** @internal */
export async function resolveNotionAuthToken(
authTokenRef: string,
options: ResolveNotionTokenOptions = {},

View file

@ -1,4 +1,4 @@
import type { KtxProjectConnectionConfig } from '../project/index.js';
import type { KtxProjectConnectionConfig } from '../../context/project/config.js';
export interface KtxSqlQueryExecutionInput {
connectionId: string;

View file

@ -49,6 +49,7 @@ function sqlitePathFromUrl(url: string): string {
return url;
}
/** @internal */
export function sqliteDatabasePathFromConnection(input: KtxSqlQueryExecutionInput): string {
const driver = connectionDriver(input);
if (driver !== 'sqlite' && driver !== 'sqlite3') {

View file

@ -2,6 +2,7 @@ import { readFileSync } from 'node:fs';
import { homedir } from 'node:os';
import { resolve } from 'node:path';
/** @internal */
export function resolveKtxHomePath(path: string): string {
if (path === '~') {
return homedir();

View file

@ -1,10 +1,10 @@
export interface KtxStorageConfig {
interface KtxStorageConfig {
configDir?: string;
homeDir?: string;
worktreesDir?: string;
}
export interface KtxGitConfig {
interface KtxGitConfig {
userName: string;
userEmail: string;
bootstrapMessage?: string;

View file

@ -1,3 +1,4 @@
/** @internal */
export const REDACTED_KTX_CREDENTIAL_VALUE = '<redacted>';
const SENSITIVE_FIELD_NAME = /(password|secret|token|api[_-]?key|private[_-]?key|passphrase|credential|authorization|url)/i;

View file

@ -5,7 +5,7 @@ import { GitService } from './git.service.js';
export type SessionOutcome = 'success' | 'empty' | 'conflict' | 'crash';
export interface SentinelPayload {
interface SentinelPayload {
outcome: SessionOutcome;
at: string;
chatId: string;

View file

@ -4,21 +4,21 @@ import { URL } from 'node:url';
import { spawn } from 'node:child_process';
import type { ResolvedSemanticLayerSource, SemanticLayerQueryInput } from '../sl/types.js';
export interface KtxSemanticLayerComputeQueryResult {
interface KtxSemanticLayerComputeQueryResult {
sql: string;
dialect: string;
columns: Array<Record<string, unknown>>;
plan: Record<string, unknown>;
}
export interface KtxSemanticLayerComputeValidationResult {
interface KtxSemanticLayerComputeValidationResult {
valid: boolean;
errors: string[];
warnings: string[];
perSourceWarnings: Record<string, string[]>;
}
export interface KtxSemanticLayerSourceGenerationColumnInput {
interface KtxSemanticLayerSourceGenerationColumnInput {
name: string;
type: string;
primaryKey?: boolean;
@ -26,7 +26,7 @@ export interface KtxSemanticLayerSourceGenerationColumnInput {
comment?: string | null;
}
export interface KtxSemanticLayerSourceGenerationTableInput {
interface KtxSemanticLayerSourceGenerationTableInput {
name: string;
catalog?: string | null;
db?: string | null;
@ -34,7 +34,7 @@ export interface KtxSemanticLayerSourceGenerationTableInput {
columns: KtxSemanticLayerSourceGenerationColumnInput[];
}
export interface KtxSemanticLayerSourceGenerationLinkInput {
interface KtxSemanticLayerSourceGenerationLinkInput {
fromTable: string;
fromColumn: string;
toTable: string;
@ -42,13 +42,13 @@ export interface KtxSemanticLayerSourceGenerationLinkInput {
relationshipType: string;
}
export interface KtxSemanticLayerSourceGenerationInput {
interface KtxSemanticLayerSourceGenerationInput {
tables: KtxSemanticLayerSourceGenerationTableInput[];
links: KtxSemanticLayerSourceGenerationLinkInput[];
dialect?: string;
}
export interface KtxSemanticLayerSourceGenerationResult {
interface KtxSemanticLayerSourceGenerationResult {
sources: Array<Record<string, unknown>>;
sourceCount: number;
}
@ -75,14 +75,14 @@ export interface KtxSemanticLayerComputePort {
generateSources(input: KtxSemanticLayerSourceGenerationInput): Promise<KtxSemanticLayerSourceGenerationResult>;
}
export type KtxDaemonCommand = 'semantic-query' | 'semantic-validate' | 'semantic-generate-sources';
type KtxDaemonCommand = 'semantic-query' | 'semantic-validate' | 'semantic-generate-sources';
export type KtxDaemonJsonRunner = (
type KtxDaemonJsonRunner = (
subcommand: KtxDaemonCommand,
payload: Record<string, unknown>,
) => Promise<Record<string, unknown>>;
export type KtxDaemonHttpJsonRunner = (path: string, payload: Record<string, unknown>) => Promise<Record<string, unknown>>;
type KtxDaemonHttpJsonRunner = (path: string, payload: Record<string, unknown>) => Promise<Record<string, unknown>>;
export interface PythonSemanticLayerComputeOptions {
command?: string;
@ -92,6 +92,7 @@ export interface PythonSemanticLayerComputeOptions {
runJson?: KtxDaemonJsonRunner;
}
/** @internal */
export interface HttpSemanticLayerComputeOptions {
baseUrl: string;
requestJson?: KtxDaemonHttpJsonRunner;
@ -272,6 +273,7 @@ export function createPythonSemanticLayerComputePort(
};
}
/** @internal */
export function createHttpSemanticLayerComputePort(
options: HttpSemanticLayerComputeOptions,
): KtxSemanticLayerComputePort {

View file

@ -2,8 +2,8 @@ import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import type { KtxEmbeddingPort } from '../core/index.js';
import { initKtxProject, loadKtxProject, type KtxLocalProject } from '../project/index.js';
import type { KtxEmbeddingPort } from '../../context/core/embedding.js';
import { initKtxProject, loadKtxProject, type KtxLocalProject } from '../../context/project/project.js';
import { SqliteKnowledgeIndex } from '../wiki/sqlite-knowledge-index.js';
import { reindexLocalIndexes } from './reindex.js';

Some files were not shown because too many files have changed in this diff Show more