mirror of
https://github.com/Kaelio/ktx.git
synced 2026-06-07 07:55:13 +02:00
chore(workspace): gate dead-code with knip production mode (#196)
* refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
This commit is contained in:
parent
a1cfb03d73
commit
2366b00301
1002 changed files with 2286 additions and 12051 deletions
34
.github/workflows/ci.yml
vendored
34
.github/workflows/ci.yml
vendored
|
|
@ -83,36 +83,8 @@ jobs:
|
||||||
- name: Run TypeScript checks
|
- name: Run TypeScript checks
|
||||||
run: pnpm run check
|
run: pnpm run check
|
||||||
|
|
||||||
slow-context-tests:
|
|
||||||
name: Slow context tests
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
steps:
|
|
||||||
- name: Checkout repository
|
|
||||||
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
||||||
|
|
||||||
- name: Setup pnpm
|
|
||||||
uses: pnpm/action-setup@0e279bb959325dab635dd2c09392533439d90093 # v6.0.8
|
|
||||||
with:
|
|
||||||
run_install: false
|
|
||||||
|
|
||||||
- name: Setup Node.js
|
|
||||||
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
|
||||||
with:
|
|
||||||
node-version: "24"
|
|
||||||
cache: "pnpm"
|
|
||||||
cache-dependency-path: "pnpm-lock.yaml"
|
|
||||||
|
|
||||||
- name: Install TypeScript dependencies
|
|
||||||
run: pnpm install --frozen-lockfile
|
|
||||||
|
|
||||||
- name: Build TypeScript packages
|
|
||||||
run: pnpm run build
|
|
||||||
|
|
||||||
- name: Run slow context tests
|
|
||||||
run: pnpm --filter @ktx/context run test:slow
|
|
||||||
|
|
||||||
slow-cli-tests:
|
slow-cli-tests:
|
||||||
name: Slow CLI tests
|
name: Slow TypeScript tests
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
steps:
|
steps:
|
||||||
- name: Checkout repository
|
- name: Checkout repository
|
||||||
|
|
@ -137,7 +109,7 @@ jobs:
|
||||||
run: pnpm run build
|
run: pnpm run build
|
||||||
|
|
||||||
- name: Run slow CLI tests
|
- name: Run slow CLI tests
|
||||||
run: pnpm --filter @ktx/cli run test:slow
|
run: pnpm --filter @kaelio/ktx run test:slow
|
||||||
|
|
||||||
cli-smoke-tests:
|
cli-smoke-tests:
|
||||||
name: CLI smoke tests
|
name: CLI smoke tests
|
||||||
|
|
@ -236,7 +208,7 @@ jobs:
|
||||||
with:
|
with:
|
||||||
token: ${{ secrets.CODECOV_TOKEN }}
|
token: ${{ secrets.CODECOV_TOKEN }}
|
||||||
slug: Kaelio/ktx
|
slug: Kaelio/ktx
|
||||||
files: ./packages/cli/coverage/lcov.info,./packages/connector-bigquery/coverage/lcov.info,./packages/connector-clickhouse/coverage/lcov.info,./packages/connector-mysql/coverage/lcov.info,./packages/connector-postgres/coverage/lcov.info,./packages/connector-snowflake/coverage/lcov.info,./packages/connector-sqlite/coverage/lcov.info,./packages/connector-sqlserver/coverage/lcov.info,./packages/context/coverage/lcov.info,./packages/llm/coverage/lcov.info
|
files: ./packages/cli/coverage/lcov.info
|
||||||
flags: typescript
|
flags: typescript
|
||||||
name: typescript
|
name: typescript
|
||||||
disable_search: true
|
disable_search: true
|
||||||
|
|
|
||||||
|
|
@ -41,8 +41,13 @@ repos:
|
||||||
language: system
|
language: system
|
||||||
pass_filenames: false
|
pass_filenames: false
|
||||||
- id: knip-dead-code
|
- id: knip-dead-code
|
||||||
name: knip dead-code check
|
name: knip dead-code (auto-fix)
|
||||||
entry: pnpm exec knip --reporter compact
|
entry: pnpm exec knip --fix --reporter compact
|
||||||
|
language: system
|
||||||
|
pass_filenames: false
|
||||||
|
- id: knip-dead-code-production
|
||||||
|
name: knip dead-code (production mode)
|
||||||
|
entry: pnpm exec knip --production --reporter compact
|
||||||
language: system
|
language: system
|
||||||
pass_filenames: false
|
pass_filenames: false
|
||||||
|
|
||||||
|
|
|
||||||
59
AGENTS.md
59
AGENTS.md
|
|
@ -1,6 +1,6 @@
|
||||||
# ktx Development Notes
|
# ktx Development Notes
|
||||||
|
|
||||||
**ktx** is a standalone open-source context layer for database agents. These
|
**ktx** is a standalone open-source context layer for data agents. These
|
||||||
instructions apply to all agents working in this repository (Codex, Claude,
|
instructions apply to all agents working in this repository (Codex, Claude,
|
||||||
Gemini, and similar tools). Do not assume an external app server, frontend,
|
Gemini, and similar tools). Do not assume an external app server, frontend,
|
||||||
database migrations, ORPC contracts, or `python-service/` layout exist here.
|
database migrations, ORPC contracts, or `python-service/` layout exist here.
|
||||||
|
|
@ -63,11 +63,10 @@ When rules conflict, follow this order:
|
||||||
|
|
||||||
**ktx** is a pnpm + uv workspace.
|
**ktx** is a pnpm + uv workspace.
|
||||||
|
|
||||||
- TypeScript packages: `packages/*`
|
- TypeScript package: `packages/cli` (the sole npm-published package source)
|
||||||
- CLI package: `packages/cli`
|
- Core context modules: `packages/cli/src/context/`
|
||||||
- Core context package: `packages/context`
|
- LLM provider modules: `packages/cli/src/llm/`
|
||||||
- LLM package: `packages/llm`
|
- Database connector modules: `packages/cli/src/connectors/<driver>/`
|
||||||
- Database connectors: `packages/connector-*`
|
|
||||||
- Python semantic layer: `python/ktx-sl`
|
- Python semantic layer: `python/ktx-sl`
|
||||||
- **ktx** daemon: `python/ktx-daemon`
|
- **ktx** daemon: `python/ktx-daemon`
|
||||||
- Examples and fixtures: `examples/`
|
- Examples and fixtures: `examples/`
|
||||||
|
|
@ -76,9 +75,8 @@ When rules conflict, follow this order:
|
||||||
commit `.agents/`, `.claude/`, or `docs/superpowers/` to this public
|
commit `.agents/`, `.claude/`, or `docs/superpowers/` to this public
|
||||||
repository.
|
repository.
|
||||||
|
|
||||||
Some package names still contain `ktx` during the split. Do not mass-rename
|
Some source identifiers still contain historical package-oriented names. Do not
|
||||||
symbols, package names, paths, or docs to `ktx` unless the task asks for that
|
mass-rename symbols, paths, or docs unless the task asks for that rename.
|
||||||
rename.
|
|
||||||
|
|
||||||
## Quick Commands
|
## Quick Commands
|
||||||
|
|
||||||
|
|
@ -91,7 +89,7 @@ pnpm run type-check
|
||||||
pnpm run test
|
pnpm run test
|
||||||
pnpm run check
|
pnpm run check
|
||||||
pnpm run dead-code
|
pnpm run dead-code
|
||||||
pnpm --filter @ktx/cli run smoke
|
pnpm --filter @kaelio/ktx run smoke
|
||||||
pnpm --filter './packages/*' run build
|
pnpm --filter './packages/*' run build
|
||||||
pnpm --filter './packages/*' run test
|
pnpm --filter './packages/*' run test
|
||||||
pnpm --filter './packages/*' run type-check
|
pnpm --filter './packages/*' run type-check
|
||||||
|
|
@ -165,11 +163,11 @@ rules there with the same weight as the ones in this file.
|
||||||
- Keep package exports, `types`, and built `dist` expectations aligned when
|
- Keep package exports, `types`, and built `dist` expectations aligned when
|
||||||
changing public APIs.
|
changing public APIs.
|
||||||
- Use `zod` schemas for runtime validation at CLI/config/API boundaries.
|
- Use `zod` schemas for runtime validation at CLI/config/API boundaries.
|
||||||
- Keep connector packages thin: connector-specific scanning/auth behavior
|
- Keep connector modules thin: connector-specific scanning/auth behavior
|
||||||
belongs in `packages/connector-*`; shared types and orchestration belong in
|
belongs in `packages/cli/src/connectors/<driver>/`; shared types and
|
||||||
`packages/context`.
|
orchestration belong in `packages/cli/src/context/`.
|
||||||
- Avoid circular package dependencies. Shared code should move to the lowest
|
- Avoid circular module dependencies. Shared code should move to the lowest
|
||||||
sensible package, not be duplicated across connectors.
|
sensible module, not be duplicated across connectors.
|
||||||
- Do not manually edit generated or built output under `dist/`; edit source and
|
- Do not manually edit generated or built output under `dist/`; edit source and
|
||||||
rebuild.
|
rebuild.
|
||||||
|
|
||||||
|
|
@ -179,7 +177,16 @@ rules there with the same weight as the ones in this file.
|
||||||
analysis. These checks are intentionally part of CI and pre-commit because the
|
analysis. These checks are intentionally part of CI and pre-commit because the
|
||||||
normal development workflow is agent-based.
|
normal development workflow is agent-based.
|
||||||
|
|
||||||
- Run `pnpm run dead-code` after TypeScript changes.
|
- `pnpm run dead-code` runs three checks: Biome (`dead-code:biome`), Knip
|
||||||
|
default-mode (`dead-code:knip`), and Knip production-mode
|
||||||
|
(`dead-code:knip:production`). All three must pass.
|
||||||
|
- Default-mode Knip catches dead code reachable from no entry at all (broken
|
||||||
|
graph). Production-mode Knip catches code reachable only via tests —
|
||||||
|
i.e. code that's tested but doesn't ship.
|
||||||
|
- Pre-commit runs `knip --fix` (auto-removes the `export` keyword from
|
||||||
|
symbols that are exported but unused) plus `knip --production` (alerts on
|
||||||
|
test-only paths). CI runs the same checks without `--fix` and fails on any
|
||||||
|
finding.
|
||||||
- Treat Knip findings as investigation prompts, not automatic deletion orders.
|
- Treat Knip findings as investigation prompts, not automatic deletion orders.
|
||||||
- Remove private dead code when you confirm there are no imports, dynamic
|
- Remove private dead code when you confirm there are no imports, dynamic
|
||||||
references, generated references, or tests that still need it.
|
references, generated references, or tests that still need it.
|
||||||
|
|
@ -190,6 +197,26 @@ normal development workflow is agent-based.
|
||||||
- Update `knip.json` when adding dynamic entrypoints, generated files, package
|
- Update `knip.json` when adding dynamic entrypoints, generated files, package
|
||||||
exports, CLI bins, or framework files that Knip cannot infer.
|
exports, CLI bins, or framework files that Knip cannot infer.
|
||||||
|
|
||||||
|
#### Internal exports for testability
|
||||||
|
|
||||||
|
When a function, type, or constant must be exported solely so a unit test can
|
||||||
|
import it (i.e. it has no production cross-file consumer), annotate the
|
||||||
|
declaration with `/** @internal */` JSDoc. Knip's production-mode check
|
||||||
|
ignores `@internal` exports, so the convention keeps the gate clean without
|
||||||
|
silencing the rest of the file.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
/** @internal */
|
||||||
|
export function reindexHasErrors(result: ReindexResult): boolean { ... }
|
||||||
|
```
|
||||||
|
|
||||||
|
Do NOT use Vitest in-source testing (`if (import.meta.vitest)` blocks). Keep
|
||||||
|
tests in separate `*.test.ts(x)` files.
|
||||||
|
|
||||||
|
If the only consumer of an export is its own test and the underlying behavior
|
||||||
|
isn't used in production, delete both the export AND the test — testing dead
|
||||||
|
code is still dead code.
|
||||||
|
|
||||||
### CLI Standards
|
### CLI Standards
|
||||||
|
|
||||||
- Use Commander for CLI command trees, arguments, options, help text, custom
|
- Use Commander for CLI command trees, arguments, options, help text, custom
|
||||||
|
|
|
||||||
22
README.md
22
README.md
|
|
@ -153,7 +153,7 @@ ktx wiki "refund policy" --json
|
||||||
ktx sl query --connection-id warehouse --measure orders.revenue --format sql
|
ktx sl query --connection-id warehouse --measure orders.revenue --format sql
|
||||||
```
|
```
|
||||||
|
|
||||||
During setup, choose **Ask data questions with ktx MCP** for client agents.
|
During setup, choose **Ask data questions with ktx MCP** for agent clients.
|
||||||
Choose **Ask data questions + manage ktx with CLI commands** when an operator
|
Choose **Ask data questions + manage ktx with CLI commands** when an operator
|
||||||
agent also needs pinned `ktx` admin commands.
|
agent also needs pinned `ktx` admin commands.
|
||||||
|
|
||||||
|
|
@ -162,20 +162,14 @@ commands to run. If the output includes `ktx mcp start --project-dir ...`, run
|
||||||
it before opening your agent. Claude Desktop uses its own launcher and prints
|
it before opening your agent. Claude Desktop uses its own launcher and prints
|
||||||
separate skill upload steps under `.ktx/agents/claude/`.
|
separate skill upload steps under `.ktx/agents/claude/`.
|
||||||
|
|
||||||
## Workspace packages
|
## Workspace layout
|
||||||
|
|
||||||
| Package | Purpose |
|
| Path | Purpose |
|
||||||
|---------|---------|
|
|------|---------|
|
||||||
| `packages/cli` | CLI entry point |
|
| `packages/cli` | TypeScript CLI package and published npm package source |
|
||||||
| `packages/context` | Core context engine |
|
| `packages/cli/src/context` | Core context engine |
|
||||||
| `packages/llm` | LLM and embedding providers |
|
| `packages/cli/src/llm` | LLM and embedding providers |
|
||||||
| `packages/connector-bigquery` | BigQuery scan connector |
|
| `packages/cli/src/connectors` | Database scan connectors |
|
||||||
| `packages/connector-clickhouse` | ClickHouse scan connector |
|
|
||||||
| `packages/connector-mysql` | MySQL scan connector |
|
|
||||||
| `packages/connector-postgres` | Postgres scan connector |
|
|
||||||
| `packages/connector-snowflake` | Snowflake scan connector |
|
|
||||||
| `packages/connector-sqlite` | SQLite scan connector |
|
|
||||||
| `packages/connector-sqlserver` | SQL Server scan connector |
|
|
||||||
| `python/ktx-sl` | Semantic-layer query planning |
|
| `python/ktx-sl` | Semantic-layer query planning |
|
||||||
| `python/ktx-daemon` | Portable compute service |
|
| `python/ktx-daemon` | Portable compute service |
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -124,7 +124,7 @@ const agent: AgentNode = {
|
||||||
position: { x: AGENT_X, y: AGENT_Y },
|
position: { x: AGENT_X, y: AGENT_Y },
|
||||||
data: {
|
data: {
|
||||||
variant: "single",
|
variant: "single",
|
||||||
title: "Analytics agent",
|
title: "Data agent",
|
||||||
subtitle:
|
subtitle:
|
||||||
"Asks: monthly net revenue and open tickets per segment, high-value orders only, no test customers",
|
"Asks: monthly net revenue and open tickets per segment, high-value orders only, no test customers",
|
||||||
},
|
},
|
||||||
|
|
|
||||||
|
|
@ -22,8 +22,8 @@ documentation, connector coverage, and examples.
|
||||||
| Area | Good first context |
|
| Area | Good first context |
|
||||||
|------|--------------------|
|
|------|--------------------|
|
||||||
| CLI and setup | `packages/cli`, especially setup steps, command definitions, status checks, and smoke tests |
|
| CLI and setup | `packages/cli`, especially setup steps, command definitions, status checks, and smoke tests |
|
||||||
| Context engine | `packages/context`, including project config, ingest orchestration, and semantic search |
|
| Context engine | `packages/cli/src/context`, including project config, ingest orchestration, and semantic search |
|
||||||
| Connectors | `packages/connector-*`, plus connector-specific tests and integration docs |
|
| Connectors | `packages/cli/src/connectors/<driver>`, plus connector-specific tests and integration docs |
|
||||||
| Python semantic layer | `python/ktx-sl` for planning and SQL compilation |
|
| Python semantic layer | `python/ktx-sl` for planning and SQL compilation |
|
||||||
| **ktx** daemon | `python/ktx-daemon` for the portable runtime API |
|
| **ktx** daemon | `python/ktx-daemon` for the portable runtime API |
|
||||||
| Documentation | `docs-site/content/docs` for public docs and `docs-site/tests` for docs behavior |
|
| Documentation | `docs-site/content/docs` for public docs and `docs-site/tests` for docs behavior |
|
||||||
|
|
@ -50,7 +50,7 @@ pnpm install
|
||||||
uv sync --all-groups
|
uv sync --all-groups
|
||||||
```
|
```
|
||||||
|
|
||||||
`pnpm install` sets up all TypeScript packages in the workspace.
|
`pnpm install` sets up the TypeScript workspace.
|
||||||
`uv sync --all-groups` installs Python dependencies for the semantic layer and
|
`uv sync --all-groups` installs Python dependencies for the semantic layer and
|
||||||
daemon, including dev and test groups.
|
daemon, including dev and test groups.
|
||||||
|
|
||||||
|
|
@ -60,11 +60,10 @@ daemon, including dev and test groups.
|
||||||
pnpm run build
|
pnpm run build
|
||||||
```
|
```
|
||||||
|
|
||||||
This builds all TypeScript packages. You can also build individual packages:
|
This builds the TypeScript package. You can also build the package directly:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pnpm --filter @ktx/cli run build
|
pnpm --filter @kaelio/ktx run build
|
||||||
pnpm --filter @ktx/context run build
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Link the CLI for local testing
|
### Link the CLI for local testing
|
||||||
|
|
@ -80,21 +79,15 @@ changes.
|
||||||
|
|
||||||
## Repository structure
|
## Repository structure
|
||||||
|
|
||||||
**ktx** is a pnpm + uv workspace. TypeScript packages live in `packages/`, Python
|
**ktx** is a pnpm + uv workspace. TypeScript source lives in `packages/cli`,
|
||||||
projects in `python/`.
|
and Python projects live in `python/`.
|
||||||
|
|
||||||
```text
|
```text
|
||||||
packages/
|
packages/
|
||||||
cli/ # CLI entry point and commands
|
cli/ # CLI package and published npm package source
|
||||||
context/ # Core context engine (scan, ingest, MCP, semantic layer)
|
src/context/ # Core context engine (scan, ingest, MCP, semantic layer)
|
||||||
llm/ # LLM client abstraction
|
src/llm/ # LLM client abstraction
|
||||||
connector-postgres/ # PostgreSQL connector
|
src/connectors/ # Database connectors
|
||||||
connector-snowflake/ # Snowflake connector
|
|
||||||
connector-bigquery/ # BigQuery connector
|
|
||||||
connector-mysql/ # MySQL connector
|
|
||||||
connector-sqlserver/ # SQL Server connector
|
|
||||||
connector-sqlite/ # SQLite connector
|
|
||||||
connector-posthog/ # PostHog connector
|
|
||||||
|
|
||||||
python/
|
python/
|
||||||
ktx-sl/ # Semantic layer - grain-aware query planning and SQL compilation
|
ktx-sl/ # Semantic layer - grain-aware query planning and SQL compilation
|
||||||
|
|
@ -105,7 +98,7 @@ scripts/ # Workspace scripts (benchmarks, verification, release)
|
||||||
docs-site/ # Documentation site (Fumadocs)
|
docs-site/ # Documentation site (Fumadocs)
|
||||||
```
|
```
|
||||||
|
|
||||||
All TypeScript packages are ESM (`"type": "module"`) and use `NodeNext` module
|
The TypeScript package is ESM (`"type": "module"`) and uses `NodeNext` module
|
||||||
resolution. The Python projects use `pyproject.toml` for dependency management.
|
resolution. The Python projects use `pyproject.toml` for dependency management.
|
||||||
|
|
||||||
## Running tests
|
## Running tests
|
||||||
|
|
@ -116,18 +109,17 @@ resolution. The Python projects use `pyproject.toml` for dependency management.
|
||||||
# Run all tests
|
# Run all tests
|
||||||
pnpm run test
|
pnpm run test
|
||||||
|
|
||||||
# Run tests for a specific package
|
# Run tests for the TypeScript package
|
||||||
pnpm --filter @ktx/cli run test
|
pnpm --filter @kaelio/ktx run test
|
||||||
pnpm --filter @ktx/context run test
|
|
||||||
|
|
||||||
# Type-check all packages
|
# Type-check all packages
|
||||||
pnpm run type-check
|
pnpm run type-check
|
||||||
|
|
||||||
# Type-check a specific package
|
# Type-check the TypeScript package
|
||||||
pnpm --filter @ktx/context run type-check
|
pnpm --filter @kaelio/ktx run type-check
|
||||||
|
|
||||||
# CLI smoke test
|
# CLI smoke test
|
||||||
pnpm --filter @ktx/cli run smoke
|
pnpm --filter @kaelio/ktx run smoke
|
||||||
```
|
```
|
||||||
|
|
||||||
### Python
|
### Python
|
||||||
|
|
@ -164,43 +156,22 @@ uv run pytest -q
|
||||||
|
|
||||||
## Adding a connector
|
## Adding a connector
|
||||||
|
|
||||||
Database connectors live in `packages/connector-<name>/`. Each connector
|
Database connectors live in `packages/cli/src/connectors/<driver>/`. Each
|
||||||
implements the `KtxScanConnector` interface from `@ktx/context`.
|
connector implements the `KtxScanConnector` interface from the internal context
|
||||||
|
modules.
|
||||||
|
|
||||||
### Step 1: Scaffold the package
|
### Step 1: Scaffold the connector
|
||||||
|
|
||||||
Create a new directory at `packages/connector-<name>/` with:
|
Create a new directory at `packages/cli/src/connectors/<driver>/` with:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
packages/connector-<name>/
|
packages/cli/src/connectors/<driver>/
|
||||||
package.json
|
index.ts # Internal connector exports
|
||||||
tsconfig.json
|
connector.ts # KtxScanConnector implementation
|
||||||
src/
|
dialect.ts # SQL dialect handling
|
||||||
index.ts # Public exports
|
|
||||||
connector.ts # KtxScanConnector implementation
|
|
||||||
dialect.ts # SQL dialect handling
|
|
||||||
```
|
```
|
||||||
|
|
||||||
The `package.json` should follow the pattern of existing connectors:
|
Add any connector-specific npm dependency to `packages/cli/package.json`.
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"name": "@ktx/connector-<name>",
|
|
||||||
"private": true,
|
|
||||||
"type": "module",
|
|
||||||
"main": "dist/index.js",
|
|
||||||
"types": "dist/index.d.ts",
|
|
||||||
"exports": {
|
|
||||||
".": {
|
|
||||||
"types": "./dist/index.d.ts",
|
|
||||||
"import": "./dist/index.js"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"dependencies": {
|
|
||||||
"@ktx/context": "workspace:*"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 2: Implement the connector
|
### Step 2: Implement the connector
|
||||||
|
|
||||||
|
|
@ -226,20 +197,20 @@ and statistics.
|
||||||
|
|
||||||
### Step 4: Wire it up
|
### Step 4: Wire it up
|
||||||
|
|
||||||
Register the new connector in `packages/context` so the CLI and scan
|
Register the new connector in `packages/cli/src/local-scan-connectors.ts` and
|
||||||
engine can instantiate it. Look at how existing connectors are registered for
|
`packages/cli/src/local-adapters.ts` so the CLI and scan engine can instantiate
|
||||||
the pattern.
|
it. Keep runtime loading dynamic when the connector is optional.
|
||||||
|
|
||||||
### Step 5: Test
|
### Step 5: Test
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pnpm --filter @ktx/connector-<name> run build
|
pnpm --filter @kaelio/ktx run build
|
||||||
pnpm --filter @ktx/connector-<name> run type-check
|
pnpm --filter @kaelio/ktx run type-check
|
||||||
pnpm --filter @ktx/connector-<name> run test
|
pnpm --filter @kaelio/ktx run test
|
||||||
```
|
```
|
||||||
|
|
||||||
Use `packages/connector-sqlite/` as a minimal reference and
|
Use `packages/cli/src/connectors/sqlite/` as a minimal reference and
|
||||||
`packages/connector-postgres/` as a full-featured one.
|
`packages/cli/src/connectors/postgres/` as a full-featured one.
|
||||||
|
|
||||||
## Code conventions
|
## Code conventions
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -6,9 +6,9 @@ description: Set up ktx with Claude Code, Claude Desktop, Cursor, Codex, and Ope
|
||||||
**ktx** exposes context to end-user agents through MCP tools. The CLI remains the
|
**ktx** exposes context to end-user agents through MCP tools. The CLI remains the
|
||||||
admin surface for setup, ingest, status, daemon lifecycle, and debugging.
|
admin surface for setup, ingest, status, daemon lifecycle, and debugging.
|
||||||
|
|
||||||
Run `ktx setup` and select your client agent targets, or configure manually
|
Run `ktx setup` and select your agent client targets, or configure manually
|
||||||
using the snippets below. Choose **Ask data questions with ktx MCP** for client
|
using the snippets below. Choose **Ask data questions with ktx MCP** for agent
|
||||||
agents. Choose **Ask data questions + manage ktx with CLI commands** only when
|
clients. Choose **Ask data questions + manage ktx with CLI commands** only when
|
||||||
a developer or operator agent also needs pinned `ktx` admin commands.
|
a developer or operator agent also needs pinned `ktx` admin commands.
|
||||||
|
|
||||||
## Install with setup
|
## Install with setup
|
||||||
|
|
|
||||||
|
|
@ -105,27 +105,36 @@ prior rc lineage is consumed cleanly.
|
||||||
## Release metadata
|
## Release metadata
|
||||||
|
|
||||||
semantic-release calls `scripts/update-public-release-version.mjs` during the
|
semantic-release calls `scripts/update-public-release-version.mjs` during the
|
||||||
prepare step before the exec publish command runs. That script updates the
|
prepare step before the exec publish command runs. That script rewrites the
|
||||||
following files **inside the CI runner only**:
|
following files in lockstep:
|
||||||
|
|
||||||
- `package.json` with the semantic-release version.
|
- `package.json` and `packages/cli/package.json` with the semantic-release
|
||||||
- `release-policy.json` with `publicNpmPackageVersion`, npm publish settings,
|
version.
|
||||||
and the published package smoke-test version.
|
- `python/ktx-daemon/pyproject.toml` and `python/ktx-sl/pyproject.toml` with
|
||||||
|
the same version, normalized for PEP 440 (e.g. `0.1.0-rc.2` →
|
||||||
|
`0.1.0rc2`). Branch-prefixed npm releases skip this step because they are
|
||||||
|
not published to PyPI.
|
||||||
|
- `release-policy.json` with the npm publish settings, release mode, and
|
||||||
|
published package smoke-test version. The package version itself is not
|
||||||
|
stored here; it is derived from `packages/cli/package.json.version` by
|
||||||
|
`scripts/public-npm-release-metadata.mjs`, so there is only one place that
|
||||||
|
owns the version.
|
||||||
|
|
||||||
The artifact packaging, readiness, and smoke-test scripts read
|
The `@semantic-release/git` plugin then commits these files back to `main`
|
||||||
`publicNpmPackageVersion` from `release-policy.json` within the same CI run.
|
with the release tag (see `scripts/semantic-release-config.cjs`). As a
|
||||||
Nothing reads these files at runtime — the daemon and CLI rely on the
|
result, the dev tree always reflects the most recently published version —
|
||||||
published `package.json` (for the installed `@kaelio/ktx` package) or
|
there is no sentinel pin. `ktx --version` and the bundled Python wheel both
|
||||||
`packages/cli/package.json` (for dev-tree runs from this repo, which always
|
report whatever was last released.
|
||||||
report `0.0.0-private`). Because the metadata mutation never has to survive
|
|
||||||
the run, no commit is pushed back to `main`. The git tag plus the published
|
|
||||||
npm artifact carry the version forward.
|
|
||||||
|
|
||||||
The bundled Python runtime wheel also derives its version from
|
At runtime the CLI reads its version from its own `package.json` via
|
||||||
`publicNpmPackageVersion`. Stable npm versions are reused as-is, and rc
|
`getKtxCliPackageInfo()` (`packages/cli/src/cli-runtime.ts`); the Python
|
||||||
versions are normalized to Python's version format. For example,
|
daemon reads its version from installed-package metadata via
|
||||||
`0.1.0-rc.2` becomes `0.1.0rc2` in the `kaelio-ktx` wheel filename and wheel
|
`importlib.metadata.version()` (`python/ktx-daemon/src/ktx_daemon/__init__.py`).
|
||||||
metadata.
|
|
||||||
|
The bundled Python runtime wheel also derives its version from the same
|
||||||
|
single source. Stable npm versions are reused as-is, and rc versions are
|
||||||
|
normalized to Python's version format. For example, `0.1.0-rc.2` becomes
|
||||||
|
`0.1.0rc2` in the `kaelio-ktx` wheel filename and wheel metadata.
|
||||||
|
|
||||||
## npm authentication
|
## npm authentication
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -58,7 +58,7 @@ Create a project and enable query history:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
export WAREHOUSE_DATABASE_URL=postgresql://ktx_reader:ktx_reader@127.0.0.1:55432/analytics # pragma: allowlist secret
|
export WAREHOUSE_DATABASE_URL=postgresql://ktx_reader:ktx_reader@127.0.0.1:55432/analytics # pragma: allowlist secret
|
||||||
pnpm --filter @ktx/cli run build
|
pnpm --filter @kaelio/ktx run build
|
||||||
node packages/cli/dist/bin.js --project-dir /tmp/ktx-postgres-historic setup \
|
node packages/cli/dist/bin.js --project-dir /tmp/ktx-postgres-historic setup \
|
||||||
--new \
|
--new \
|
||||||
--skip-agents \
|
--skip-agents \
|
||||||
|
|
|
||||||
|
|
@ -198,8 +198,7 @@ NODE
|
||||||
}
|
}
|
||||||
|
|
||||||
cd "$KTX_ROOT"
|
cd "$KTX_ROOT"
|
||||||
pnpm --filter @ktx/context run build
|
pnpm --filter @kaelio/ktx run build
|
||||||
pnpm --filter @ktx/cli run build
|
|
||||||
|
|
||||||
docker compose -f "$COMPOSE_FILE" up -d --wait
|
docker compose -f "$COMPOSE_FILE" up -d --wait
|
||||||
"$EXAMPLE_DIR/scripts/generate-workload.sh" base
|
"$EXAMPLE_DIR/scripts/generate-workload.sh" base
|
||||||
|
|
|
||||||
132
knip.json
132
knip.json
|
|
@ -2,121 +2,41 @@
|
||||||
"$schema": "https://unpkg.com/knip@6/schema.json",
|
"$schema": "https://unpkg.com/knip@6/schema.json",
|
||||||
"workspaces": {
|
"workspaces": {
|
||||||
".": {
|
".": {
|
||||||
"entry": ["scripts/**/*.mjs"],
|
"entry": [
|
||||||
"project": ["scripts/**/*.mjs"],
|
"scripts/**/*.mjs",
|
||||||
"ignoreDependencies": [
|
"scripts/**/*.cjs",
|
||||||
"@semantic-release/commit-analyzer",
|
".releaserc.cjs!"
|
||||||
"@semantic-release/github",
|
|
||||||
"@semantic-release/npm",
|
|
||||||
"@semantic-release/release-notes-generator",
|
|
||||||
"conventional-changelog-conventionalcommits"
|
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"packages/cli": {
|
"packages/cli": {
|
||||||
"entry": [
|
"entry": [
|
||||||
"src/index.ts",
|
"src/print-command-tree.ts!",
|
||||||
"src/bin.ts",
|
"scripts/**/*.mjs",
|
||||||
"src/**/*.test.ts",
|
"src/**/*.test-utils.ts",
|
||||||
"src/**/*.test.tsx",
|
"src/**/acceptance-fixtures.ts",
|
||||||
"scripts/**/*.mjs"
|
"src/context/scan/relationship-benchmarks.ts!",
|
||||||
],
|
"src/context/scan/relationship-benchmark-report.ts!"
|
||||||
"project": ["src/**/*.{ts,tsx}", "scripts/**/*.mjs", "vitest.config.ts"]
|
]
|
||||||
},
|
|
||||||
"packages/context": {
|
|
||||||
"entry": [
|
|
||||||
"src/index.ts",
|
|
||||||
"src/agent/index.ts",
|
|
||||||
"src/core/index.ts",
|
|
||||||
"src/connections/index.ts",
|
|
||||||
"src/daemon/index.ts",
|
|
||||||
"src/ingest/index.ts",
|
|
||||||
"src/ingest/memory-flow/index.ts",
|
|
||||||
"src/ingest/metabase-mapping.ts",
|
|
||||||
"src/scan/index.ts",
|
|
||||||
"src/search/index.ts",
|
|
||||||
"src/sql-analysis/index.ts",
|
|
||||||
"src/memory/index.ts",
|
|
||||||
"src/mcp/index.ts",
|
|
||||||
"src/project/index.ts",
|
|
||||||
"src/prompts/index.ts",
|
|
||||||
"src/skills/index.ts",
|
|
||||||
"src/sl/index.ts",
|
|
||||||
"src/sl/descriptions.ts",
|
|
||||||
"src/tools/index.ts",
|
|
||||||
"src/wiki/index.ts",
|
|
||||||
"src/**/*.test.ts",
|
|
||||||
"scripts/**/*.mjs"
|
|
||||||
],
|
|
||||||
"project": ["src/**/*.ts", "scripts/**/*.mjs", "vitest.config.ts"]
|
|
||||||
},
|
|
||||||
"packages/llm": {
|
|
||||||
"entry": ["src/index.ts", "src/**/*.test.ts"],
|
|
||||||
"project": ["src/**/*.ts", "vitest.config.ts"]
|
|
||||||
},
|
|
||||||
"packages/connector-*": {
|
|
||||||
"entry": ["src/index.ts", "src/**/*.test.ts"],
|
|
||||||
"project": ["src/**/*.ts"]
|
|
||||||
},
|
},
|
||||||
"docs-site": {
|
"docs-site": {
|
||||||
"entry": [
|
"entry": [
|
||||||
"app/**/*.{ts,tsx}",
|
"components/**/*.{ts,tsx}!",
|
||||||
"components/**/*.{ts,tsx}",
|
"source.config.ts!"
|
||||||
"lib/**/*.{ts,tsx}",
|
|
||||||
"middleware.ts",
|
|
||||||
"next.config.mjs",
|
|
||||||
"source.config.ts",
|
|
||||||
"tests/**/*.mjs"
|
|
||||||
],
|
],
|
||||||
"project": [
|
"ignoreDependencies": [
|
||||||
"app/**/*.{ts,tsx}",
|
"tailwindcss"
|
||||||
"components/**/*.{ts,tsx}",
|
]
|
||||||
"lib/**/*.{ts,tsx}",
|
|
||||||
"*.ts",
|
|
||||||
"*.mjs",
|
|
||||||
"tests/**/*.mjs"
|
|
||||||
],
|
|
||||||
"ignoreDependencies": ["tailwindcss"]
|
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"ignore": [
|
"ignoreDependencies": [
|
||||||
"**/dist/**",
|
"@semantic-release/commit-analyzer",
|
||||||
"**/coverage/**",
|
"@semantic-release/github",
|
||||||
"**/.next/**",
|
"@semantic-release/npm",
|
||||||
"**/node_modules/**",
|
"@semantic-release/release-notes-generator",
|
||||||
"**/*.gen.ts",
|
"conventional-changelog-conventionalcommits"
|
||||||
"**/*.generated.ts"
|
|
||||||
],
|
],
|
||||||
"ignoreBinaries": ["uv", "lsof"],
|
"ignoreBinaries": [
|
||||||
"ignoreIssues": {
|
"uv",
|
||||||
"packages/cli/src/clack.ts": ["exports"],
|
"lsof"
|
||||||
"packages/cli/src/commands/connection-metabase-setup.ts": ["exports", "types"],
|
]
|
||||||
"packages/cli/src/ingest.test-utils.ts": ["exports"],
|
|
||||||
"packages/cli/src/io/symbols.ts": ["exports"],
|
|
||||||
"packages/cli/src/managed-python-command.ts": ["types"],
|
|
||||||
"packages/cli/src/managed-python-daemon.ts": ["types"],
|
|
||||||
"packages/cli/src/managed-python-http.ts": ["exports", "types"],
|
|
||||||
"packages/cli/src/managed-python-runtime.ts": ["types"],
|
|
||||||
"packages/cli/src/memory-flow-tui.tsx": ["types"],
|
|
||||||
"packages/cli/src/next-steps.ts": ["exports"],
|
|
||||||
"packages/cli/src/print-command-tree.ts": ["exports"],
|
|
||||||
"packages/cli/src/setup-agents.ts": ["exports", "types"],
|
|
||||||
"packages/cli/src/setup-context.ts": ["types"],
|
|
||||||
"packages/cli/src/setup-demo-tour.ts": ["exports"],
|
|
||||||
"packages/cli/src/setup-models.ts": ["exports"],
|
|
||||||
"packages/cli/src/setup-project.ts": ["types"],
|
|
||||||
"packages/cli/src/setup-ready-menu.ts": ["types"],
|
|
||||||
"packages/cli/src/setup-sources.ts": ["types"],
|
|
||||||
"packages/context/src/ingest/adapters/historic-sql/pattern-inputs.ts": ["exports", "types"],
|
|
||||||
"packages/context/src/ingest/adapters/lookml/pull-config.ts": ["exports"],
|
|
||||||
"packages/context/src/ingest/adapters/metabase/serialize-card.ts": ["types"],
|
|
||||||
"packages/context/src/ingest/adapters/metabase/types.ts": ["exports"],
|
|
||||||
"packages/context/src/ingest/adapters/metricflow/parse.ts": ["types"],
|
|
||||||
"packages/context/src/ingest/ports.ts": ["types"],
|
|
||||||
"packages/context/src/ingest/stages/stage-3-work-units.ts": ["types"],
|
|
||||||
"packages/context/src/ingest/stages/stage-index.types.ts": ["types"],
|
|
||||||
"packages/context/src/project/config.ts": ["types"],
|
|
||||||
"packages/context/src/scan/relationship-candidates.ts": ["types"],
|
|
||||||
"packages/context/src/scan/relationship-diagnostics.ts": ["types"],
|
|
||||||
"packages/context/src/tools/context-evidence-tool-store.ts": ["types"]
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
|
||||||
10
package.json
10
package.json
|
|
@ -19,10 +19,11 @@
|
||||||
"artifacts:verify-manifest": "node scripts/package-artifacts.mjs verify-manifest",
|
"artifacts:verify-manifest": "node scripts/package-artifacts.mjs verify-manifest",
|
||||||
"build": "pnpm --filter './packages/*' run build",
|
"build": "pnpm --filter './packages/*' run build",
|
||||||
"check": "node scripts/check-boundaries.mjs && node --test scripts/*.test.mjs && pnpm --filter './packages/*' run build && pnpm --filter './packages/*' run test",
|
"check": "node scripts/check-boundaries.mjs && node --test scripts/*.test.mjs && pnpm --filter './packages/*' run build && pnpm --filter './packages/*' run test",
|
||||||
"dead-code": "pnpm run dead-code:biome && pnpm run dead-code:knip",
|
"dead-code": "pnpm run dead-code:biome && pnpm run dead-code:knip && pnpm run dead-code:knip:production",
|
||||||
"dead-code:biome": "biome ci . --formatter-enabled=false --assist-enabled=false",
|
"dead-code:biome": "biome ci . --formatter-enabled=false --assist-enabled=false",
|
||||||
"dead-code:fix": "biome check . --formatter-enabled=false --assist-enabled=false --write && knip --fix --format",
|
"dead-code:fix": "biome check . --formatter-enabled=false --assist-enabled=false --write && knip --fix --format",
|
||||||
"dead-code:knip": "knip --reporter compact",
|
"dead-code:knip": "knip --reporter compact",
|
||||||
|
"dead-code:knip:production": "knip --production --reporter compact",
|
||||||
"docs": "kill $(lsof -ti:3000) 2>/dev/null; pnpm --filter ktx-docs run dev",
|
"docs": "kill $(lsof -ti:3000) 2>/dev/null; pnpm --filter ktx-docs run dev",
|
||||||
"ktx": "node scripts/run-ktx.mjs",
|
"ktx": "node scripts/run-ktx.mjs",
|
||||||
"link:dev": "node scripts/link-dev-cli.mjs",
|
"link:dev": "node scripts/link-dev-cli.mjs",
|
||||||
|
|
@ -39,16 +40,18 @@
|
||||||
"semantic-release": "semantic-release",
|
"semantic-release": "semantic-release",
|
||||||
"semantic-release:debug": "semantic-release --dry-run --debug",
|
"semantic-release:debug": "semantic-release --dry-run --debug",
|
||||||
"semantic-release:dry-run": "semantic-release --dry-run --no-ci",
|
"semantic-release:dry-run": "semantic-release --dry-run --no-ci",
|
||||||
"smoke": "pnpm run build && pnpm --filter @ktx/cli run smoke",
|
"smoke": "pnpm run build && pnpm --filter @kaelio/ktx run smoke",
|
||||||
"test": "node --test scripts/*.test.mjs && pnpm --filter './packages/*' run test",
|
"test": "node --test scripts/*.test.mjs && pnpm --filter './packages/*' run test",
|
||||||
"test:coverage": "pnpm run test:coverage:ts && pnpm run test:coverage:py",
|
"test:coverage": "pnpm run test:coverage:ts && pnpm run test:coverage:py",
|
||||||
"test:coverage:py": "uv run pytest --cov=python/ktx-sl/semantic_layer --cov=python/ktx-daemon/src/ktx_daemon --cov-report=xml:coverage/python.xml --cov-report=term",
|
"test:coverage:py": "uv run pytest --cov=python/ktx-sl/semantic_layer --cov=python/ktx-daemon/src/ktx_daemon --cov-report=xml:coverage/python.xml --cov-report=term",
|
||||||
"test:coverage:ts": "pnpm --filter './packages/*' run build && pnpm --filter './packages/*' run test --coverage --coverage.reporter=lcov --coverage.exclude='dist/**' && node scripts/normalize-lcov-paths.mjs packages/*/coverage/lcov.info",
|
"test:coverage:ts": "pnpm --filter './packages/*' run build && pnpm --filter './packages/*' run test --coverage --coverage.reporter=lcov --coverage.exclude='dist/**' && node scripts/normalize-lcov-paths.mjs packages/*/coverage/lcov.info",
|
||||||
"test:slow": "pnpm --filter @ktx/context run test:slow && pnpm --filter @ktx/cli run test:slow",
|
"test:slow": "pnpm --filter @kaelio/ktx run test:slow",
|
||||||
"type-check": "pnpm --filter './packages/*' run type-check"
|
"type-check": "pnpm --filter './packages/*' run type-check"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"@biomejs/biome": "^2.4.15",
|
"@biomejs/biome": "^2.4.15",
|
||||||
|
"@electric-sql/pglite": "^0.4.5",
|
||||||
|
"@electric-sql/pglite-socket": "^0.1.5",
|
||||||
"@semantic-release/commit-analyzer": "^13.0.1",
|
"@semantic-release/commit-analyzer": "^13.0.1",
|
||||||
"@semantic-release/exec": "^7.1.0",
|
"@semantic-release/exec": "^7.1.0",
|
||||||
"@semantic-release/git": "^10.0.1",
|
"@semantic-release/git": "^10.0.1",
|
||||||
|
|
@ -59,6 +62,7 @@
|
||||||
"better-sqlite3": "^12.10.0",
|
"better-sqlite3": "^12.10.0",
|
||||||
"conventional-changelog-conventionalcommits": "^9.3.1",
|
"conventional-changelog-conventionalcommits": "^9.3.1",
|
||||||
"knip": "^6.12.2",
|
"knip": "^6.12.2",
|
||||||
|
"pg": "^8.20.0",
|
||||||
"semantic-release": "^25.0.3",
|
"semantic-release": "^25.0.3",
|
||||||
"typescript": "^6.0.3",
|
"typescript": "^6.0.3",
|
||||||
"yaml": "^2.9.0"
|
"yaml": "^2.9.0"
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,7 @@
|
||||||
{
|
{
|
||||||
"name": "@ktx/cli",
|
"name": "@kaelio/ktx",
|
||||||
"version": "0.4.1",
|
"version": "0.4.1",
|
||||||
"description": "CLI wrapper for ktx context packages",
|
"description": "Standalone ktx context layer for data agents",
|
||||||
"private": true,
|
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"engines": {
|
"engines": {
|
||||||
"node": ">=22.0.0"
|
"node": ">=22.0.0"
|
||||||
|
|
@ -24,41 +23,68 @@
|
||||||
"dist",
|
"dist",
|
||||||
"assets"
|
"assets"
|
||||||
],
|
],
|
||||||
|
"publishConfig": {
|
||||||
|
"access": "public",
|
||||||
|
"provenance": true
|
||||||
|
},
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"assets:demo": "node scripts/build-demo-assets.mjs",
|
"assets:demo": "node scripts/build-demo-assets.mjs",
|
||||||
"build": "tsc -p tsconfig.json && node scripts/copy-runtime-assets.mjs && node ../../scripts/prepare-cli-bin.mjs",
|
"build": "tsc -p tsconfig.json && node scripts/copy-runtime-assets.mjs && node ../../scripts/prepare-cli-bin.mjs",
|
||||||
"clean": "node -e \"fs.rmSync('dist', { recursive: true, force: true }); fs.rmSync('node_modules/.cache/tsc.tsbuildinfo', { force: true })\"",
|
"clean": "node -e \"fs.rmSync('dist', { recursive: true, force: true })\"",
|
||||||
"docs:commands": "pnpm run build && node dist/print-command-tree.js",
|
"docs:commands": "pnpm run build && node dist/print-command-tree.js",
|
||||||
"smoke": "vitest run src/standalone-smoke.test.ts src/example-smoke.test.ts --testTimeout 30000",
|
"smoke": "vitest run src/standalone-smoke.test.ts src/example-smoke.test.ts --testTimeout 30000",
|
||||||
"test": "vitest run --exclude src/standalone-smoke.test.ts --exclude src/example-smoke.test.ts --exclude src/setup-databases.test.ts --exclude src/scan.test.ts --exclude src/commands/connection-metabase-setup.test.ts --exclude src/setup-models.test.ts --exclude src/setup-sources.test.ts --exclude src/setup.test.ts --exclude src/connection.test.ts --exclude src/setup-embeddings.test.ts --exclude src/ingest.test.ts --exclude src/commands/connection-mapping.test.ts --exclude src/ingest-viz.test.ts --exclude src/demo.test.ts --exclude src/setup-project.test.ts --exclude src/sl.test.ts --exclude src/local-scan-connectors.test.ts --exclude src/commands/connection-notion.test.ts",
|
"test": "vitest run --exclude src/standalone-smoke.test.ts --exclude src/example-smoke.test.ts --exclude src/setup-databases.test.ts --exclude src/scan.test.ts --exclude src/commands/connection-metabase-setup.test.ts --exclude src/setup-models.test.ts --exclude src/setup-sources.test.ts --exclude src/setup.test.ts --exclude src/connection.test.ts --exclude src/setup-embeddings.test.ts --exclude src/ingest.test.ts --exclude src/commands/connection-mapping.test.ts --exclude src/ingest-viz.test.ts --exclude src/demo.test.ts --exclude src/setup-project.test.ts --exclude src/sl.test.ts --exclude src/local-scan-connectors.test.ts --exclude src/commands/connection-notion.test.ts --exclude src/context/scan/local-scan.test.ts --exclude src/context/mcp/local-project-ports.test.ts --exclude src/context/ingest/local-stage-ingest.test.ts --exclude src/context/sl/pglite-sl-search-prototype.test.ts --exclude src/context/core/git.service.test.ts --exclude src/context/ingest/local-adapters.test.ts --exclude src/context/ingest/local-bundle-ingest.test.ts --exclude src/context/ingest/local-metabase-ingest.test.ts --exclude src/context/sl/local-sl.test.ts --exclude src/context/search/pglite-owner-process.test.ts --exclude src/context/scan/local-enrichment-artifacts.test.ts --exclude src/context/search/pglite-spike.test.ts --exclude src/context/wiki/local-knowledge.test.ts --exclude src/context/sl/local-query.test.ts --exclude src/context/scan/relationship-review-decisions.test.ts --exclude src/context/scan/relationship-profiling.test.ts",
|
||||||
"test:slow": "vitest run src/setup-databases.test.ts src/scan.test.ts src/commands/connection-metabase-setup.test.ts src/setup-models.test.ts src/setup-sources.test.ts src/setup.test.ts src/connection.test.ts src/setup-embeddings.test.ts src/ingest.test.ts src/commands/connection-mapping.test.ts src/ingest-viz.test.ts src/demo.test.ts src/setup-project.test.ts src/sl.test.ts src/local-scan-connectors.test.ts src/commands/connection-notion.test.ts --testTimeout 30000",
|
"test:slow": "vitest run src/setup-databases.test.ts src/scan.test.ts src/commands/connection-metabase-setup.test.ts src/setup-models.test.ts src/setup-sources.test.ts src/setup.test.ts src/connection.test.ts src/setup-embeddings.test.ts src/ingest.test.ts src/commands/connection-mapping.test.ts src/ingest-viz.test.ts src/demo.test.ts src/setup-project.test.ts src/sl.test.ts src/local-scan-connectors.test.ts src/commands/connection-notion.test.ts src/context/scan/local-scan.test.ts src/context/mcp/local-project-ports.test.ts src/context/ingest/local-stage-ingest.test.ts src/context/sl/pglite-sl-search-prototype.test.ts src/context/core/git.service.test.ts src/context/ingest/local-adapters.test.ts src/context/ingest/local-bundle-ingest.test.ts src/context/ingest/local-metabase-ingest.test.ts src/context/sl/local-sl.test.ts src/context/search/pglite-owner-process.test.ts src/context/scan/local-enrichment-artifacts.test.ts src/context/search/pglite-spike.test.ts src/context/wiki/local-knowledge.test.ts src/context/sl/local-query.test.ts src/context/scan/relationship-review-decisions.test.ts src/context/scan/relationship-profiling.test.ts --testTimeout 30000",
|
||||||
"type-check": "tsc -p tsconfig.json --noEmit"
|
"type-check": "tsc -p tsconfig.json --noEmit",
|
||||||
|
"relationships:benchmarks": "pnpm --silent run build && node ../../scripts/relationship-benchmark-report.mjs",
|
||||||
|
"relationships:benchmarks:test": "KTX_RUN_RELATIONSHIP_BENCHMARKS=1 vitest run src/context/scan/relationship-benchmarks.test.ts",
|
||||||
|
"search:pglite-spike": "node ../../scripts/pglite-hybrid-search-spike.mjs",
|
||||||
|
"search:pglite-owner-prototype": "node ../../scripts/pglite-owner-process-prototype.mjs",
|
||||||
|
"search:pglite-sl-prototype": "node ../../scripts/pglite-sl-search-prototype.mjs"
|
||||||
},
|
},
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
|
"@ai-sdk/anthropic": "3.0.77",
|
||||||
|
"@ai-sdk/devtools": "0.0.17",
|
||||||
|
"@ai-sdk/google-vertex": "^4.0.128",
|
||||||
|
"@anthropic-ai/claude-agent-sdk": "0.3.142",
|
||||||
"@clack/prompts": "1.4.0",
|
"@clack/prompts": "1.4.0",
|
||||||
|
"@clickhouse/client": "^1.18.4",
|
||||||
"@commander-js/extra-typings": "14.0.0",
|
"@commander-js/extra-typings": "14.0.0",
|
||||||
"@ktx/connector-bigquery": "workspace:*",
|
"@google-cloud/bigquery": "^8.3.1",
|
||||||
"@ktx/connector-clickhouse": "workspace:*",
|
"@looker/sdk": "^26.8.0",
|
||||||
"@ktx/connector-mysql": "workspace:*",
|
"@looker/sdk-node": "^26.8.0",
|
||||||
"@ktx/connector-postgres": "workspace:*",
|
"@looker/sdk-rtl": "^21.6.5",
|
||||||
"@ktx/connector-snowflake": "workspace:*",
|
|
||||||
"@ktx/connector-sqlite": "workspace:*",
|
|
||||||
"@ktx/connector-sqlserver": "workspace:*",
|
|
||||||
"@ktx/context": "workspace:*",
|
|
||||||
"@ktx/llm": "workspace:*",
|
|
||||||
"@modelcontextprotocol/sdk": "^1.29.0",
|
"@modelcontextprotocol/sdk": "^1.29.0",
|
||||||
|
"@notionhq/client": "^5.21.0",
|
||||||
|
"ai": "^6.0.180",
|
||||||
|
"better-sqlite3": "^12.10.0",
|
||||||
"commander": "14.0.3",
|
"commander": "14.0.3",
|
||||||
"fflate": "^0.8.2",
|
"fflate": "^0.8.2",
|
||||||
|
"handlebars": "^4.7.9",
|
||||||
"ink": "^7.0.2",
|
"ink": "^7.0.2",
|
||||||
|
"lookml-parser": "7.1.0",
|
||||||
|
"minimatch": "^10.2.5",
|
||||||
|
"mssql": "^12.5.2",
|
||||||
|
"mysql2": "^3.22.3",
|
||||||
|
"openai": "^6.37.0",
|
||||||
|
"p-limit": "^7.3.0",
|
||||||
|
"pg": "^8.20.0",
|
||||||
"react": "^19.2.6",
|
"react": "^19.2.6",
|
||||||
|
"simple-git": "3.36.0",
|
||||||
|
"snowflake-sdk": "^2.4.1",
|
||||||
|
"yaml": "^2.9.0",
|
||||||
"zod": "^4.4.3"
|
"zod": "^4.4.3"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
|
"@electric-sql/pglite": "^0.4.5",
|
||||||
|
"@electric-sql/pglite-socket": "^0.1.5",
|
||||||
"@types/better-sqlite3": "^7.6.13",
|
"@types/better-sqlite3": "^7.6.13",
|
||||||
|
"@types/mssql": "^12.3.0",
|
||||||
"@types/node": "^25.7.0",
|
"@types/node": "^25.7.0",
|
||||||
|
"@types/pg": "^8.20.0",
|
||||||
"@types/react": "^19.2.14",
|
"@types/react": "^19.2.14",
|
||||||
"@vitest/coverage-v8": "^4.1.6",
|
"@vitest/coverage-v8": "^4.1.6",
|
||||||
"better-sqlite3": "^12.10.0",
|
"ajv": "8.20.0",
|
||||||
"ink-testing-library": "^4.0.0",
|
"ink-testing-library": "^4.0.0",
|
||||||
"typescript": "^6.0.3",
|
"typescript": "^6.0.3",
|
||||||
"vitest": "^4.1.6"
|
"vitest": "^4.1.6"
|
||||||
|
|
@ -66,11 +92,11 @@
|
||||||
"license": "Apache-2.0",
|
"license": "Apache-2.0",
|
||||||
"repository": {
|
"repository": {
|
||||||
"type": "git",
|
"type": "git",
|
||||||
"url": "git+https://github.com/kaelio/ktx.git",
|
"url": "https://github.com/Kaelio/ktx",
|
||||||
"directory": "packages/cli"
|
"directory": "packages/cli"
|
||||||
},
|
},
|
||||||
"bugs": {
|
"bugs": {
|
||||||
"url": "https://github.com/kaelio/ktx/issues"
|
"url": "https://github.com/Kaelio/ktx/issues"
|
||||||
},
|
},
|
||||||
"homepage": "https://github.com/kaelio/ktx#readme"
|
"homepage": "https://github.com/Kaelio/ktx#readme"
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -3,9 +3,14 @@ import { dirname, join } from 'node:path';
|
||||||
import { fileURLToPath } from 'node:url';
|
import { fileURLToPath } from 'node:url';
|
||||||
|
|
||||||
const packageRoot = fileURLToPath(new URL('..', import.meta.url));
|
const packageRoot = fileURLToPath(new URL('..', import.meta.url));
|
||||||
|
const promptsSource = join(packageRoot, 'src', 'prompts');
|
||||||
|
const promptsTarget = join(packageRoot, 'dist', 'prompts');
|
||||||
const skillsSource = join(packageRoot, 'src', 'skills');
|
const skillsSource = join(packageRoot, 'src', 'skills');
|
||||||
const skillsTarget = join(packageRoot, 'dist', 'skills');
|
const skillsTarget = join(packageRoot, 'dist', 'skills');
|
||||||
|
|
||||||
|
await rm(promptsTarget, { recursive: true, force: true });
|
||||||
await rm(skillsTarget, { recursive: true, force: true });
|
await rm(skillsTarget, { recursive: true, force: true });
|
||||||
|
await mkdir(dirname(promptsTarget), { recursive: true });
|
||||||
await mkdir(dirname(skillsTarget), { recursive: true });
|
await mkdir(dirname(skillsTarget), { recursive: true });
|
||||||
|
await cp(promptsSource, promptsTarget, { recursive: true });
|
||||||
await cp(skillsSource, skillsTarget, { recursive: true });
|
await cp(skillsSource, skillsTarget, { recursive: true });
|
||||||
|
|
|
||||||
|
|
@ -1,11 +1,11 @@
|
||||||
import { createRequire } from 'node:module';
|
import { createRequire } from 'node:module';
|
||||||
|
|
||||||
import type { ReindexSummary } from '@ktx/context/index-sync';
|
import type { ReindexSummary } from './context/index-sync/types.js';
|
||||||
import { describe, expect, it, vi } from 'vitest';
|
import { describe, expect, it, vi } from 'vitest';
|
||||||
import { renderReindexJson, renderReindexPlain, reindexHasErrors } from './admin-reindex.js';
|
import { renderReindexJson, renderReindexPlain, reindexHasErrors } from './admin-reindex.js';
|
||||||
import { runKtxCli } from './index.js';
|
import { runKtxCli } from './index.js';
|
||||||
|
|
||||||
const cliVersion = (createRequire(import.meta.url)('@ktx/cli/package.json') as { version: string })
|
const cliVersion = (createRequire(import.meta.url)('@kaelio/ktx/package.json') as { version: string })
|
||||||
.version;
|
.version;
|
||||||
|
|
||||||
function makeIo(options: { stdoutIsTTY?: boolean } = {}) {
|
function makeIo(options: { stdoutIsTTY?: boolean } = {}) {
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,8 @@
|
||||||
import { KtxIngestEmbeddingPortAdapter, type KtxEmbeddingPort } from '@ktx/context';
|
import { KtxIngestEmbeddingPortAdapter } from './context/llm/embedding-port.js';
|
||||||
import { reindexLocalIndexes, type ReindexScopeResult, type ReindexSummary } from '@ktx/context/index-sync';
|
import type { KtxEmbeddingPort } from './context/core/embedding.js';
|
||||||
import { loadKtxProject } from '@ktx/context/project';
|
import { reindexLocalIndexes } from './context/index-sync/reindex.js';
|
||||||
|
import type { ReindexScopeResult, ReindexSummary } from './context/index-sync/types.js';
|
||||||
|
import { loadKtxProject } from './context/project/project.js';
|
||||||
import { Option, type Command } from '@commander-js/extra-typings';
|
import { Option, type Command } from '@commander-js/extra-typings';
|
||||||
import { cancel, intro, log, note, outro } from '@clack/prompts';
|
import { cancel, intro, log, note, outro } from '@clack/prompts';
|
||||||
import type { KtxCliCommandContext } from './cli-program.js';
|
import type { KtxCliCommandContext } from './cli-program.js';
|
||||||
|
|
@ -55,10 +57,12 @@ function quotePlainValue(value: string): string {
|
||||||
return `"${value.replaceAll('\\', '\\\\').replaceAll('"', '\\"')}"`;
|
return `"${value.replaceAll('\\', '\\\\').replaceAll('"', '\\"')}"`;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export function reindexHasErrors(summary: ReindexSummary): boolean {
|
export function reindexHasErrors(summary: ReindexSummary): boolean {
|
||||||
return summary.scopes.some((scope) => scope.error);
|
return summary.scopes.some((scope) => scope.error);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export function renderReindexPlain(summary: ReindexSummary, io: KtxCliIo): void {
|
export function renderReindexPlain(summary: ReindexSummary, io: KtxCliIo): void {
|
||||||
const updateKey = summary.force ? 'rebuilt' : 'updated';
|
const updateKey = summary.force ? 'rebuilt' : 'updated';
|
||||||
for (const scope of summary.scopes) {
|
for (const scope of summary.scopes) {
|
||||||
|
|
@ -88,6 +92,7 @@ export function renderReindexPlain(summary: ReindexSummary, io: KtxCliIo): void
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export function renderReindexJson(summary: ReindexSummary, io: KtxCliIo): void {
|
export function renderReindexJson(summary: ReindexSummary, io: KtxCliIo): void {
|
||||||
io.stdout.write(`${JSON.stringify({ kind: 'reindex', data: summary, meta: { command: 'admin reindex' } }, null, 2)}\n`);
|
io.stdout.write(`${JSON.stringify({ kind: 'reindex', data: summary, meta: { command: 'admin reindex' } }, null, 2)}\n`);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -50,7 +50,7 @@ export function registerAdminCommands(program: Command, context: KtxCliCommandCo
|
||||||
.description('Print a JSON Schema describing ktx.yaml (for editors and LLM agents)')
|
.description('Print a JSON Schema describing ktx.yaml (for editors and LLM agents)')
|
||||||
.option('--output <file>', 'Write the schema to a file instead of stdout')
|
.option('--output <file>', 'Write the schema to a file instead of stdout')
|
||||||
.action(async (options: { output?: string }) => {
|
.action(async (options: { output?: string }) => {
|
||||||
const { generateKtxProjectConfigJsonSchema } = await import('@ktx/context/project');
|
const { generateKtxProjectConfigJsonSchema } = await import('./context/project/config.js');;
|
||||||
const json = `${JSON.stringify(generateKtxProjectConfigJsonSchema(), null, 2)}\n`;
|
const json = `${JSON.stringify(generateKtxProjectConfigJsonSchema(), null, 2)}\n`;
|
||||||
if (options.output) {
|
if (options.output) {
|
||||||
const { writeFile } = await import('node:fs/promises');
|
const { writeFile } = await import('node:fs/promises');
|
||||||
|
|
|
||||||
|
|
@ -26,7 +26,7 @@ export interface KtxCliPromptAdapter {
|
||||||
spinner(): KtxCliSpinner;
|
spinner(): KtxCliSpinner;
|
||||||
}
|
}
|
||||||
|
|
||||||
export class KtxCliPromptCancelledError extends Error {
|
class KtxCliPromptCancelledError extends Error {
|
||||||
constructor(message = 'Operation cancelled.') {
|
constructor(message = 'Operation cancelled.') {
|
||||||
super(message);
|
super(message);
|
||||||
this.name = 'KtxCliPromptCancelledError';
|
this.name = 'KtxCliPromptCancelledError';
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
import type { KtxProjectLlmConfig } from '@ktx/context/project';
|
import type { KtxProjectLlmConfig } from './context/project/config.js';
|
||||||
|
|
||||||
const CLAUDE_CODE_IGNORED_PROMPT_CACHING_FIELDS = [
|
const CLAUDE_CODE_IGNORED_PROMPT_CACHING_FIELDS = [
|
||||||
'systemTtl',
|
'systemTtl',
|
||||||
|
|
|
||||||
|
|
@ -12,9 +12,8 @@ function stubIo(): KtxCliIo {
|
||||||
|
|
||||||
function stubPackageInfo(): KtxCliPackageInfo {
|
function stubPackageInfo(): KtxCliPackageInfo {
|
||||||
return {
|
return {
|
||||||
name: '@ktx/cli',
|
name: '@kaelio/ktx',
|
||||||
version: '0.0.0-test',
|
version: '0.0.0-test',
|
||||||
contextPackageName: '@ktx/context',
|
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,26 +0,0 @@
|
||||||
import { describe, expect, it, vi } from 'vitest';
|
|
||||||
import { buildDefaultKtxProjectConfig, type KtxLocalProject, type KtxProjectConfig } from '@ktx/context/project';
|
|
||||||
import { loadKtxCliProject } from './cli-project.js';
|
|
||||||
|
|
||||||
function projectWithConfig(config: KtxProjectConfig): KtxLocalProject {
|
|
||||||
return {
|
|
||||||
projectDir: '/work/proj',
|
|
||||||
configPath: '/work/proj/ktx.yaml',
|
|
||||||
config,
|
|
||||||
coreConfig: {} as KtxLocalProject['coreConfig'],
|
|
||||||
git: {} as KtxLocalProject['git'],
|
|
||||||
fileStore: {} as KtxLocalProject['fileStore'],
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
describe('loadKtxCliProject', () => {
|
|
||||||
it('delegates to loadKtxProject and returns the project unchanged', async () => {
|
|
||||||
const project = projectWithConfig(buildDefaultKtxProjectConfig());
|
|
||||||
const loadProject = vi.fn(async () => project);
|
|
||||||
|
|
||||||
const result = await loadKtxCliProject({ projectDir: '/work/proj' }, { loadProject });
|
|
||||||
|
|
||||||
expect(result).toBe(project);
|
|
||||||
expect(loadProject).toHaveBeenCalledWith({ projectDir: '/work/proj' });
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
@ -1,20 +0,0 @@
|
||||||
import { loadKtxProject, type KtxLocalProject } from '@ktx/context/project';
|
|
||||||
|
|
||||||
export interface LoadKtxCliProjectOptions {
|
|
||||||
projectDir: string;
|
|
||||||
}
|
|
||||||
|
|
||||||
export interface LoadKtxCliProjectDeps {
|
|
||||||
loadProject?: typeof loadKtxProject;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Thin wrapper around `loadKtxProject`. Kept as a single entrypoint so the CLI can grow shared
|
|
||||||
* pre-load behavior later (telemetry, project lock, etc.). Today it does no extra work.
|
|
||||||
*/
|
|
||||||
export async function loadKtxCliProject(
|
|
||||||
options: LoadKtxCliProjectOptions,
|
|
||||||
deps: LoadKtxCliProjectDeps = {},
|
|
||||||
): Promise<KtxLocalProject> {
|
|
||||||
return (deps.loadProject ?? loadKtxProject)({ projectDir: options.projectDir });
|
|
||||||
}
|
|
||||||
|
|
@ -20,7 +20,6 @@ const requirePackageJson = createRequire(import.meta.url);
|
||||||
export interface KtxCliPackageInfo {
|
export interface KtxCliPackageInfo {
|
||||||
name: string;
|
name: string;
|
||||||
version: string;
|
version: string;
|
||||||
contextPackageName: '@ktx/context';
|
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface KtxCliIo {
|
export interface KtxCliIo {
|
||||||
|
|
@ -67,12 +66,11 @@ export function packageInfoFromJson(packageJson: unknown): KtxCliPackageInfo {
|
||||||
return {
|
return {
|
||||||
name: packageJson.name,
|
name: packageJson.name,
|
||||||
version: assertCliVersion(packageJson.version, `${packageJson.name}/package.json`),
|
version: assertCliVersion(packageJson.version, `${packageJson.name}/package.json`),
|
||||||
contextPackageName: '@ktx/context',
|
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
async function runInit(args: { projectDir: string; force: boolean }, io: KtxCliIo): Promise<number> {
|
async function runInit(args: { projectDir: string; force: boolean }, io: KtxCliIo): Promise<number> {
|
||||||
const { initKtxProject } = await import('@ktx/context/project');
|
const { initKtxProject } = await import('./context/project/project.js');;
|
||||||
const result = await initKtxProject({
|
const result = await initKtxProject({
|
||||||
projectDir: args.projectDir,
|
projectDir: args.projectDir,
|
||||||
force: args.force,
|
force: args.force,
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,7 @@ function makeContext(overrides: Partial<KtxCliCommandContext> = {}): KtxCliComma
|
||||||
stderr: { write: vi.fn() },
|
stderr: { write: vi.fn() },
|
||||||
},
|
},
|
||||||
deps: {},
|
deps: {},
|
||||||
packageInfo: { name: '@ktx/cli', version: '0.0.0-test', contextPackageName: '@ktx/context' },
|
packageInfo: { name: '@kaelio/ktx', version: '0.0.0-test' },
|
||||||
setExitCode: (code) => {
|
setExitCode: (code) => {
|
||||||
exitCode = code;
|
exitCode = code;
|
||||||
},
|
},
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,7 @@ function makeContext(overrides: Partial<KtxCliCommandContext> = {}): KtxCliComma
|
||||||
stderr: { write: vi.fn() },
|
stderr: { write: vi.fn() },
|
||||||
},
|
},
|
||||||
deps: {},
|
deps: {},
|
||||||
packageInfo: { name: '@ktx/cli', version: '0.0.0-test', contextPackageName: '@ktx/context' },
|
packageInfo: { name: '@kaelio/ktx', version: '0.0.0-test' },
|
||||||
setExitCode: (code) => {
|
setExitCode: (code) => {
|
||||||
exitCode = code;
|
exitCode = code;
|
||||||
},
|
},
|
||||||
|
|
|
||||||
|
|
@ -1,9 +1,12 @@
|
||||||
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
|
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
|
||||||
import { tmpdir } from 'node:os';
|
import { tmpdir } from 'node:os';
|
||||||
import { join } from 'node:path';
|
import { join } from 'node:path';
|
||||||
import type { LookerClient, MetabaseRuntimeClient, NotionClient } from '@ktx/context/ingest';
|
import type { LookerClient } from './context/ingest/adapters/looker/client.js';
|
||||||
import { initKtxProject, parseKtxProjectConfig, serializeKtxProjectConfig } from '@ktx/context/project';
|
import type { MetabaseRuntimeClient } from './context/ingest/adapters/metabase/client-port.js';
|
||||||
import type { KtxConnectionDriver, KtxScanConnector } from '@ktx/context/scan';
|
import type { NotionClient } from './context/ingest/adapters/notion/notion-client.js';
|
||||||
|
import { initKtxProject } from './context/project/project.js';
|
||||||
|
import { parseKtxProjectConfig, serializeKtxProjectConfig } from './context/project/config.js';
|
||||||
|
import type { KtxConnectionDriver, KtxScanConnector } from './context/scan/types.js';
|
||||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||||
import { runKtxConnection } from './connection.js';
|
import { runKtxConnection } from './connection.js';
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,19 +1,15 @@
|
||||||
import {
|
import { DEFAULT_METABASE_CLIENT_CONFIG, DefaultMetabaseConnectionClientFactory } from './context/ingest/adapters/metabase/client.js';
|
||||||
DEFAULT_METABASE_CLIENT_CONFIG,
|
import { DefaultLookerConnectionClientFactory } from './context/ingest/adapters/looker/factory.js';
|
||||||
DefaultLookerConnectionClientFactory,
|
import type { LookerClient } from './context/ingest/adapters/looker/client.js';
|
||||||
DefaultMetabaseConnectionClientFactory,
|
import type { MetabaseRuntimeClient } from './context/ingest/adapters/metabase/client-port.js';
|
||||||
type LookerClient,
|
import { type NotionBotInfo, NotionClient } from './context/ingest/adapters/notion/notion-client.js';
|
||||||
type MetabaseRuntimeClient,
|
import { createLocalLookerCredentialResolver } from './context/ingest/adapters/looker/local-looker.adapter.js';
|
||||||
type NotionBotInfo,
|
import { metabaseRuntimeConfigFromLocalConnection } from './context/ingest/adapters/metabase/local-metabase.adapter.js';
|
||||||
NotionClient,
|
import { testRepoConnection } from './context/ingest/repo-fetch.js';
|
||||||
createLocalLookerCredentialResolver,
|
import { parseNotionConnectionConfig, resolveNotionConnectionAuthToken } from './context/connections/notion-config.js';
|
||||||
metabaseRuntimeConfigFromLocalConnection,
|
import { resolveKtxConfigReference } from './context/core/config-reference.js';
|
||||||
testRepoConnection,
|
import { type KtxLocalProject, loadKtxProject } from './context/project/project.js';
|
||||||
} from '@ktx/context/ingest';
|
import type { KtxScanConnector } from './context/scan/types.js';
|
||||||
import { parseNotionConnectionConfig, resolveNotionConnectionAuthToken } from '@ktx/context/connections';
|
|
||||||
import { resolveKtxConfigReference } from '@ktx/context/core';
|
|
||||||
import { type KtxLocalProject, loadKtxProject } from '@ktx/context/project';
|
|
||||||
import type { KtxScanConnector } from '@ktx/context/scan';
|
|
||||||
import type { KtxCliIo } from './index.js';
|
import type { KtxCliIo } from './index.js';
|
||||||
import { bold, dim, green, red, SYMBOLS } from './io/symbols.js';
|
import { bold, dim, green, red, SYMBOLS } from './io/symbols.js';
|
||||||
import { createKtxCliScanConnector } from './local-scan-connectors.js';
|
import { createKtxCliScanConnector } from './local-scan-connectors.js';
|
||||||
|
|
|
||||||
|
|
@ -1,15 +1,6 @@
|
||||||
import { describe, expect, it, vi } from 'vitest';
|
import { describe, expect, it, vi } from 'vitest';
|
||||||
import {
|
import { bigQueryConnectionConfigFromConfig, isKtxBigQueryConnectionConfig, type KtxBigQueryClient, KtxBigQueryScanConnector, type KtxBigQueryClientFactory, type KtxBigQueryDataset, type KtxBigQueryQueryJob, type KtxBigQueryTableRef } from '../../connectors/bigquery/connector.js';
|
||||||
bigQueryConnectionConfigFromConfig,
|
import { createBigQueryLiveDatabaseIntrospection } from '../../connectors/bigquery/live-database-introspection.js';
|
||||||
createBigQueryLiveDatabaseIntrospection,
|
|
||||||
isKtxBigQueryConnectionConfig,
|
|
||||||
type KtxBigQueryClient,
|
|
||||||
KtxBigQueryScanConnector,
|
|
||||||
type KtxBigQueryClientFactory,
|
|
||||||
type KtxBigQueryDataset,
|
|
||||||
type KtxBigQueryQueryJob,
|
|
||||||
type KtxBigQueryTableRef,
|
|
||||||
} from './index.js';
|
|
||||||
|
|
||||||
function fakeClientFactory(): KtxBigQueryClientFactory {
|
function fakeClientFactory(): KtxBigQueryClientFactory {
|
||||||
const queryResults = vi.fn(async (): ReturnType<KtxBigQueryQueryJob['getQueryResults']> => [
|
const queryResults = vi.fn(async (): ReturnType<KtxBigQueryQueryJob['getQueryResults']> => [
|
||||||
|
|
@ -1,24 +1,6 @@
|
||||||
import { BigQuery, type TableField } from '@google-cloud/bigquery';
|
import { BigQuery, type TableField } from '@google-cloud/bigquery';
|
||||||
import { assertReadOnlySql, limitSqlForExecution } from '@ktx/context/connections';
|
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
|
||||||
import {
|
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaColumn, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableListEntry, type KtxTableRef, type KtxTableSampleInput, type KtxTableSampleResult } from '../../context/scan/types.js';
|
||||||
createKtxConnectorCapabilities,
|
|
||||||
type KtxColumnSampleInput,
|
|
||||||
type KtxColumnSampleResult,
|
|
||||||
type KtxColumnStatsInput,
|
|
||||||
type KtxColumnStatsResult,
|
|
||||||
type KtxQueryResult,
|
|
||||||
type KtxReadOnlyQueryInput,
|
|
||||||
type KtxScanConnector,
|
|
||||||
type KtxScanContext,
|
|
||||||
type KtxScanInput,
|
|
||||||
type KtxSchemaColumn,
|
|
||||||
type KtxSchemaSnapshot,
|
|
||||||
type KtxSchemaTable,
|
|
||||||
type KtxTableListEntry,
|
|
||||||
type KtxTableRef,
|
|
||||||
type KtxTableSampleInput,
|
|
||||||
type KtxTableSampleResult,
|
|
||||||
} from '@ktx/context/scan';
|
|
||||||
import { readFileSync } from 'node:fs';
|
import { readFileSync } from 'node:fs';
|
||||||
import { homedir } from 'node:os';
|
import { homedir } from 'node:os';
|
||||||
import { resolve } from 'node:path';
|
import { resolve } from 'node:path';
|
||||||
|
|
@ -57,12 +39,14 @@ export interface KtxBigQueryColumnDistinctValuesResult {
|
||||||
cardinality: number;
|
cardinality: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export interface KtxBigQueryQueryJob {
|
export interface KtxBigQueryQueryJob {
|
||||||
getQueryResults(): Promise<
|
getQueryResults(): Promise<
|
||||||
[Array<Record<string, unknown>>, unknown, { schema?: { fields?: TableField[] } }?, ...unknown[]]
|
[Array<Record<string, unknown>>, unknown, { schema?: { fields?: TableField[] } }?, ...unknown[]]
|
||||||
>;
|
>;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export interface KtxBigQueryTableRef {
|
export interface KtxBigQueryTableRef {
|
||||||
id?: string;
|
id?: string;
|
||||||
metadata?: { type?: string };
|
metadata?: { type?: string };
|
||||||
|
|
@ -81,6 +65,7 @@ export interface KtxBigQueryTableRef {
|
||||||
>;
|
>;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export interface KtxBigQueryDataset {
|
export interface KtxBigQueryDataset {
|
||||||
get(): Promise<unknown>;
|
get(): Promise<unknown>;
|
||||||
getTables(): Promise<[KtxBigQueryTableRef[], ...unknown[]]>;
|
getTables(): Promise<[KtxBigQueryTableRef[], ...unknown[]]>;
|
||||||
|
|
@ -223,6 +208,7 @@ export function isKtxBigQueryConnectionConfig(
|
||||||
return String(connection?.driver ?? '').toLowerCase() === 'bigquery';
|
return String(connection?.driver ?? '').toLowerCase() === 'bigquery';
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export function bigQueryConnectionConfigFromConfig(input: {
|
export function bigQueryConnectionConfigFromConfig(input: {
|
||||||
connectionId: string;
|
connectionId: string;
|
||||||
connection: KtxBigQueryConnectionConfig | undefined;
|
connection: KtxBigQueryConnectionConfig | undefined;
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
import type { KtxSchemaDimensionType, KtxTableRef } from '@ktx/context/scan';
|
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
|
||||||
|
|
||||||
type BigQueryTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
type BigQueryTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
||||||
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
import type { LiveDatabaseIntrospectionPort } from '@ktx/context/ingest';
|
import type { LiveDatabaseIntrospectionPort } from '../../context/ingest/adapters/live-database/types.js';
|
||||||
import type { KtxProjectConnectionConfig } from '@ktx/context/project';
|
import type { KtxProjectConnectionConfig } from '../../context/project/config.js';
|
||||||
import {
|
import {
|
||||||
KtxBigQueryScanConnector,
|
KtxBigQueryScanConnector,
|
||||||
type KtxBigQueryClientFactory,
|
type KtxBigQueryClientFactory,
|
||||||
|
|
@ -1,11 +1,6 @@
|
||||||
import { describe, expect, it, vi } from 'vitest';
|
import { describe, expect, it, vi } from 'vitest';
|
||||||
import {
|
import { clickHouseClientConfigFromConfig, isKtxClickHouseConnectionConfig, KtxClickHouseScanConnector, type KtxClickHouseClientFactory } from '../../connectors/clickhouse/connector.js';
|
||||||
clickHouseClientConfigFromConfig,
|
import { createClickHouseLiveDatabaseIntrospection } from '../../connectors/clickhouse/live-database-introspection.js';
|
||||||
createClickHouseLiveDatabaseIntrospection,
|
|
||||||
isKtxClickHouseConnectionConfig,
|
|
||||||
KtxClickHouseScanConnector,
|
|
||||||
type KtxClickHouseClientFactory,
|
|
||||||
} from './index.js';
|
|
||||||
|
|
||||||
function result<T>(payload: T) {
|
function result<T>(payload: T) {
|
||||||
return {
|
return {
|
||||||
|
|
@ -1,24 +1,6 @@
|
||||||
import { createClient } from '@clickhouse/client';
|
import { createClient } from '@clickhouse/client';
|
||||||
import { assertReadOnlySql, limitSqlForExecution } from '@ktx/context/connections';
|
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
|
||||||
import {
|
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaColumn, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableRef, type KtxTableSampleInput, type KtxTableListEntry, type KtxTableSampleResult } from '../../context/scan/types.js';
|
||||||
createKtxConnectorCapabilities,
|
|
||||||
type KtxColumnSampleInput,
|
|
||||||
type KtxColumnSampleResult,
|
|
||||||
type KtxColumnStatsInput,
|
|
||||||
type KtxColumnStatsResult,
|
|
||||||
type KtxQueryResult,
|
|
||||||
type KtxReadOnlyQueryInput,
|
|
||||||
type KtxScanConnector,
|
|
||||||
type KtxScanContext,
|
|
||||||
type KtxScanInput,
|
|
||||||
type KtxSchemaColumn,
|
|
||||||
type KtxSchemaSnapshot,
|
|
||||||
type KtxSchemaTable,
|
|
||||||
type KtxTableRef,
|
|
||||||
type KtxTableSampleInput,
|
|
||||||
type KtxTableListEntry,
|
|
||||||
type KtxTableSampleResult,
|
|
||||||
} from '@ktx/context/scan';
|
|
||||||
import { readFileSync } from 'node:fs';
|
import { readFileSync } from 'node:fs';
|
||||||
import { Agent as HttpsAgent } from 'node:https';
|
import { Agent as HttpsAgent } from 'node:https';
|
||||||
import { homedir } from 'node:os';
|
import { homedir } from 'node:os';
|
||||||
|
|
@ -198,6 +180,7 @@ export function isKtxClickHouseConnectionConfig(
|
||||||
return String(connection?.driver ?? '').toLowerCase() === 'clickhouse';
|
return String(connection?.driver ?? '').toLowerCase() === 'clickhouse';
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export function clickHouseClientConfigFromConfig(input: {
|
export function clickHouseClientConfigFromConfig(input: {
|
||||||
connectionId: string;
|
connectionId: string;
|
||||||
connection: KtxClickHouseConnectionConfig | undefined;
|
connection: KtxClickHouseConnectionConfig | undefined;
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
import type { KtxSchemaDimensionType, KtxTableRef } from '@ktx/context/scan';
|
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
|
||||||
|
|
||||||
type ClickHouseTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
type ClickHouseTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
||||||
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
import type { LiveDatabaseIntrospectionPort } from '@ktx/context/ingest';
|
import type { LiveDatabaseIntrospectionPort } from '../../context/ingest/adapters/live-database/types.js';
|
||||||
import type { KtxProjectConnectionConfig } from '@ktx/context/project';
|
import type { KtxProjectConnectionConfig } from '../../context/project/config.js';
|
||||||
import {
|
import {
|
||||||
KtxClickHouseScanConnector,
|
KtxClickHouseScanConnector,
|
||||||
type KtxClickHouseClientFactory,
|
type KtxClickHouseClientFactory,
|
||||||
|
|
@ -1,12 +1,7 @@
|
||||||
import { describe, expect, it, vi } from 'vitest';
|
import { describe, expect, it, vi } from 'vitest';
|
||||||
import type { FieldPacket, RowDataPacket } from 'mysql2/promise';
|
import type { FieldPacket, RowDataPacket } from 'mysql2/promise';
|
||||||
import {
|
import { createMysqlLiveDatabaseIntrospection } from '../../connectors/mysql/live-database-introspection.js';
|
||||||
createMysqlLiveDatabaseIntrospection,
|
import { isKtxMysqlConnectionConfig, KtxMysqlScanConnector, mysqlConnectionPoolConfigFromConfig, type KtxMysqlPoolFactory } from '../../connectors/mysql/connector.js';
|
||||||
isKtxMysqlConnectionConfig,
|
|
||||||
KtxMysqlScanConnector,
|
|
||||||
mysqlConnectionPoolConfigFromConfig,
|
|
||||||
type KtxMysqlPoolFactory,
|
|
||||||
} from './index.js';
|
|
||||||
|
|
||||||
function mysqlResult(rows: Record<string, unknown>[], fields: Array<{ name: string; type?: number }>): [RowDataPacket[], FieldPacket[]] {
|
function mysqlResult(rows: Record<string, unknown>[], fields: Array<{ name: string; type?: number }>): [RowDataPacket[], FieldPacket[]] {
|
||||||
return [rows as RowDataPacket[], fields as FieldPacket[]];
|
return [rows as RowDataPacket[], fields as FieldPacket[]];
|
||||||
|
|
@ -2,27 +2,8 @@ import mysql, { type FieldPacket, type Pool, type RowDataPacket } from 'mysql2/p
|
||||||
import { readFileSync } from 'node:fs';
|
import { readFileSync } from 'node:fs';
|
||||||
import { homedir } from 'node:os';
|
import { homedir } from 'node:os';
|
||||||
import { resolve } from 'node:path';
|
import { resolve } from 'node:path';
|
||||||
import { assertReadOnlySql, limitSqlForExecution } from '@ktx/context/connections';
|
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
|
||||||
import {
|
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaColumn, type KtxTableListEntry, type KtxSchemaForeignKey, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableRef, type KtxTableSampleInput, type KtxTableSampleResult } from '../../context/scan/types.js';
|
||||||
createKtxConnectorCapabilities,
|
|
||||||
type KtxColumnSampleInput,
|
|
||||||
type KtxColumnSampleResult,
|
|
||||||
type KtxColumnStatsInput,
|
|
||||||
type KtxColumnStatsResult,
|
|
||||||
type KtxQueryResult,
|
|
||||||
type KtxReadOnlyQueryInput,
|
|
||||||
type KtxScanConnector,
|
|
||||||
type KtxScanContext,
|
|
||||||
type KtxScanInput,
|
|
||||||
type KtxSchemaColumn,
|
|
||||||
type KtxTableListEntry,
|
|
||||||
type KtxSchemaForeignKey,
|
|
||||||
type KtxSchemaSnapshot,
|
|
||||||
type KtxSchemaTable,
|
|
||||||
type KtxTableRef,
|
|
||||||
type KtxTableSampleInput,
|
|
||||||
type KtxTableSampleResult,
|
|
||||||
} from '@ktx/context/scan';
|
|
||||||
import { KtxMysqlDialect } from './dialect.js';
|
import { KtxMysqlDialect } from './dialect.js';
|
||||||
|
|
||||||
export interface KtxMysqlConnectionConfig {
|
export interface KtxMysqlConnectionConfig {
|
||||||
|
|
@ -237,6 +218,7 @@ export function isKtxMysqlConnectionConfig(
|
||||||
return String(connection?.driver ?? '').toLowerCase() === 'mysql';
|
return String(connection?.driver ?? '').toLowerCase() === 'mysql';
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export function mysqlConnectionPoolConfigFromConfig(input: {
|
export function mysqlConnectionPoolConfigFromConfig(input: {
|
||||||
connectionId: string;
|
connectionId: string;
|
||||||
connection: KtxMysqlConnectionConfig | undefined;
|
connection: KtxMysqlConnectionConfig | undefined;
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
import type { KtxSchemaDimensionType, KtxTableRef } from '@ktx/context/scan';
|
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
|
||||||
|
|
||||||
type MysqlTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
type MysqlTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
||||||
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
import type { LiveDatabaseIntrospectionPort } from '@ktx/context/ingest';
|
import type { LiveDatabaseIntrospectionPort } from '../../context/ingest/adapters/live-database/types.js';
|
||||||
import type { KtxProjectConnectionConfig } from '@ktx/context/project';
|
import type { KtxProjectConnectionConfig } from '../../context/project/config.js';
|
||||||
import {
|
import {
|
||||||
KtxMysqlScanConnector,
|
KtxMysqlScanConnector,
|
||||||
type KtxMysqlConnectionConfig,
|
type KtxMysqlConnectionConfig,
|
||||||
|
|
@ -1,11 +1,6 @@
|
||||||
import { describe, expect, it, vi } from 'vitest';
|
import { describe, expect, it, vi } from 'vitest';
|
||||||
import {
|
import { createPostgresLiveDatabaseIntrospection } from '../../connectors/postgres/live-database-introspection.js';
|
||||||
createPostgresLiveDatabaseIntrospection,
|
import { isKtxPostgresConnectionConfig, KtxPostgresScanConnector, postgresPoolConfigFromConfig, type KtxPostgresPoolFactory } from '../../connectors/postgres/connector.js';
|
||||||
isKtxPostgresConnectionConfig,
|
|
||||||
KtxPostgresScanConnector,
|
|
||||||
postgresPoolConfigFromConfig,
|
|
||||||
type KtxPostgresPoolFactory,
|
|
||||||
} from './index.js';
|
|
||||||
|
|
||||||
interface FakeQueryResult {
|
interface FakeQueryResult {
|
||||||
rows: Record<string, unknown>[];
|
rows: Record<string, unknown>[];
|
||||||
|
|
@ -1,27 +1,8 @@
|
||||||
import { readFileSync } from 'node:fs';
|
import { readFileSync } from 'node:fs';
|
||||||
import { homedir } from 'node:os';
|
import { homedir } from 'node:os';
|
||||||
import { resolve } from 'node:path';
|
import { resolve } from 'node:path';
|
||||||
import { assertReadOnlySql, limitSqlForExecution } from '@ktx/context/connections';
|
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
|
||||||
import {
|
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaColumn, type KtxSchemaForeignKey, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableListEntry, type KtxTableRef, type KtxTableSampleInput, type KtxTableSampleResult } from '../../context/scan/types.js';
|
||||||
createKtxConnectorCapabilities,
|
|
||||||
type KtxColumnSampleInput,
|
|
||||||
type KtxColumnSampleResult,
|
|
||||||
type KtxColumnStatsInput,
|
|
||||||
type KtxColumnStatsResult,
|
|
||||||
type KtxQueryResult,
|
|
||||||
type KtxReadOnlyQueryInput,
|
|
||||||
type KtxScanConnector,
|
|
||||||
type KtxScanContext,
|
|
||||||
type KtxScanInput,
|
|
||||||
type KtxSchemaColumn,
|
|
||||||
type KtxSchemaForeignKey,
|
|
||||||
type KtxSchemaSnapshot,
|
|
||||||
type KtxSchemaTable,
|
|
||||||
type KtxTableListEntry,
|
|
||||||
type KtxTableRef,
|
|
||||||
type KtxTableSampleInput,
|
|
||||||
type KtxTableSampleResult,
|
|
||||||
} from '@ktx/context/scan';
|
|
||||||
import { Pool } from 'pg';
|
import { Pool } from 'pg';
|
||||||
import { KtxPostgresDialect } from './dialect.js';
|
import { KtxPostgresDialect } from './dialect.js';
|
||||||
|
|
||||||
|
|
@ -297,6 +278,7 @@ export function isKtxPostgresConnectionConfig(
|
||||||
return driver === 'postgres' || driver === 'postgresql';
|
return driver === 'postgres' || driver === 'postgresql';
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export function postgresPoolConfigFromConfig(input: {
|
export function postgresPoolConfigFromConfig(input: {
|
||||||
connectionId: string;
|
connectionId: string;
|
||||||
connection: KtxPostgresConnectionConfig | undefined;
|
connection: KtxPostgresConnectionConfig | undefined;
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
import type { KtxSchemaDimensionType, KtxTableRef } from '@ktx/context/scan';
|
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
|
||||||
|
|
||||||
type PostgresTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
type PostgresTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
||||||
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
import type { KtxPostgresQueryClient } from '@ktx/context/ingest';
|
import type { KtxPostgresQueryClient } from '../../context/ingest/adapters/historic-sql/types.js';
|
||||||
import { KtxPostgresScanConnector, type KtxPostgresScanConnectorOptions } from './connector.js';
|
import { KtxPostgresScanConnector, type KtxPostgresScanConnectorOptions } from './connector.js';
|
||||||
|
|
||||||
export type KtxPostgresHistoricSqlQueryClientOptions = KtxPostgresScanConnectorOptions;
|
export type KtxPostgresHistoricSqlQueryClientOptions = KtxPostgresScanConnectorOptions;
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
import type { LiveDatabaseIntrospectionPort } from '@ktx/context/ingest';
|
import type { LiveDatabaseIntrospectionPort } from '../../context/ingest/adapters/live-database/types.js';
|
||||||
import type { KtxProjectConnectionConfig } from '@ktx/context/project';
|
import type { KtxProjectConnectionConfig } from '../../context/project/config.js';
|
||||||
import {
|
import {
|
||||||
KtxPostgresScanConnector,
|
KtxPostgresScanConnector,
|
||||||
type KtxPostgresConnectionConfig,
|
type KtxPostgresConnectionConfig,
|
||||||
|
|
@ -1,12 +1,6 @@
|
||||||
import { describe, expect, it, vi } from 'vitest';
|
import { describe, expect, it, vi } from 'vitest';
|
||||||
import {
|
import { createSnowflakeLiveDatabaseIntrospection } from '../../connectors/snowflake/live-database-introspection.js';
|
||||||
createSnowflakeLiveDatabaseIntrospection,
|
import { isKtxSnowflakeConnectionConfig, KtxSnowflakeScanConnector, snowflakeConnectionConfigFromConfig, type KtxSnowflakeDriver, type KtxSnowflakeDriverFactory } from '../../connectors/snowflake/connector.js';
|
||||||
isKtxSnowflakeConnectionConfig,
|
|
||||||
KtxSnowflakeScanConnector,
|
|
||||||
snowflakeConnectionConfigFromConfig,
|
|
||||||
type KtxSnowflakeDriver,
|
|
||||||
type KtxSnowflakeDriverFactory,
|
|
||||||
} from './index.js';
|
|
||||||
|
|
||||||
function fakeDriverFactory(): KtxSnowflakeDriverFactory {
|
function fakeDriverFactory(): KtxSnowflakeDriverFactory {
|
||||||
const driver: KtxSnowflakeDriver = {
|
const driver: KtxSnowflakeDriver = {
|
||||||
|
|
@ -2,26 +2,8 @@ import { createPrivateKey } from 'node:crypto';
|
||||||
import { readFileSync } from 'node:fs';
|
import { readFileSync } from 'node:fs';
|
||||||
import { homedir } from 'node:os';
|
import { homedir } from 'node:os';
|
||||||
import { resolve } from 'node:path';
|
import { resolve } from 'node:path';
|
||||||
import { assertReadOnlySql, limitSqlForExecution } from '@ktx/context/connections';
|
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
|
||||||
import {
|
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaColumn, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableRef, type KtxTableSampleInput, type KtxTableListEntry, type KtxTableSampleResult } from '../../context/scan/types.js';
|
||||||
createKtxConnectorCapabilities,
|
|
||||||
type KtxColumnSampleInput,
|
|
||||||
type KtxColumnSampleResult,
|
|
||||||
type KtxColumnStatsInput,
|
|
||||||
type KtxColumnStatsResult,
|
|
||||||
type KtxQueryResult,
|
|
||||||
type KtxReadOnlyQueryInput,
|
|
||||||
type KtxScanConnector,
|
|
||||||
type KtxScanContext,
|
|
||||||
type KtxScanInput,
|
|
||||||
type KtxSchemaColumn,
|
|
||||||
type KtxSchemaSnapshot,
|
|
||||||
type KtxSchemaTable,
|
|
||||||
type KtxTableRef,
|
|
||||||
type KtxTableSampleInput,
|
|
||||||
type KtxTableListEntry,
|
|
||||||
type KtxTableSampleResult,
|
|
||||||
} from '@ktx/context/scan';
|
|
||||||
import * as snowflake from 'snowflake-sdk';
|
import * as snowflake from 'snowflake-sdk';
|
||||||
import { KtxSnowflakeDialect } from './dialect.js';
|
import { KtxSnowflakeDialect } from './dialect.js';
|
||||||
|
|
||||||
|
|
@ -196,6 +178,7 @@ export function isKtxSnowflakeConnectionConfig(
|
||||||
return String(connection?.driver ?? '').toLowerCase() === 'snowflake';
|
return String(connection?.driver ?? '').toLowerCase() === 'snowflake';
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export function snowflakeConnectionConfigFromConfig(input: {
|
export function snowflakeConnectionConfigFromConfig(input: {
|
||||||
connectionId: string;
|
connectionId: string;
|
||||||
connection: KtxSnowflakeConnectionConfig | undefined;
|
connection: KtxSnowflakeConnectionConfig | undefined;
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
import type { KtxSchemaDimensionType, KtxTableRef } from '@ktx/context/scan';
|
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
|
||||||
|
|
||||||
type SnowflakeTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
type SnowflakeTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
||||||
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
import type { LiveDatabaseIntrospectionPort } from '@ktx/context/ingest';
|
import type { LiveDatabaseIntrospectionPort } from '../../context/ingest/adapters/live-database/types.js';
|
||||||
import type { KtxProjectConnectionConfig } from '@ktx/context/project';
|
import type { KtxProjectConnectionConfig } from '../../context/project/config.js';
|
||||||
import {
|
import {
|
||||||
KtxSnowflakeScanConnector,
|
KtxSnowflakeScanConnector,
|
||||||
type KtxSnowflakeConnectionConfig,
|
type KtxSnowflakeConnectionConfig,
|
||||||
|
|
@ -4,12 +4,8 @@ import { mkdtemp, rm } from 'node:fs/promises';
|
||||||
import { tmpdir } from 'node:os';
|
import { tmpdir } from 'node:os';
|
||||||
import { join } from 'node:path';
|
import { join } from 'node:path';
|
||||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
||||||
import {
|
import { createSqliteLiveDatabaseIntrospection } from '../../connectors/sqlite/live-database-introspection.js';
|
||||||
createSqliteLiveDatabaseIntrospection,
|
import { isKtxSqliteConnectionConfig, KtxSqliteScanConnector, sqliteDatabasePathFromConfig } from '../../connectors/sqlite/connector.js';
|
||||||
isKtxSqliteConnectionConfig,
|
|
||||||
KtxSqliteScanConnector,
|
|
||||||
sqliteDatabasePathFromConfig,
|
|
||||||
} from './index.js';
|
|
||||||
|
|
||||||
describe('KtxSqliteScanConnector', () => {
|
describe('KtxSqliteScanConnector', () => {
|
||||||
let tempDir: string;
|
let tempDir: string;
|
||||||
|
|
@ -3,25 +3,9 @@ import { existsSync, readFileSync, statSync } from 'node:fs';
|
||||||
import { homedir } from 'node:os';
|
import { homedir } from 'node:os';
|
||||||
import { isAbsolute, resolve } from 'node:path';
|
import { isAbsolute, resolve } from 'node:path';
|
||||||
import { fileURLToPath } from 'node:url';
|
import { fileURLToPath } from 'node:url';
|
||||||
import { assertReadOnlySql, limitSqlForExecution, normalizeQueryRows } from '@ktx/context/connections';
|
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
|
||||||
import {
|
import { normalizeQueryRows } from '../../context/connections/query-executor.js';
|
||||||
createKtxConnectorCapabilities,
|
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaForeignKey, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableRef, type KtxTableSampleInput, type KtxTableSampleResult } from '../../context/scan/types.js';
|
||||||
type KtxColumnSampleInput,
|
|
||||||
type KtxColumnSampleResult,
|
|
||||||
type KtxColumnStatsInput,
|
|
||||||
type KtxColumnStatsResult,
|
|
||||||
type KtxQueryResult,
|
|
||||||
type KtxReadOnlyQueryInput,
|
|
||||||
type KtxScanConnector,
|
|
||||||
type KtxScanContext,
|
|
||||||
type KtxScanInput,
|
|
||||||
type KtxSchemaForeignKey,
|
|
||||||
type KtxSchemaSnapshot,
|
|
||||||
type KtxSchemaTable,
|
|
||||||
type KtxTableRef,
|
|
||||||
type KtxTableSampleInput,
|
|
||||||
type KtxTableSampleResult,
|
|
||||||
} from '@ktx/context/scan';
|
|
||||||
import { KtxSqliteDialect } from './dialect.js';
|
import { KtxSqliteDialect } from './dialect.js';
|
||||||
|
|
||||||
export interface KtxSqliteConnectionConfig {
|
export interface KtxSqliteConnectionConfig {
|
||||||
|
|
@ -31,6 +15,7 @@ export interface KtxSqliteConnectionConfig {
|
||||||
[key: string]: unknown;
|
[key: string]: unknown;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export interface SqliteDatabasePathInput {
|
export interface SqliteDatabasePathInput {
|
||||||
connectionId: string;
|
connectionId: string;
|
||||||
projectDir?: string;
|
projectDir?: string;
|
||||||
|
|
@ -142,6 +127,7 @@ export function isKtxSqliteConnectionConfig(
|
||||||
return driver === 'sqlite' || driver === 'sqlite3';
|
return driver === 'sqlite' || driver === 'sqlite3';
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export function sqliteDatabasePathFromConfig(input: SqliteDatabasePathInput): string {
|
export function sqliteDatabasePathFromConfig(input: SqliteDatabasePathInput): string {
|
||||||
const inputDriver = input.connection?.driver ?? 'unknown';
|
const inputDriver = input.connection?.driver ?? 'unknown';
|
||||||
if (!isKtxSqliteConnectionConfig(input.connection)) {
|
if (!isKtxSqliteConnectionConfig(input.connection)) {
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
import type { KtxSchemaDimensionType, KtxTableRef } from '@ktx/context/scan';
|
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
|
||||||
|
|
||||||
type SqliteTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
type SqliteTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
||||||
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
import type { LiveDatabaseIntrospectionPort } from '@ktx/context/ingest';
|
import type { LiveDatabaseIntrospectionPort } from '../../context/ingest/adapters/live-database/types.js';
|
||||||
import type { KtxProjectConnectionConfig } from '@ktx/context/project';
|
import type { KtxProjectConnectionConfig } from '../../context/project/config.js';
|
||||||
import { KtxSqliteScanConnector, type KtxSqliteConnectionConfig } from './connector.js';
|
import { KtxSqliteScanConnector, type KtxSqliteConnectionConfig } from './connector.js';
|
||||||
|
|
||||||
export interface CreateSqliteLiveDatabaseIntrospectionOptions {
|
export interface CreateSqliteLiveDatabaseIntrospectionOptions {
|
||||||
|
|
@ -1,12 +1,6 @@
|
||||||
import { describe, expect, it, vi } from 'vitest';
|
import { describe, expect, it, vi } from 'vitest';
|
||||||
import {
|
import { createSqlServerLiveDatabaseIntrospection } from '../../connectors/sqlserver/live-database-introspection.js';
|
||||||
createSqlServerLiveDatabaseIntrospection,
|
import { isKtxSqlServerConnectionConfig, KtxSqlServerScanConnector, sqlServerConnectionPoolConfigFromConfig, type KtxSqlServerPoolFactory, type KtxSqlServerQueryResult } from '../../connectors/sqlserver/connector.js';
|
||||||
isKtxSqlServerConnectionConfig,
|
|
||||||
KtxSqlServerScanConnector,
|
|
||||||
sqlServerConnectionPoolConfigFromConfig,
|
|
||||||
type KtxSqlServerPoolFactory,
|
|
||||||
type KtxSqlServerQueryResult,
|
|
||||||
} from './index.js';
|
|
||||||
|
|
||||||
function recordset<T extends Record<string, unknown>>(
|
function recordset<T extends Record<string, unknown>>(
|
||||||
rows: T[],
|
rows: T[],
|
||||||
|
|
@ -1,24 +1,5 @@
|
||||||
import { assertReadOnlySql } from '@ktx/context/connections';
|
import { assertReadOnlySql } from '../../context/connections/read-only-sql.js';
|
||||||
import {
|
import { createKtxConnectorCapabilities, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaColumn, type KtxSchemaForeignKey, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableListEntry, type KtxTableRef, type KtxTableSampleInput, type KtxTableSampleResult } from '../../context/scan/types.js';
|
||||||
createKtxConnectorCapabilities,
|
|
||||||
type KtxColumnSampleInput,
|
|
||||||
type KtxColumnSampleResult,
|
|
||||||
type KtxColumnStatsInput,
|
|
||||||
type KtxColumnStatsResult,
|
|
||||||
type KtxQueryResult,
|
|
||||||
type KtxReadOnlyQueryInput,
|
|
||||||
type KtxScanConnector,
|
|
||||||
type KtxScanContext,
|
|
||||||
type KtxScanInput,
|
|
||||||
type KtxSchemaColumn,
|
|
||||||
type KtxSchemaForeignKey,
|
|
||||||
type KtxSchemaSnapshot,
|
|
||||||
type KtxSchemaTable,
|
|
||||||
type KtxTableListEntry,
|
|
||||||
type KtxTableRef,
|
|
||||||
type KtxTableSampleInput,
|
|
||||||
type KtxTableSampleResult,
|
|
||||||
} from '@ktx/context/scan';
|
|
||||||
import { readFileSync } from 'node:fs';
|
import { readFileSync } from 'node:fs';
|
||||||
import { homedir } from 'node:os';
|
import { homedir } from 'node:os';
|
||||||
import { resolve } from 'node:path';
|
import { resolve } from 'node:path';
|
||||||
|
|
@ -50,6 +31,7 @@ export interface KtxSqlServerPoolConfig {
|
||||||
pool: { max: number; min: number; idleTimeoutMillis: number };
|
pool: { max: number; min: number; idleTimeoutMillis: number };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export interface KtxSqlServerQueryResult {
|
export interface KtxSqlServerQueryResult {
|
||||||
recordset?: Array<Record<string, unknown>> & { columns?: Record<string, { type?: { declaration?: string } }> };
|
recordset?: Array<Record<string, unknown>> & { columns?: Record<string, { type?: { declaration?: string } }> };
|
||||||
}
|
}
|
||||||
|
|
@ -239,6 +221,7 @@ export function isKtxSqlServerConnectionConfig(
|
||||||
return String(connection?.driver ?? '').toLowerCase() === 'sqlserver';
|
return String(connection?.driver ?? '').toLowerCase() === 'sqlserver';
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export function sqlServerConnectionPoolConfigFromConfig(input: {
|
export function sqlServerConnectionPoolConfigFromConfig(input: {
|
||||||
connectionId: string;
|
connectionId: string;
|
||||||
connection: KtxSqlServerConnectionConfig | undefined;
|
connection: KtxSqlServerConnectionConfig | undefined;
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
import type { KtxSchemaDimensionType, KtxTableRef } from '@ktx/context/scan';
|
import type { KtxSchemaDimensionType, KtxTableRef } from '../../context/scan/types.js';
|
||||||
|
|
||||||
type SqlServerTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
type SqlServerTableNameRef = Pick<KtxTableRef, 'name'> & Partial<Pick<KtxTableRef, 'catalog' | 'db'>>;
|
||||||
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
import type { LiveDatabaseIntrospectionPort } from '@ktx/context/ingest';
|
import type { LiveDatabaseIntrospectionPort } from '../../context/ingest/adapters/live-database/types.js';
|
||||||
import type { KtxProjectConnectionConfig } from '@ktx/context/project';
|
import type { KtxProjectConnectionConfig } from '../../context/project/config.js';
|
||||||
import {
|
import {
|
||||||
KtxSqlServerScanConnector,
|
KtxSqlServerScanConnector,
|
||||||
type KtxSqlServerConnectionConfig,
|
type KtxSqlServerConnectionConfig,
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
import { buildDefaultKtxProjectConfig, type KtxProjectConfig } from '@ktx/context/project';
|
import { buildDefaultKtxProjectConfig, type KtxProjectConfig } from './context/project/config.js';
|
||||||
import { describe, expect, it, vi } from 'vitest';
|
import { describe, expect, it, vi } from 'vitest';
|
||||||
import type { KtxPublicIngestProject, KtxPublicIngestTargetResult } from './public-ingest.js';
|
import type { KtxPublicIngestProject, KtxPublicIngestTargetResult } from './public-ingest.js';
|
||||||
import {
|
import {
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
import type { KtxProgressPort, KtxProgressUpdateOptions } from '@ktx/context/scan';
|
import type { KtxProgressPort, KtxProgressUpdateOptions } from './context/scan/types.js';
|
||||||
import type { KtxCliIo } from './index.js';
|
import type { KtxCliIo } from './index.js';
|
||||||
import type { KtxIngestProgressUpdate } from './ingest.js';
|
import type { KtxIngestProgressUpdate } from './ingest.js';
|
||||||
import type { KtxManagedPythonInstallPolicy } from './managed-python-command.js';
|
import type { KtxManagedPythonInstallPolicy } from './managed-python-command.js';
|
||||||
|
|
@ -444,17 +444,20 @@ export function renderContextBuildView(
|
||||||
const ESC_K_RE = new RegExp(`${ESC.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}\\[K`, 'g');
|
const ESC_K_RE = new RegExp(`${ESC.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}\\[K`, 'g');
|
||||||
const ANSI_RE = /\x1b\[[0-9;]*m/g;
|
const ANSI_RE = /\x1b\[[0-9;]*m/g;
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export function extractProgressMessage(chunk: string): string | null {
|
export function extractProgressMessage(chunk: string): string | null {
|
||||||
const cleaned = chunk.replace(/^\r/, '').replace(ESC_K_RE, '').replace(/\n$/, '').trim();
|
const cleaned = chunk.replace(/^\r/, '').replace(ESC_K_RE, '').replace(/\n$/, '').trim();
|
||||||
const match = cleaned.match(/^\[(\d+)%\]\s*(.+)$/);
|
const match = cleaned.match(/^\[(\d+)%\]\s*(.+)$/);
|
||||||
return match ? `[${match[1]}%] ${match[2]}` : null;
|
return match ? `[${match[1]}%] ${match[2]}` : null;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export function parseScanSummary(output: string): string | null {
|
export function parseScanSummary(output: string): string | null {
|
||||||
const match = output.match(/(\d+) changes? across (\d+) tables?/);
|
const match = output.match(/(\d+) changes? across (\d+) tables?/);
|
||||||
return match ? `${match[2]} tables` : null;
|
return match ? `${match[2]} tables` : null;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export function parseIngestSummary(output: string): string | null {
|
export function parseIngestSummary(output: string): string | null {
|
||||||
const savedMemory = output.match(/Saved memory: (.+)/);
|
const savedMemory = output.match(/Saved memory: (.+)/);
|
||||||
if (savedMemory) return savedMemory[1];
|
if (savedMemory) return savedMemory[1];
|
||||||
|
|
@ -560,6 +563,7 @@ function collectSourceProgress(targets: ContextBuildTargetState[]): ContextBuild
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export function viewStateFromSourceProgress(
|
export function viewStateFromSourceProgress(
|
||||||
sources: ContextBuildSourceProgressUpdate[],
|
sources: ContextBuildSourceProgressUpdate[],
|
||||||
now: number,
|
now: number,
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
import type { KtxSchemaDimensionType, KtxTableRef } from '../scan/types.js';
|
import type { KtxSchemaDimensionType, KtxTableRef } from '../scan/types.js';
|
||||||
|
|
||||||
export type SupportedDriver =
|
type SupportedDriver =
|
||||||
| 'postgres'
|
| 'postgres'
|
||||||
| 'postgresql'
|
| 'postgresql'
|
||||||
| 'mysql'
|
| 'mysql'
|
||||||
|
|
@ -8,7 +8,7 @@ import {
|
||||||
} from '../ingest/adapters/notion/types.js';
|
} from '../ingest/adapters/notion/types.js';
|
||||||
import type { KtxProjectConnectionConfig } from '../project/config.js';
|
import type { KtxProjectConnectionConfig } from '../project/config.js';
|
||||||
|
|
||||||
export const KTX_NOTION_ORG_KNOWLEDGE_WARNING =
|
const KTX_NOTION_ORG_KNOWLEDGE_WARNING =
|
||||||
'Anything accessible to this Notion integration can become organization knowledge.';
|
'Anything accessible to this Notion integration can become organization knowledge.';
|
||||||
|
|
||||||
type KtxNotionCrawlMode = 'all_accessible' | 'selected_roots';
|
type KtxNotionCrawlMode = 'all_accessible' | 'selected_roots';
|
||||||
|
|
@ -39,6 +39,7 @@ export type KtxNotionConnectionConfig = Omit<
|
||||||
max_knowledge_updates_per_run: number;
|
max_knowledge_updates_per_run: number;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export interface RedactedKtxNotionConnectionConfig {
|
export interface RedactedKtxNotionConnectionConfig {
|
||||||
driver: 'notion';
|
driver: 'notion';
|
||||||
hasAuthToken: boolean;
|
hasAuthToken: boolean;
|
||||||
|
|
@ -152,6 +153,7 @@ export function parseNotionConnectionConfig(raw: unknown): KtxNotionConnectionCo
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export function redactNotionConnectionConfig(config: KtxNotionConnectionConfig): RedactedKtxNotionConnectionConfig {
|
export function redactNotionConnectionConfig(config: KtxNotionConnectionConfig): RedactedKtxNotionConnectionConfig {
|
||||||
return {
|
return {
|
||||||
driver: 'notion',
|
driver: 'notion',
|
||||||
|
|
@ -171,6 +173,7 @@ function expandHome(path: string): string {
|
||||||
return path === '~' || path.startsWith('~/') ? resolve(homedir(), path.slice(2)) : path;
|
return path === '~' || path.startsWith('~/') ? resolve(homedir(), path.slice(2)) : path;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export async function resolveNotionAuthToken(
|
export async function resolveNotionAuthToken(
|
||||||
authTokenRef: string,
|
authTokenRef: string,
|
||||||
options: ResolveNotionTokenOptions = {},
|
options: ResolveNotionTokenOptions = {},
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
import type { KtxProjectConnectionConfig } from '../project/index.js';
|
import type { KtxProjectConnectionConfig } from '../../context/project/config.js';
|
||||||
|
|
||||||
export interface KtxSqlQueryExecutionInput {
|
export interface KtxSqlQueryExecutionInput {
|
||||||
connectionId: string;
|
connectionId: string;
|
||||||
|
|
@ -49,6 +49,7 @@ function sqlitePathFromUrl(url: string): string {
|
||||||
return url;
|
return url;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export function sqliteDatabasePathFromConnection(input: KtxSqlQueryExecutionInput): string {
|
export function sqliteDatabasePathFromConnection(input: KtxSqlQueryExecutionInput): string {
|
||||||
const driver = connectionDriver(input);
|
const driver = connectionDriver(input);
|
||||||
if (driver !== 'sqlite' && driver !== 'sqlite3') {
|
if (driver !== 'sqlite' && driver !== 'sqlite3') {
|
||||||
|
|
@ -2,6 +2,7 @@ import { readFileSync } from 'node:fs';
|
||||||
import { homedir } from 'node:os';
|
import { homedir } from 'node:os';
|
||||||
import { resolve } from 'node:path';
|
import { resolve } from 'node:path';
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export function resolveKtxHomePath(path: string): string {
|
export function resolveKtxHomePath(path: string): string {
|
||||||
if (path === '~') {
|
if (path === '~') {
|
||||||
return homedir();
|
return homedir();
|
||||||
|
|
@ -1,10 +1,10 @@
|
||||||
export interface KtxStorageConfig {
|
interface KtxStorageConfig {
|
||||||
configDir?: string;
|
configDir?: string;
|
||||||
homeDir?: string;
|
homeDir?: string;
|
||||||
worktreesDir?: string;
|
worktreesDir?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface KtxGitConfig {
|
interface KtxGitConfig {
|
||||||
userName: string;
|
userName: string;
|
||||||
userEmail: string;
|
userEmail: string;
|
||||||
bootstrapMessage?: string;
|
bootstrapMessage?: string;
|
||||||
|
|
@ -1,3 +1,4 @@
|
||||||
|
/** @internal */
|
||||||
export const REDACTED_KTX_CREDENTIAL_VALUE = '<redacted>';
|
export const REDACTED_KTX_CREDENTIAL_VALUE = '<redacted>';
|
||||||
|
|
||||||
const SENSITIVE_FIELD_NAME = /(password|secret|token|api[_-]?key|private[_-]?key|passphrase|credential|authorization|url)/i;
|
const SENSITIVE_FIELD_NAME = /(password|secret|token|api[_-]?key|private[_-]?key|passphrase|credential|authorization|url)/i;
|
||||||
|
|
@ -5,7 +5,7 @@ import { GitService } from './git.service.js';
|
||||||
|
|
||||||
export type SessionOutcome = 'success' | 'empty' | 'conflict' | 'crash';
|
export type SessionOutcome = 'success' | 'empty' | 'conflict' | 'crash';
|
||||||
|
|
||||||
export interface SentinelPayload {
|
interface SentinelPayload {
|
||||||
outcome: SessionOutcome;
|
outcome: SessionOutcome;
|
||||||
at: string;
|
at: string;
|
||||||
chatId: string;
|
chatId: string;
|
||||||
|
|
@ -4,21 +4,21 @@ import { URL } from 'node:url';
|
||||||
import { spawn } from 'node:child_process';
|
import { spawn } from 'node:child_process';
|
||||||
import type { ResolvedSemanticLayerSource, SemanticLayerQueryInput } from '../sl/types.js';
|
import type { ResolvedSemanticLayerSource, SemanticLayerQueryInput } from '../sl/types.js';
|
||||||
|
|
||||||
export interface KtxSemanticLayerComputeQueryResult {
|
interface KtxSemanticLayerComputeQueryResult {
|
||||||
sql: string;
|
sql: string;
|
||||||
dialect: string;
|
dialect: string;
|
||||||
columns: Array<Record<string, unknown>>;
|
columns: Array<Record<string, unknown>>;
|
||||||
plan: Record<string, unknown>;
|
plan: Record<string, unknown>;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface KtxSemanticLayerComputeValidationResult {
|
interface KtxSemanticLayerComputeValidationResult {
|
||||||
valid: boolean;
|
valid: boolean;
|
||||||
errors: string[];
|
errors: string[];
|
||||||
warnings: string[];
|
warnings: string[];
|
||||||
perSourceWarnings: Record<string, string[]>;
|
perSourceWarnings: Record<string, string[]>;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface KtxSemanticLayerSourceGenerationColumnInput {
|
interface KtxSemanticLayerSourceGenerationColumnInput {
|
||||||
name: string;
|
name: string;
|
||||||
type: string;
|
type: string;
|
||||||
primaryKey?: boolean;
|
primaryKey?: boolean;
|
||||||
|
|
@ -26,7 +26,7 @@ export interface KtxSemanticLayerSourceGenerationColumnInput {
|
||||||
comment?: string | null;
|
comment?: string | null;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface KtxSemanticLayerSourceGenerationTableInput {
|
interface KtxSemanticLayerSourceGenerationTableInput {
|
||||||
name: string;
|
name: string;
|
||||||
catalog?: string | null;
|
catalog?: string | null;
|
||||||
db?: string | null;
|
db?: string | null;
|
||||||
|
|
@ -34,7 +34,7 @@ export interface KtxSemanticLayerSourceGenerationTableInput {
|
||||||
columns: KtxSemanticLayerSourceGenerationColumnInput[];
|
columns: KtxSemanticLayerSourceGenerationColumnInput[];
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface KtxSemanticLayerSourceGenerationLinkInput {
|
interface KtxSemanticLayerSourceGenerationLinkInput {
|
||||||
fromTable: string;
|
fromTable: string;
|
||||||
fromColumn: string;
|
fromColumn: string;
|
||||||
toTable: string;
|
toTable: string;
|
||||||
|
|
@ -42,13 +42,13 @@ export interface KtxSemanticLayerSourceGenerationLinkInput {
|
||||||
relationshipType: string;
|
relationshipType: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface KtxSemanticLayerSourceGenerationInput {
|
interface KtxSemanticLayerSourceGenerationInput {
|
||||||
tables: KtxSemanticLayerSourceGenerationTableInput[];
|
tables: KtxSemanticLayerSourceGenerationTableInput[];
|
||||||
links: KtxSemanticLayerSourceGenerationLinkInput[];
|
links: KtxSemanticLayerSourceGenerationLinkInput[];
|
||||||
dialect?: string;
|
dialect?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface KtxSemanticLayerSourceGenerationResult {
|
interface KtxSemanticLayerSourceGenerationResult {
|
||||||
sources: Array<Record<string, unknown>>;
|
sources: Array<Record<string, unknown>>;
|
||||||
sourceCount: number;
|
sourceCount: number;
|
||||||
}
|
}
|
||||||
|
|
@ -75,14 +75,14 @@ export interface KtxSemanticLayerComputePort {
|
||||||
generateSources(input: KtxSemanticLayerSourceGenerationInput): Promise<KtxSemanticLayerSourceGenerationResult>;
|
generateSources(input: KtxSemanticLayerSourceGenerationInput): Promise<KtxSemanticLayerSourceGenerationResult>;
|
||||||
}
|
}
|
||||||
|
|
||||||
export type KtxDaemonCommand = 'semantic-query' | 'semantic-validate' | 'semantic-generate-sources';
|
type KtxDaemonCommand = 'semantic-query' | 'semantic-validate' | 'semantic-generate-sources';
|
||||||
|
|
||||||
export type KtxDaemonJsonRunner = (
|
type KtxDaemonJsonRunner = (
|
||||||
subcommand: KtxDaemonCommand,
|
subcommand: KtxDaemonCommand,
|
||||||
payload: Record<string, unknown>,
|
payload: Record<string, unknown>,
|
||||||
) => Promise<Record<string, unknown>>;
|
) => Promise<Record<string, unknown>>;
|
||||||
|
|
||||||
export type KtxDaemonHttpJsonRunner = (path: string, payload: Record<string, unknown>) => Promise<Record<string, unknown>>;
|
type KtxDaemonHttpJsonRunner = (path: string, payload: Record<string, unknown>) => Promise<Record<string, unknown>>;
|
||||||
|
|
||||||
export interface PythonSemanticLayerComputeOptions {
|
export interface PythonSemanticLayerComputeOptions {
|
||||||
command?: string;
|
command?: string;
|
||||||
|
|
@ -92,6 +92,7 @@ export interface PythonSemanticLayerComputeOptions {
|
||||||
runJson?: KtxDaemonJsonRunner;
|
runJson?: KtxDaemonJsonRunner;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export interface HttpSemanticLayerComputeOptions {
|
export interface HttpSemanticLayerComputeOptions {
|
||||||
baseUrl: string;
|
baseUrl: string;
|
||||||
requestJson?: KtxDaemonHttpJsonRunner;
|
requestJson?: KtxDaemonHttpJsonRunner;
|
||||||
|
|
@ -272,6 +273,7 @@ export function createPythonSemanticLayerComputePort(
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @internal */
|
||||||
export function createHttpSemanticLayerComputePort(
|
export function createHttpSemanticLayerComputePort(
|
||||||
options: HttpSemanticLayerComputeOptions,
|
options: HttpSemanticLayerComputeOptions,
|
||||||
): KtxSemanticLayerComputePort {
|
): KtxSemanticLayerComputePort {
|
||||||
|
|
@ -2,8 +2,8 @@ import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises';
|
||||||
import { tmpdir } from 'node:os';
|
import { tmpdir } from 'node:os';
|
||||||
import { join } from 'node:path';
|
import { join } from 'node:path';
|
||||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
||||||
import type { KtxEmbeddingPort } from '../core/index.js';
|
import type { KtxEmbeddingPort } from '../../context/core/embedding.js';
|
||||||
import { initKtxProject, loadKtxProject, type KtxLocalProject } from '../project/index.js';
|
import { initKtxProject, loadKtxProject, type KtxLocalProject } from '../../context/project/project.js';
|
||||||
import { SqliteKnowledgeIndex } from '../wiki/sqlite-knowledge-index.js';
|
import { SqliteKnowledgeIndex } from '../wiki/sqlite-knowledge-index.js';
|
||||||
import { reindexLocalIndexes } from './reindex.js';
|
import { reindexLocalIndexes } from './reindex.js';
|
||||||
|
|
||||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue