mirror of
https://github.com/Kaelio/ktx.git
synced 2026-06-25 08:48:08 +02:00
feat: merge ingest and scan
* docs: add CLI component reuse guidance * docs: add unified ingest ux design * Refine unified ingest UX design after adversarial review iteration 1 * Refine unified ingest UX design after adversarial review iteration 2 * Refine unified ingest UX design after adversarial review iteration 3 * feat(cli): route public connection ingest command * feat(cli): hide standalone scan from public help * feat(cli): plan public ingest depth and query history * feat(cli): execute public database ingest facets * feat(ingest): read connection query history config * fix(cli): use public ingest wording * fix(config): stop generating ingest adapter allow lists * docs: document public ingest command * test: align ingest surface expectations * docs: add unified ingest public CLI surface plan * feat(cli): preflight deep public ingest readiness * feat(setup): store query history in connection context * feat(setup): store database context depth * feat(setup): verify context readiness by database depth * fix(setup): keep context build foreground only * fix(config): reject reserved ingest connection ids * test: close unified ingest v1 expectations * docs: add unified ingest v1 closure plan * fix(ingest): bypass adapter allow-list for public source ingest * fix(ingest): honor query history window intent * fix(ingest): hide scan internals from public database ingest * feat(ingest): use foreground view for interactive public ingest * fix(setup): use schema context and query history wording * test(cli): verify unified ingest public output * docs: add unified ingest v1 public output closure plan * fix(setup): forward query history flags * fix(setup): prompt for postgres query history * fix(status): report query history readiness * fix(ingest): remove legacy public guidance * fix(ingest): polish foreground retry copy * docs(examples): use unified query history wording * chore(ingest): finish public query history cleanup * docs: add unified ingest v1 query history status cleanup plan * test(docs): cover unified ingest public docs * docs: align ingest CLI reference with unified UX * docs: update context build guides for unified ingest * docs: update setup and primary source ingest wording * docs: stop advertising adapter-backed example ingest * docs: close unified ingest public docs gaps * docs: add unified ingest v1 docs site closure plan * fix: render unified ingest foreground warnings * fix: explain query history schema order * fix: add public ingest retry guidance * fix: align setup next steps with unified ingest * fix: remove scan wording from demo progress * test: verify unified ingest ux closure * docs: add unified ingest v1 foreground and retry closure plan * fix(cli): preserve query-history pull config in public ingest * fix(cli): omit hidden commands from docs command tree * test(cli): close unified ingest final public surface checks * docs: add unified ingest v1 final public surface closure plan * fix(cli): use public source labels in ingest reports * fix(cli): suppress low-level public ingest output * test(cli): verify unified ingest public plain output * docs: add unified ingest v1 public plain output closure plan * fix(cli): add public ingest copy sanitizers * fix(cli): sanitize public ingest progress copy * fix(cli): rename setup schema scope prompt * docs(plan): add progress copy closure; test: align setup back-nav fixture Adds the iter9 plan and updates the setup back-navigation test fixture to pass disableQueryHistory plus listSchemas/listTables stubs that the unified ingest setup step now requires. * docs(plan): add final ux labels plan with narrowed label scans * fix(cli): aggregate unsupported query-history warnings * fix(cli): align setup database labels * test(cli): fix setup database test type-check * fix(cli): remove primary-source wording from setup output * test(cli): verify unified ingest setup closure * docs(plan): add unified ingest v1 verification copy closure plan * fix(cli): remove top-level scan command * fix(cli): remove legacy ingest and wiki commands * Merge scan into ingest flow * feat(cli): split ingest progress into per-phase rows, rename work units to tasks Each database target in the unified ingest dashboard now renders one row per real subprocess (Schema, then Query history when enabled) instead of a single combined bar. Each phase has its own monotonic 0-100% bar so the progress never snaps back to zero when historic-sql starts after scan completes. Completed phases keep their final bar, summary, and elapsed time visible as an inline audit trail; queued and skipped phases are shown explicitly. Also rename user-facing "work units" / "Failed work units" to "tasks" / "Failed tasks" in ingest output and parseIngestSummary. The parser still accepts the legacy "Work units:" wording in captured output for backward compat. Internal memory-flow event names and type fields are left alone. * Fix test harness failures * Fix CI smoke checks --------- Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
This commit is contained in:
Andrey Avtomonov
GitHub
parent
1a472cf3ed
commit
b00c1a11a9
118 changed files with 16890 additions and 2992 deletions
|
|
@ -125,8 +125,6 @@ All supported agent clients call the same KTX CLI commands:
|
|||
|---------|-------------|
|
||||
| `ktx status --json` | Return project setup and context readiness |
|
||||
| `ktx wiki search <query> --json` | Search wiki pages |
|
||||
| `ktx wiki read <key> --json` | Read a wiki page |
|
||||
| `ktx wiki write <key>` | Write or update a wiki page |
|
||||
| `ktx sl list --json` | List semantic-layer sources |
|
||||
| `ktx sl search <query> --json` | Search semantic-layer sources |
|
||||
| `ktx sl validate <source> --connection-id <id>` | Validate semantic source definitions |
|
||||
|
|
|
|||
|
|
@ -9,12 +9,13 @@ All context sources are configured in `ktx.yaml` under `connections` with their
|
|||
|
||||
## Ingestion workflow
|
||||
|
||||
Agents should configure and ingest context sources in this order:
|
||||
Agents must configure and ingest context sources in this order:
|
||||
|
||||
1. Add the context source connection in `ktx.yaml` or with `ktx setup`.
|
||||
2. Store tokens as `env:NAME` or `file:/path/to/secret`.
|
||||
3. Run `ktx ingest run --connection-id <connectionId> --adapter <adapter>` for one source or `ktx ingest run --connection-id <id> --adapter <adapter>`.
|
||||
4. Check progress with `ktx ingest status --json`.
|
||||
3. Run `ktx ingest <connectionId>` for one source or `ktx ingest --all` for
|
||||
every configured source.
|
||||
4. Review the foreground ingest output.
|
||||
5. Review generated `semantic-layer/` YAML and `wiki/` Markdown files in git.
|
||||
6. Validate changed semantic sources with `ktx sl validate`.
|
||||
|
||||
|
|
|
|||
|
|
@ -3,13 +3,17 @@ title: Primary Sources
|
|||
description: Connect KTX to PostgreSQL, Snowflake, BigQuery, ClickHouse, MySQL, SQL Server, or SQLite.
|
||||
---
|
||||
|
||||
KTX connects to your data warehouse or database to scan schemas, discover relationships, and execute semantic layer queries. Each connection is defined in `ktx.yaml` under the `connections` key.
|
||||
KTX connects to your data warehouse or database to build schema context,
|
||||
discover relationships, and execute semantic layer queries. Each connection is
|
||||
defined in `ktx.yaml` under the `connections` key.
|
||||
|
||||
All connectors share these conventions:
|
||||
|
||||
- Sensitive values support `env:VAR_NAME` (read from environment) and `file:/path/to/secret` (read from file) references
|
||||
- Connections are read-only — KTX never writes to your database
|
||||
- Schema scanning discovers tables, columns, types, and constraints automatically
|
||||
- Sensitive values support `env:VAR_NAME` (read from environment) and
|
||||
`file:/path/to/secret` (read from file) references
|
||||
- Connections are read-only; KTX never writes to your database
|
||||
- Database ingest discovers tables, columns, types, and constraints
|
||||
automatically
|
||||
|
||||
## Connection field reference
|
||||
|
||||
|
|
@ -21,7 +25,7 @@ Agents should prefer environment or file references over literal secrets.
|
|||
| `url` | One of the connection methods | URL-style connectors | Database URL, `env:NAME`, or `file:/path/to/secret` |
|
||||
| `host`, `port`, `database`, `username`, `password` | One of the connection methods | PostgreSQL, MySQL, ClickHouse, SQL Server | Field-by-field connection values |
|
||||
| `schema` or `schemas` | No | schema-aware warehouses | Single schema or list of schemas to scan |
|
||||
| `historicSql` | No | supported warehouses | Enables query-history ingestion when the warehouse supports it |
|
||||
| `context.queryHistory` | No | PostgreSQL, Snowflake, BigQuery | Enables query-history ingestion when the warehouse supports it |
|
||||
| `path` | Yes for path-style SQLite | SQLite | Local SQLite database path or `env:NAME` reference |
|
||||
| `max_bytes_billed` | No | BigQuery | Maximum bytes billed per query job |
|
||||
| `job_timeout_ms` | No | BigQuery | BigQuery query job timeout in milliseconds |
|
||||
|
|
@ -29,7 +33,7 @@ Agents should prefer environment or file references over literal secrets.
|
|||
|
||||
## PostgreSQL
|
||||
|
||||
The most full-featured connector. Supports schema introspection, foreign key detection, column statistics, and historic SQL via `pg_stat_statements`.
|
||||
The most full-featured connector. Supports schema introspection, foreign key detection, column statistics, and query history via `pg_stat_statements`.
|
||||
|
||||
### Connection config
|
||||
|
||||
|
|
@ -75,12 +79,13 @@ connections:
|
|||
| Foreign keys | Yes | Full constraint detection |
|
||||
| Row count estimates | Yes | Via `pg_class.reltuples` |
|
||||
| Column statistics | Yes | Requires `pg_read_all_stats` role |
|
||||
| Historic SQL | Yes | Via `pg_stat_statements` extension |
|
||||
| Query history | Yes | Via `pg_stat_statements` extension |
|
||||
| Table sampling | Yes | `TABLESAMPLE SYSTEM` |
|
||||
|
||||
### Historic SQL
|
||||
### Query history
|
||||
|
||||
PostgreSQL Historic SQL mines real query patterns from `pg_stat_statements`. This is the most mature local Historic SQL path and helps KTX understand how your team actually queries the data.
|
||||
PostgreSQL query history mines real query patterns from `pg_stat_statements`.
|
||||
This helps KTX understand how your team actually queries the data.
|
||||
|
||||
**Requirements:**
|
||||
- `pg_stat_statements` extension enabled
|
||||
|
|
@ -89,12 +94,12 @@ PostgreSQL Historic SQL mines real query patterns from `pg_stat_statements`. Thi
|
|||
**Config options:**
|
||||
|
||||
```yaml
|
||||
historicSql:
|
||||
enabled: true
|
||||
dialect: postgres
|
||||
minExecutions: 5
|
||||
filters:
|
||||
dropTrivialProbes: true
|
||||
context:
|
||||
queryHistory:
|
||||
enabled: true
|
||||
minExecutions: 5
|
||||
filters:
|
||||
dropTrivialProbes: true
|
||||
```
|
||||
|
||||
### Dialect notes
|
||||
|
|
@ -108,7 +113,7 @@ historicSql:
|
|||
|
||||
## Snowflake
|
||||
|
||||
Connects via the Snowflake SDK. Supports multi-schema scanning, RSA key authentication, and Historic SQL configuration for Snowflake query history.
|
||||
Connects via the Snowflake SDK. Supports multi-schema scanning, RSA key authentication, and query-history configuration for Snowflake query history.
|
||||
|
||||
### Connection config
|
||||
|
||||
|
|
@ -150,27 +155,27 @@ For multiple schemas:
|
|||
| Foreign keys | No | Not available in Snowflake |
|
||||
| Row count estimates | Yes | From `INFORMATION_SCHEMA.TABLES.ROW_COUNT` |
|
||||
| Column statistics | No | — |
|
||||
| Historic SQL | Yes | Via `SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY` when enabled |
|
||||
| Query history | Yes | Via `SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY` when enabled |
|
||||
| Table sampling | Yes | — |
|
||||
|
||||
### Historic SQL
|
||||
### Query history
|
||||
|
||||
Snowflake Historic SQL reads aggregated query-history templates from
|
||||
Snowflake query history reads aggregated query-history templates from
|
||||
`SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY` and feeds the same unified staged
|
||||
artifact shape as Postgres and BigQuery.
|
||||
|
||||
```yaml
|
||||
historicSql:
|
||||
enabled: true
|
||||
dialect: snowflake
|
||||
windowDays: 90
|
||||
minExecutions: 5
|
||||
filters:
|
||||
dropTrivialProbes: true
|
||||
serviceAccounts:
|
||||
patterns: ['^svc_']
|
||||
mode: exclude
|
||||
redactionPatterns: []
|
||||
context:
|
||||
queryHistory:
|
||||
enabled: true
|
||||
windowDays: 90
|
||||
minExecutions: 5
|
||||
filters:
|
||||
dropTrivialProbes: true
|
||||
serviceAccounts:
|
||||
patterns: ['^svc_']
|
||||
mode: exclude
|
||||
redactionPatterns: []
|
||||
```
|
||||
|
||||
### Dialect notes
|
||||
|
|
@ -184,7 +189,7 @@ historicSql:
|
|||
|
||||
## BigQuery
|
||||
|
||||
Authenticates via GCP service account credentials. Supports multi-dataset scanning and Historic SQL configuration for `INFORMATION_SCHEMA.JOBS_BY_PROJECT`.
|
||||
Authenticates via GCP service account credentials. Supports multi-dataset scanning and query-history configuration for `INFORMATION_SCHEMA.JOBS_BY_PROJECT`.
|
||||
|
||||
### Connection config
|
||||
|
||||
|
|
@ -227,27 +232,27 @@ mapping metadata. The BigQuery connector still authenticates with the
|
|||
| Foreign keys | No | Not available in BigQuery |
|
||||
| Row count estimates | Yes | From table metadata |
|
||||
| Column statistics | No | — |
|
||||
| Historic SQL | Yes | Via region-scoped `INFORMATION_SCHEMA.JOBS_BY_PROJECT` when enabled |
|
||||
| Query history | Yes | Via region-scoped `INFORMATION_SCHEMA.JOBS_BY_PROJECT` when enabled |
|
||||
| Table sampling | Yes | — |
|
||||
|
||||
### Historic SQL
|
||||
### Query history
|
||||
|
||||
BigQuery Historic SQL reads aggregated query-history templates from
|
||||
BigQuery query history reads aggregated query-history templates from
|
||||
region-scoped `INFORMATION_SCHEMA.JOBS_BY_PROJECT` and feeds the same unified
|
||||
staged artifact shape as Postgres and Snowflake.
|
||||
|
||||
```yaml
|
||||
historicSql:
|
||||
enabled: true
|
||||
dialect: bigquery
|
||||
windowDays: 90
|
||||
minExecutions: 5
|
||||
filters:
|
||||
dropTrivialProbes: true
|
||||
serviceAccounts:
|
||||
patterns: ['@bot\\.']
|
||||
mode: exclude
|
||||
redactionPatterns: []
|
||||
context:
|
||||
queryHistory:
|
||||
enabled: true
|
||||
windowDays: 90
|
||||
minExecutions: 5
|
||||
filters:
|
||||
dropTrivialProbes: true
|
||||
serviceAccounts:
|
||||
patterns: ['@bot\\.']
|
||||
mode: exclude
|
||||
redactionPatterns: []
|
||||
```
|
||||
|
||||
### Dialect notes
|
||||
|
|
@ -303,7 +308,7 @@ connections:
|
|||
| Foreign keys | No | Not a ClickHouse concept |
|
||||
| Row count estimates | Yes | Via `system.parts` aggregation |
|
||||
| Column statistics | No | — |
|
||||
| Historic SQL | No | — |
|
||||
| Query history | No | — |
|
||||
| Table sampling | Yes | — |
|
||||
|
||||
### Dialect notes
|
||||
|
|
@ -360,7 +365,7 @@ connections:
|
|||
| Foreign keys | Yes | Via `REFERENTIAL_CONSTRAINTS` |
|
||||
| Row count estimates | Yes | From `TABLE_ROWS` (InnoDB estimate) |
|
||||
| Column statistics | No | — |
|
||||
| Historic SQL | No | — |
|
||||
| Query history | No | — |
|
||||
| Table sampling | Yes | Uses `RAND()` filter |
|
||||
|
||||
### Dialect notes
|
||||
|
|
@ -426,7 +431,7 @@ For multiple schemas:
|
|||
| Foreign keys | Yes | Via `REFERENTIAL_CONSTRAINTS` |
|
||||
| Row count estimates | Yes | Via `sys.dm_db_partition_stats` |
|
||||
| Column statistics | No | — |
|
||||
| Historic SQL | No | — |
|
||||
| Query history | No | — |
|
||||
| Table sampling | Yes | — |
|
||||
| Nested analysis | No | — |
|
||||
|
||||
|
|
@ -484,7 +489,7 @@ No authentication required — SQLite is file-based. The file must be readable b
|
|||
| Foreign keys | Yes | Via `PRAGMA foreign_key_list()` (requires `PRAGMA foreign_keys = ON`) |
|
||||
| Row count estimates | Yes | Exact count via `SELECT COUNT(*)` |
|
||||
| Column statistics | No | — |
|
||||
| Historic SQL | No | — |
|
||||
| Query history | No | — |
|
||||
| Table sampling | Yes | — |
|
||||
| Nested analysis | No | — |
|
||||
|
||||
|
|
@ -502,7 +507,7 @@ No authentication required — SQLite is file-based. The file must be readable b
|
|||
| Error or symptom | Likely cause | Recovery |
|
||||
|------------------|--------------|----------|
|
||||
| Connection URL appears in git diff | A literal credential URL was written to `ktx.yaml` | Replace it with `env:NAME` or `file:/path/to/secret` and rotate exposed credentials |
|
||||
| Scan returns no tables | Schema/database/project filter is wrong or the user lacks metadata permissions | Verify the schema list and grant metadata read permissions |
|
||||
| Historic SQL is empty | Query history extension or warehouse history view is unavailable | Enable the warehouse-specific history feature, then rerun scan or setup |
|
||||
| Column statistics are missing | Connector cannot access stats tables or the warehouse does not expose them | Grant stats permissions where supported; otherwise rely on structural scan output |
|
||||
| Database ingest returns no tables | Schema, database, or project filter is wrong, or the user lacks metadata permissions | Verify the schema list and grant metadata read permissions |
|
||||
| Query history is empty | Query history extension or warehouse history view is unavailable | Enable the warehouse-specific history feature, then rerun `ktx ingest <connectionId> --query-history` or `ktx setup` |
|
||||
| Column statistics are missing | Connector cannot access stats tables or the warehouse does not expose them | Grant stats permissions where supported; otherwise rely on fast schema context |
|
||||
| Semantic query execution fails | Connection is missing, unreachable, or query execution is disabled | Run `ktx connection test <id>` and check the `ktx sl query` flags |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue