mirror of
https://github.com/Kaelio/ktx.git
synced 2026-06-28 08:49:38 +02:00
Merge origin/main into merge-scan-into-ingest-v1
This commit is contained in:
commit
9131c82724
98 changed files with 3207 additions and 1007 deletions
|
|
@ -24,7 +24,6 @@ Agents must configure and ingest context sources in this order:
|
|||
| Field | Required | Description |
|
||||
|-------|----------|-------------|
|
||||
| `driver` | Yes | Source adapter: `dbt`, `metricflow`, `lookml`, `metabase`, `looker`, or `notion` |
|
||||
| `readonly` | Strongly recommended | Marks the source as read-only for KTX |
|
||||
| `source_dir` | For local file sources | Absolute or project-relative source directory |
|
||||
| `repo_url` | For Git-hosted sources | Git repository URL |
|
||||
| `branch` | No | Git branch to read |
|
||||
|
|
@ -50,7 +49,6 @@ connections:
|
|||
my-dbt:
|
||||
driver: dbt
|
||||
source_dir: /path/to/dbt/project
|
||||
readonly: true
|
||||
```
|
||||
|
||||
For a Git-hosted project:
|
||||
|
|
@ -63,7 +61,6 @@ connections:
|
|||
branch: main
|
||||
path: analytics/dbt # For monorepos
|
||||
auth_token_ref: env:GITHUB_TOKEN
|
||||
readonly: true
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
|
@ -111,7 +108,6 @@ connections:
|
|||
branch: main
|
||||
path: dbt_metrics # Subdirectory for monorepos
|
||||
auth_token_ref: env:GITHUB_TOKEN
|
||||
readonly: true
|
||||
```
|
||||
|
||||
For a local path:
|
||||
|
|
@ -158,7 +154,6 @@ connections:
|
|||
branch: main
|
||||
path: analytics # Subdirectory for monorepos
|
||||
auth_token_ref: env:GITHUB_TOKEN
|
||||
readonly: true
|
||||
```
|
||||
|
||||
For a local path:
|
||||
|
|
@ -220,7 +215,6 @@ connections:
|
|||
syncEnabled:
|
||||
"3": true
|
||||
syncMode: ONLY # Only ingest mapped databases
|
||||
readonly: true
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
|
@ -277,7 +271,6 @@ connections:
|
|||
mappings:
|
||||
connectionMappings:
|
||||
postgres_connection: postgres-main # Looker conn → KTX conn
|
||||
readonly: true
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
|
@ -330,7 +323,6 @@ connections:
|
|||
crawl_mode: selected_roots
|
||||
root_page_ids:
|
||||
- "abc123def456..."
|
||||
readonly: true
|
||||
```
|
||||
|
||||
For crawling all accessible pages:
|
||||
|
|
@ -341,7 +333,6 @@ connections:
|
|||
driver: notion
|
||||
auth_token_ref: env:NOTION_TOKEN
|
||||
crawl_mode: all_accessible
|
||||
readonly: true
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
|
|
|||
|
|
@ -25,7 +25,6 @@ Agents should prefer environment or file references over literal secrets.
|
|||
| `url` | One of the connection methods | URL-style connectors | Database URL, `env:NAME`, or `file:/path/to/secret` |
|
||||
| `host`, `port`, `database`, `username`, `password` | One of the connection methods | PostgreSQL, MySQL, ClickHouse, SQL Server | Field-by-field connection values |
|
||||
| `schema` or `schemas` | No | schema-aware warehouses | Single schema or list of schemas to scan |
|
||||
| `readonly` | Strongly recommended | all primary sources | Marks the connection as read-only in KTX config |
|
||||
| `context.queryHistory` | No | PostgreSQL, Snowflake, BigQuery | Enables query-history ingestion when the warehouse supports it |
|
||||
| `path` | Yes for path-style SQLite | SQLite | Local SQLite database path or `env:NAME` reference |
|
||||
|
||||
|
|
@ -39,9 +38,8 @@ The most full-featured connector. Supports schema introspection, foreign key det
|
|||
connections:
|
||||
my-postgres:
|
||||
driver: postgres
|
||||
url: postgresql://user:password@host:5432/database
|
||||
url: env:DATABASE_URL
|
||||
schema: public
|
||||
readonly: true
|
||||
```
|
||||
|
||||
Or with individual fields:
|
||||
|
|
@ -59,7 +57,6 @@ connections:
|
|||
- public
|
||||
- analytics
|
||||
ssl: true
|
||||
readonly: true
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
|
@ -128,7 +125,6 @@ connections:
|
|||
username: KTX_SERVICE
|
||||
password: env:SNOWFLAKE_PASSWORD
|
||||
role: ANALYST
|
||||
readonly: true
|
||||
```
|
||||
|
||||
For multiple schemas:
|
||||
|
|
@ -201,7 +197,6 @@ connections:
|
|||
credentials_json: file:~/.config/gcloud/bq-service-account.json
|
||||
dataset_id: analytics
|
||||
location: US
|
||||
readonly: true
|
||||
```
|
||||
|
||||
For multiple datasets:
|
||||
|
|
@ -274,7 +269,6 @@ connections:
|
|||
my-clickhouse:
|
||||
driver: clickhouse
|
||||
url: http://localhost:8123/analytics
|
||||
readonly: true
|
||||
```
|
||||
|
||||
Or with individual fields:
|
||||
|
|
@ -289,7 +283,6 @@ connections:
|
|||
username: default
|
||||
password: env:CH_PASSWORD
|
||||
ssl: false
|
||||
readonly: true
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
|
@ -332,8 +325,7 @@ Standard MySQL/MariaDB connector with full foreign key support and schema intros
|
|||
connections:
|
||||
my-mysql:
|
||||
driver: mysql
|
||||
url: mysql://user:password@host:3306/database
|
||||
readonly: true
|
||||
url: env:MYSQL_DATABASE_URL
|
||||
```
|
||||
|
||||
Or with individual fields:
|
||||
|
|
@ -348,7 +340,6 @@ connections:
|
|||
username: ktx_reader
|
||||
password: env:MYSQL_PASSWORD
|
||||
ssl: true
|
||||
readonly: true
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
|
@ -391,8 +382,7 @@ Connects to Microsoft SQL Server and Azure SQL. Supports multi-schema scanning w
|
|||
connections:
|
||||
my-sqlserver:
|
||||
driver: sqlserver
|
||||
url: mssql://user:password@host:1433/database?trustServerCertificate=true
|
||||
readonly: true
|
||||
url: env:SQLSERVER_DATABASE_URL
|
||||
```
|
||||
|
||||
Or with individual fields:
|
||||
|
|
@ -408,7 +398,6 @@ connections:
|
|||
password: env:MSSQL_PASSWORD
|
||||
schema: dbo
|
||||
trustServerCertificate: true
|
||||
readonly: true
|
||||
```
|
||||
|
||||
For multiple schemas:
|
||||
|
|
@ -460,7 +449,6 @@ connections:
|
|||
my-sqlite:
|
||||
driver: sqlite
|
||||
path: ./data/warehouse.sqlite
|
||||
readonly: true
|
||||
```
|
||||
|
||||
Path supports multiple formats:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue