mirror of
https://github.com/Kaelio/ktx.git
synced 2026-07-10 11:12:10 +02:00
feat(sigma): add Sigma Computing context-source adapter (#316)
* feat(sigma): add Sigma Computing context-source adapter Closes #168 Adds a full ingest adapter for Sigma Computing so `ktx ingest` can pull data model specs and workbook summaries into the ktx context layer. The implementation follows the same fetch → chunk → project → LLM pattern used by the Looker, Metabase, and MetricFlow adapters. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(sigma): address PR review comments - Remove manifest from rawFiles; moves to peerFileIndex so fetchedAt changes don't mark all work units dirty every run - Fix workbookFilter.updatedSince eviction bug: fetch full universe first, apply filter client-side, evict only on archived/deleted - Remove measure projection entirely; project() writes measures: [] and the sigma_ingest skill surfaces Lookup/aggregation formulas as wiki prose - Remove joins projection (v1 limitation); project() writes joins: [] and Lookup relationships are described in wiki prose instead - Remove write-back dead code: createDataModel, updateDataModel, SigmaDataModelPushResult, mutate/post/put - Fix emitBatches notes pluralization bug ('2 data modelss' → '2 data models') - Add tokenInflight dedup on ensureToken to coalesce concurrent auth requests - Retry spec fetch when existing staged spec is null (transient failure cache) - Drop unused WorkbookFilter import from client-port.ts - Note in docs that joins are not projected from Sigma data models in this release Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * updates * fix(sigma): restore sigma in local adapter test + small cleanups The gdrive↔sigma merge dropped 'sigma' from the expected adapter source list in local-adapters.test.ts while keeping gdrive, so the slow TS suite failed even though the source registers both. Add 'sigma' back at its registration position (after metabase, before gdrive). Also: - Move the orphaned SigmaPullConfig docstring onto the schema it documents and drop the stale BullMQ reference (standalone ktx has no BullMQ; the config lives in the ingest job's bundleRef.config). - Drop an O(n^2) find() round-trip in fetch() when building the active data-model list; filter once and reuse for the eviction id set. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Andrey Avtomonov <andreybavt@gmail.com> Co-authored-by: Luca Martial <48870843+luca-martial@users.noreply.github.com>
This commit is contained in:
parent
139ac08320
commit
acd20ac248
41 changed files with 3610 additions and 6 deletions
|
|
@ -119,6 +119,7 @@ context-source drivers share the map.
|
|||
| `dbt` | Context source | `driver`, one of `source_dir` or `repo_url` | `branch`, `path`, `profiles_path`, `target`, `project_name` |
|
||||
| `metricflow` | Context source | `driver`, `metricflow.repoUrl` | `metricflow.branch`, `metricflow.path`, `metricflow.auth_token_ref` |
|
||||
| `notion` | Context source | `driver`, `auth_token_ref` | `crawl_mode`, `root_*_ids`, `max_*_per_run` |
|
||||
| `sigma` | Context source | `driver`, `client_id`, `client_secret_ref` | `api_url` |
|
||||
|
||||
### Warehouse drivers
|
||||
|
||||
|
|
@ -345,6 +346,31 @@ connections:
|
|||
| `max_knowledge_creates_per_run` | Max new wiki pages created per run (0-25). |
|
||||
| `max_knowledge_updates_per_run` | Max existing wiki pages updated per run (0-100). |
|
||||
|
||||
### Sigma
|
||||
|
||||
```yaml
|
||||
connections:
|
||||
sigma-main:
|
||||
driver: sigma
|
||||
api_url: https://api.sigmacomputing.com
|
||||
client_id: "<your-client-id>"
|
||||
client_secret_ref: env:SIGMA_CLIENT_SECRET
|
||||
workbookFilter:
|
||||
includeArchived: false
|
||||
includeExplorations: false
|
||||
updatedSince: "2026-01-01T00:00:00Z"
|
||||
```
|
||||
|
||||
| Field | Purpose |
|
||||
|-------|---------|
|
||||
| `api_url` | Sigma API base URL. Defaults to `https://api.sigmacomputing.com` (GCP US). Override for AWS US (`https://aws-api.sigmacomputing.com`) or other regions. |
|
||||
| `client_id` | Sigma OAuth client ID. Required. |
|
||||
| `client_secret` / `client_secret_ref` | Literal secret or reference. Prefer the `_ref`. |
|
||||
| `connectionMappings` | Maps Sigma internal connection UUIDs to **ktx** warehouse connection IDs. Enables `sl_validate` for projected semantic-layer sources. |
|
||||
| `workbookFilter.includeArchived` | Include archived workbooks during ingest. Default: `false`. |
|
||||
| `workbookFilter.includeExplorations` | Include exploration workbooks during ingest. Default: `false`. |
|
||||
| `workbookFilter.updatedSince` | ISO 8601 date string. Only workbooks updated on or after this date are fetched. Useful for limiting ingest scope at large scale. |
|
||||
|
||||
## `setup`
|
||||
|
||||
Captured by the setup wizard. The only field **ktx** still reads is
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue