ktx/packages/cli/test/context/ingest/adapters/sigma/sigma.adapter.test.ts
Matt Senick (Sigma) acd20ac248
feat(sigma): add Sigma Computing context-source adapter (#316)
* feat(sigma): add Sigma Computing context-source adapter

Closes #168

Adds a full ingest adapter for Sigma Computing so `ktx ingest` can pull
data model specs and workbook summaries into the ktx context layer. The
implementation follows the same fetch → chunk → project → LLM pattern
used by the Looker, Metabase, and MetricFlow adapters.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(sigma): address PR review comments

- Remove manifest from rawFiles; moves to peerFileIndex so fetchedAt
  changes don't mark all work units dirty every run
- Fix workbookFilter.updatedSince eviction bug: fetch full universe first,
  apply filter client-side, evict only on archived/deleted
- Remove measure projection entirely; project() writes measures: [] and
  the sigma_ingest skill surfaces Lookup/aggregation formulas as wiki prose
- Remove joins projection (v1 limitation); project() writes joins: [] and
  Lookup relationships are described in wiki prose instead
- Remove write-back dead code: createDataModel, updateDataModel,
  SigmaDataModelPushResult, mutate/post/put
- Fix emitBatches notes pluralization bug ('2 data modelss' → '2 data models')
- Add tokenInflight dedup on ensureToken to coalesce concurrent auth requests
- Retry spec fetch when existing staged spec is null (transient failure cache)
- Drop unused WorkbookFilter import from client-port.ts
- Note in docs that joins are not projected from Sigma data models in this release

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* updates

* fix(sigma): restore sigma in local adapter test + small cleanups

The gdrive↔sigma merge dropped 'sigma' from the expected adapter source
list in local-adapters.test.ts while keeping gdrive, so the slow TS suite
failed even though the source registers both. Add 'sigma' back at its
registration position (after metabase, before gdrive).

Also:
- Move the orphaned SigmaPullConfig docstring onto the schema it documents
  and drop the stale BullMQ reference (standalone ktx has no BullMQ; the
  config lives in the ingest job's bundleRef.config).
- Drop an O(n^2) find() round-trip in fetch() when building the active
  data-model list; filter once and reuse for the eviction id set.

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Andrey Avtomonov <andreybavt@gmail.com>
Co-authored-by: Luca Martial <48870843+luca-martial@users.noreply.github.com>
2026-07-01 01:14:57 +02:00

64 lines
2.6 KiB
TypeScript

import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { SigmaSourceAdapter } from '../../../../../src/context/ingest/adapters/sigma/sigma.adapter.js';
import type { SigmaClientFactory } from '../../../../../src/context/ingest/adapters/sigma/client-port.js';
function makeFactory(): SigmaClientFactory {
return { createClient: vi.fn() };
}
describe('SigmaSourceAdapter.listTargetConnectionIds', () => {
let stagedDir: string;
beforeEach(async () => {
stagedDir = await mkdtemp(join(tmpdir(), 'sigma-adapter-'));
});
afterEach(async () => {
await rm(stagedDir, { recursive: true, force: true });
});
async function writeProjectionConfig(mappings: Record<string, string>) {
await writeFile(
join(stagedDir, 'sigma-projection-config.json'),
JSON.stringify({ connectionMappings: mappings }),
'utf-8',
);
}
it('returns mapped warehouse connection IDs when mappings are present', async () => {
await writeProjectionConfig({ 'uuid-a': 'snowflake-prod', 'uuid-b': 'snowflake-prod', 'uuid-c': 'bigquery-prod' });
const adapter = new SigmaSourceAdapter({ clientFactory: makeFactory() });
const ids = await adapter.listTargetConnectionIds(stagedDir);
expect(ids).toEqual(['bigquery-prod', 'snowflake-prod']);
});
it('returns empty array when connectionMappings is empty', async () => {
await writeProjectionConfig({});
const adapter = new SigmaSourceAdapter({ clientFactory: makeFactory() });
const ids = await adapter.listTargetConnectionIds(stagedDir);
expect(ids).toEqual([]);
});
it('returns empty array when the projection config file is missing', async () => {
const adapter = new SigmaSourceAdapter({ clientFactory: makeFactory() });
const ids = await adapter.listTargetConnectionIds(stagedDir);
expect(ids).toEqual([]);
});
it('returns empty array when the projection config is malformed', async () => {
await mkdir(stagedDir, { recursive: true });
await writeFile(join(stagedDir, 'sigma-projection-config.json'), 'not json', 'utf-8');
const adapter = new SigmaSourceAdapter({ clientFactory: makeFactory() });
const ids = await adapter.listTargetConnectionIds(stagedDir);
expect(ids).toEqual([]);
});
it('returns empty array when both projection config and manifest are missing', async () => {
const adapter = new SigmaSourceAdapter({ clientFactory: makeFactory() });
const ids = await adapter.listTargetConnectionIds(stagedDir);
expect(ids).toEqual([]);
});
});