mirror of
https://github.com/Kaelio/ktx.git
synced 2026-06-22 08:38:08 +02:00
* refactor(workspace): relocate @ktx/llm source into packages/cli/src/llm * refactor(workspace): rewrite @ktx/llm imports to relative paths * refactor(workspace): fold internal packages into cli * chore(workspace): gate dead-code with knip production mode Turn on production-mode knip plus an autofix run in pre-commit and the `pnpm dead-code` script, document the `/** @internal */` convention for test-only exports in AGENTS.md, annotate test-only exports across the CLI with that JSDoc, and drop dead exports/wrappers the new gate surfaced (e.g. `cli-project.ts`, `lookerRuntimeSourceToFileAdapterSource`, `createLocalScanEnrichmentProvidersFromConfig`, `PGLITE_OWNER_PROCESS_BACKEND_CAPABILITIES`, stale type re-exports). Replace the loose `ignoreIssues` allowlist in `knip.json` with explicit production entries so cross-package barrel leaks are caught. * refactor(cli): delete internal barrel index.ts files The 34 `index.ts` re-export barrels inside `packages/cli/src/` were holdovers from the pre-fold multi-workspace structure. Post-fold-in they served no production purpose: external consumers go through the single package main entry, and in-repo callers mostly imported through them only because the path was short. Internally, knip flagged most barrel re-exports as production-dead (only reached via tests). This change: - Deletes every internal barrel except `packages/cli/src/index.ts` (the published package entry). - Rewrites ~270 source/test files to import each name directly from the file that defines it. - Moves `tools/warehouse-verification/index.ts` to `create-warehouse-verification-tools.ts` (the function it defined locally) and updates its single consumer. - Renames `search/backend-conformance.ts` → `.test-utils.ts` to match the existing test-helper file convention. - Deletes 13 dead test-only chains (dbt-descriptions/*, live-database/extracted-schema, live-database/structural-sync, relationship-* feedback/review chain) plus their tests and a cascading orphan integration test. - Updates test mocks that pointed at deleted barrel paths (notion-client, connector barrels in scan/local-scan-connectors tests) to mock the source files instead. - Points the maintainer benchmark script (`scripts/relationship-benchmark-report.mjs`) at source files instead of `dist/context/scan/index.js`. - Drops the barrel `!` entries from `knip.json`; adds explicit production entries only for the benchmark code reached via dist by the maintainer script. Net: 413 files changed, ~1.2k insertions, ~9.4k deletions. `pnpm run dead-code` (Biome + knip default + knip production) and `pnpm run type-check` are clean; 2277 tests pass. * refactor(workspace): rename @ktx/cli to @kaelio/ktx and pack it directly Promote the CLI workspace package to the public name `@kaelio/ktx` and drop the separate `scripts/build-public-npm-package.mjs` wrapper. The CLI package is now publishable in place (`publishConfig.access: public`, `provenance: true`), so artifact packing uses `pnpm pack` against `packages/cli/` instead of assembling a parallel package tree. Updates all workspace filter invocations, docs, tests, and release readiness checks to reference the new package name, and folds the tarball-name helper into `scripts/public-npm-release-metadata.mjs`. * docs: align "agent clients" and "data agents" terminology Replace "client agents" with "agent clients" and "database agents" with "data agents" across AGENTS.md, README.md, the docs-site copy, and the matching setup-agents test description, matching the canonical vocabulary in docs/terminology.md. Also moves packages/cli/tsconfig.json's tsBuildInfoFile from node_modules/.cache/ to dist/.tsbuildinfo so incremental builds survive node_modules reinstalls. * refactor(release): single source of truth for package version Make packages/cli/package.json the single source of truth for the @kaelio/ktx version. publicNpmPackageVersion() now reads it directly, so artifact filenames, release-readiness checks, and the Python wheel version all derive from one field. The duplicate release-policy.json.publicNpmPackageVersion is removed. Previously the two fields could drift: tarballs were named kaelio-ktx-0.4.1.tgz while internally containing @kaelio/ktx@0.0.0-private. - update-public-release-version.mjs rewrites both Python pyproject.toml files (ktx-daemon, ktx-sl) alongside the npm package.jsons, normalizing the version for PEP 440 (e.g. 0.1.0-rc.2 -> 0.1.0rc2). - semantic-release-config.cjs adds the two pyproject.toml files to @semantic-release/git assets so the release commit back to main carries every version source in lockstep. - The six "?? '0.0.0-private'" fallback literals across the CLI are replaced with "?? getKtxCliPackageInfo().version", and createDefaultKtxMcpServer makes its version arg required. - docs/release.md describes the actual commit-back model: the dev tree always reflects the most recent release; no sentinel pin to maintain. Verified: pnpm run artifacts:build now produces kaelio-ktx-0.4.1.tgz and kaelio_ktx-0.4.1-py3-none-any.whl with @kaelio/ktx@0.4.1 inside. Full type-check, dead-code, and 2287 vitests + 173 script tests pass. * refactor(cli): inject embedding provider resolution and detect sentence-transformers runtime Make resolveProjectEmbeddingProvider and runtimeIo injectable in ingest and scan command entrypoints so tests can stub them, and teach resolvePublicIngestRuntimeRequirements to flag the local-embeddings runtime feature when ktx.yaml selects sentence-transformers. * chore(cli): mark buildLocalStatsStatus and LocalStatsStatus as @internal Both symbols are consumed only by status-project.test.ts. Annotating with /** @internal */ keeps knip's production-mode check clean without changing runtime behavior. * fix(cli): use real package metadata in print-command-tree The stubbed package name embedded a forbidden product identifier that tripped the boundary check in CI. Read the metadata from package.json instead — keeps the rendered tree unchanged and removes a duplicate source of truth. * feat(cli): show embedding coverage in `ktx status`, drop duplicate disk counts Inline `(N embedded)` next to the Wiki scope counts and Semantic-layer source counts, computed with `SUM(embedding_json IS NOT NULL)` over `knowledge_pages` and `local_sl_sources`. Rename the "Knowledge" label to "Wiki" (canonical per `docs/terminology.md`) and rename the matching `localStats.knowledgePages` field to `localStats.wikiPages`. Drop `wiki=N md` and `semantic-layer=N yaml` from the Disk row — those duplicated the per-surface rows above. Disk now reports only actual byte usage (db, cache, raw-sources). The unused `wikiGlobalMarkdownCount` / `semanticLayerYamlCount` fields, the `isMarkdownEntry` / `isYamlEntry` helpers, and the `filter` arg on `summarizeDir` are removed.
645 lines
24 KiB
TypeScript
645 lines
24 KiB
TypeScript
import { mkdtemp, readdir, readFile, rm } from 'node:fs/promises';
|
|
import { tmpdir } from 'node:os';
|
|
import { join } from 'node:path';
|
|
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
|
import { chunkLookerStagedDir } from './chunk.js';
|
|
import { fetchLookerRuntimeBundle, type LookerRuntimeClient } from './fetch.js';
|
|
|
|
const connectionId = '11111111-1111-4111-8111-111111111111';
|
|
|
|
function makeClient(): LookerRuntimeClient {
|
|
return {
|
|
listDashboards: vi.fn().mockResolvedValue([{ id: '10' }]),
|
|
getDashboard: vi.fn().mockResolvedValue({
|
|
lookerId: '10',
|
|
title: 'Sales Pipeline',
|
|
description: 'Pipeline health',
|
|
folderId: '7',
|
|
ownerId: '3',
|
|
updatedAt: '2026-04-30T12:00:00.000Z',
|
|
tiles: [{ id: '100', title: 'ARR', lookId: null, query: { model: 'b2b', view: 'sales_pipeline' } }],
|
|
}),
|
|
listLooks: vi.fn().mockResolvedValue([{ id: '20' }]),
|
|
getLook: vi.fn().mockResolvedValue({
|
|
lookerId: '20',
|
|
title: 'Open Pipeline',
|
|
description: null,
|
|
folderId: '7',
|
|
ownerId: '3',
|
|
updatedAt: '2026-04-30T12:00:00.000Z',
|
|
query: { model: 'b2b', view: 'sales_pipeline', fields: ['opportunities.arr'] },
|
|
}),
|
|
listFolders: vi
|
|
.fn()
|
|
.mockResolvedValue({ folders: [{ id: '7', name: 'Sandbox', parentId: null, path: ['Sandbox'] }] }),
|
|
listUsers: vi.fn().mockResolvedValue([{ id: '3', displayName: 'Ada Lovelace', email: null }]),
|
|
listGroups: vi.fn().mockResolvedValue([{ id: '4', name: 'Sales' }]),
|
|
listLookmlModels: vi.fn().mockResolvedValue({
|
|
models: [{ name: 'b2b', label: 'B2B', explores: [{ name: 'sales_pipeline', label: 'Sales Pipeline' }] }],
|
|
}),
|
|
getExplore: vi.fn().mockResolvedValue({
|
|
modelName: 'b2b',
|
|
exploreName: 'sales_pipeline',
|
|
label: 'Sales Pipeline',
|
|
description: null,
|
|
fields: { dimensions: [{ name: 'opportunities.id' }], measures: [{ name: 'opportunities.arr' }] },
|
|
joins: [],
|
|
}),
|
|
getSignals: vi.fn().mockResolvedValue({
|
|
dashboardUsage: [{ contentId: '10', queryCount30d: 50, uniqueUsers30d: 8, lastRunAt: null, topUsers: ['3'] }],
|
|
lookUsage: [{ contentId: '20', queryCount30d: 20, uniqueUsers30d: 5, lastRunAt: null, topUsers: ['3'] }],
|
|
scheduledPlans: [
|
|
{ contentId: '10', contentType: 'dashboard', isScheduled: true, scheduleCount: 1, recipientCount: 3 },
|
|
],
|
|
favorites: [{ contentId: '10', contentType: 'dashboard', favoriteCount: 4 }],
|
|
}),
|
|
cleanup: vi.fn().mockResolvedValue(undefined),
|
|
};
|
|
}
|
|
|
|
describe('fetchLookerRuntimeBundle', () => {
|
|
let stagedDir: string;
|
|
|
|
beforeEach(async () => {
|
|
stagedDir = await mkdtemp(join(tmpdir(), 'looker-fetch-'));
|
|
});
|
|
|
|
afterEach(async () => {
|
|
await rm(stagedDir, { recursive: true, force: true });
|
|
});
|
|
|
|
it('writes dashboards, looks, folders, users, groups, models, explores, signals, and sync config', async () => {
|
|
const client = makeClient();
|
|
await fetchLookerRuntimeBundle({
|
|
pullConfig: { lookerConnectionId: connectionId, instanceBaseUrl: 'https://example.looker.com' },
|
|
stagedDir,
|
|
ctx: { connectionId, sourceKey: 'looker' },
|
|
clientFactory: { createClient: vi.fn().mockResolvedValue(client) },
|
|
now: () => new Date('2026-04-30T12:30:00.000Z'),
|
|
});
|
|
|
|
expect(await readdir(join(stagedDir, 'dashboards'))).toEqual(['10.json']);
|
|
expect(await readdir(join(stagedDir, 'looks'))).toEqual(['20.json']);
|
|
expect(await readdir(join(stagedDir, 'users'))).toEqual(['3.json']);
|
|
expect(await readdir(join(stagedDir, 'groups'))).toEqual(['4.json']);
|
|
expect(await readdir(join(stagedDir, 'explores/b2b'))).toEqual(['sales_pipeline.json']);
|
|
|
|
const syncConfig = JSON.parse(await readFile(join(stagedDir, 'sync-config.json'), 'utf-8'));
|
|
expect(syncConfig).toEqual({
|
|
lookerConnectionId: connectionId,
|
|
fetchedAt: '2026-04-30T12:30:00.000Z',
|
|
instanceBaseUrl: 'https://example.looker.com',
|
|
previousCursors: {
|
|
dashboardsLastSyncedAt: null,
|
|
looksLastSyncedAt: null,
|
|
},
|
|
nextCursors: {
|
|
dashboardsLastSyncedAt: null,
|
|
looksLastSyncedAt: null,
|
|
},
|
|
});
|
|
|
|
const scope = JSON.parse(await readFile(join(stagedDir, 'looker-scope.json'), 'utf-8'));
|
|
expect(scope).toEqual({
|
|
mode: 'full',
|
|
knownCurrentRawPaths: ['dashboards/10.json', 'looks/20.json'],
|
|
fetchedRawPaths: ['dashboards/10.json', 'looks/20.json'],
|
|
});
|
|
|
|
const dashboardUsage = JSON.parse(await readFile(join(stagedDir, 'signals/dashboard_usage.json'), 'utf-8'));
|
|
expect(dashboardUsage).toEqual([
|
|
{ contentId: '10', queryCount30d: 50, uniqueUsers30d: 8, lastRunAt: null, topUsers: ['3'] },
|
|
]);
|
|
|
|
const lookUsage = JSON.parse(await readFile(join(stagedDir, 'signals/look_usage.json'), 'utf-8'));
|
|
const scheduledPlans = JSON.parse(await readFile(join(stagedDir, 'signals/scheduled_plans.json'), 'utf-8'));
|
|
const favorites = JSON.parse(await readFile(join(stagedDir, 'signals/favorites.json'), 'utf-8'));
|
|
|
|
expect(lookUsage).toEqual([
|
|
{ contentId: '20', queryCount30d: 20, uniqueUsers30d: 5, lastRunAt: null, topUsers: ['3'] },
|
|
]);
|
|
expect(scheduledPlans).toEqual([
|
|
{ contentId: '10', contentType: 'dashboard', isScheduled: true, scheduleCount: 1, recipientCount: 3 },
|
|
]);
|
|
expect(favorites).toEqual([{ contentId: '10', contentType: 'dashboard', favoriteCount: 4 }]);
|
|
});
|
|
|
|
it('stages only changed Dashboard and Look entity bodies during incremental pulls', async () => {
|
|
const client = makeClient();
|
|
vi.mocked(client.listDashboards).mockResolvedValue([
|
|
{ id: '10', updatedAt: '2026-04-30T12:00:00.000Z' },
|
|
{ id: '11', updatedAt: '2026-04-30T12:10:00.000Z' },
|
|
]);
|
|
vi.mocked(client.getDashboard).mockImplementation(async (id: string) => ({
|
|
lookerId: id,
|
|
title: `Dashboard ${id}`,
|
|
description: null,
|
|
folderId: '7',
|
|
ownerId: '3',
|
|
updatedAt: id === '11' ? '2026-04-30T12:10:00.000Z' : '2026-04-30T12:00:00.000Z',
|
|
tiles: [],
|
|
}));
|
|
vi.mocked(client.listLooks).mockResolvedValue([
|
|
{ id: '20', updatedAt: '2026-04-30T11:00:00.000Z' },
|
|
{ id: '21', updatedAt: null },
|
|
]);
|
|
vi.mocked(client.getLook).mockImplementation(async (id: string) => ({
|
|
lookerId: id,
|
|
title: `Look ${id}`,
|
|
description: null,
|
|
folderId: '7',
|
|
ownerId: '3',
|
|
updatedAt: id === '21' ? null : '2026-04-30T11:00:00.000Z',
|
|
query: null,
|
|
}));
|
|
|
|
await fetchLookerRuntimeBundle({
|
|
pullConfig: {
|
|
lookerConnectionId: connectionId,
|
|
dashboardUpdatedSince: '2026-04-30T12:00:00.000Z',
|
|
lookUpdatedSince: '2026-04-30T11:00:00.000Z',
|
|
},
|
|
stagedDir,
|
|
ctx: { connectionId, sourceKey: 'looker' },
|
|
clientFactory: { createClient: vi.fn().mockResolvedValue(client) },
|
|
now: () => new Date('2026-04-30T12:30:00.000Z'),
|
|
});
|
|
|
|
expect(client.getDashboard).toHaveBeenCalledTimes(1);
|
|
expect(client.getDashboard).toHaveBeenCalledWith('11');
|
|
expect(client.getLook).toHaveBeenCalledTimes(1);
|
|
expect(client.getLook).toHaveBeenCalledWith('21');
|
|
|
|
await expect(readdir(join(stagedDir, 'dashboards'))).resolves.toEqual(['11.json']);
|
|
await expect(readdir(join(stagedDir, 'looks'))).resolves.toEqual(['21.json']);
|
|
|
|
const syncConfig = JSON.parse(await readFile(join(stagedDir, 'sync-config.json'), 'utf-8'));
|
|
expect(syncConfig.previousCursors).toEqual({
|
|
dashboardsLastSyncedAt: '2026-04-30T12:00:00.000Z',
|
|
looksLastSyncedAt: '2026-04-30T11:00:00.000Z',
|
|
});
|
|
expect(syncConfig.nextCursors).toEqual({
|
|
dashboardsLastSyncedAt: '2026-04-30T12:10:00.000Z',
|
|
looksLastSyncedAt: '2026-04-30T11:00:00.000Z',
|
|
});
|
|
|
|
const scope = JSON.parse(await readFile(join(stagedDir, 'looker-scope.json'), 'utf-8'));
|
|
expect(scope).toEqual({
|
|
mode: 'incremental',
|
|
knownCurrentRawPaths: ['dashboards/10.json', 'dashboards/11.json', 'looks/20.json', 'looks/21.json'],
|
|
fetchedRawPaths: ['dashboards/11.json', 'looks/21.json'],
|
|
});
|
|
});
|
|
|
|
it('falls back to empty signal files when the client has no signal support', async () => {
|
|
const client = makeClient();
|
|
delete client.getSignals;
|
|
|
|
await fetchLookerRuntimeBundle({
|
|
pullConfig: { lookerConnectionId: connectionId },
|
|
stagedDir,
|
|
ctx: { connectionId, sourceKey: 'looker' },
|
|
clientFactory: { createClient: vi.fn().mockResolvedValue(client) },
|
|
now: () => new Date('2026-04-30T12:30:00.000Z'),
|
|
});
|
|
|
|
expect(JSON.parse(await readFile(join(stagedDir, 'signals/look_usage.json'), 'utf-8'))).toEqual([]);
|
|
});
|
|
|
|
it('stamps explore warehouse targets from pull config and reports unmapped Looker connections', async () => {
|
|
const client = makeClient();
|
|
const warehouseConnectionId = '22222222-2222-4222-8222-222222222222';
|
|
vi.mocked(client.listLookmlModels).mockResolvedValue({
|
|
models: [
|
|
{
|
|
name: 'b2b',
|
|
label: 'B2B',
|
|
explores: [
|
|
{ name: 'sales_pipeline', label: 'Sales Pipeline' },
|
|
{ name: 'marketing', label: 'Marketing' },
|
|
],
|
|
},
|
|
],
|
|
});
|
|
vi.mocked(client.getExplore).mockImplementation(async (_modelName: string, exploreName: string) => {
|
|
if (exploreName === 'marketing') {
|
|
return {
|
|
modelName: 'b2b',
|
|
exploreName: 'marketing',
|
|
label: 'Marketing',
|
|
description: null,
|
|
rawSqlTableName: 'proj.dataset.marketing',
|
|
connectionName: 'missing_mapping',
|
|
viewName: 'marketing',
|
|
fields: {
|
|
dimensions: [{ name: 'marketing.id', label: null, type: null, sql: null, description: null }],
|
|
measures: [{ name: 'marketing.spend', label: null, type: null, sql: null, description: null }],
|
|
},
|
|
joins: [],
|
|
targetWarehouseConnectionId: null,
|
|
targetTable: null,
|
|
};
|
|
}
|
|
return {
|
|
modelName: 'b2b',
|
|
exploreName: 'sales_pipeline',
|
|
label: 'Sales Pipeline',
|
|
description: null,
|
|
rawSqlTableName: 'proj.dataset.opportunities AS opportunities',
|
|
connectionName: 'b2b_sandbox_bq',
|
|
viewName: 'opportunities',
|
|
fields: {
|
|
dimensions: [{ name: 'opportunities.id', label: null, type: null, sql: null, description: null }],
|
|
measures: [{ name: 'opportunities.arr', label: null, type: null, sql: null, description: null }],
|
|
},
|
|
joins: [
|
|
{
|
|
name: 'accounts',
|
|
type: 'left_outer',
|
|
relationship: 'many_to_one',
|
|
rawSqlTableName: 'proj.dataset.accounts',
|
|
sqlOn: '$' + '{opportunities.account_id} = $' + '{accounts.id}',
|
|
from: null,
|
|
targetTable: null,
|
|
},
|
|
],
|
|
targetWarehouseConnectionId: null,
|
|
targetTable: null,
|
|
};
|
|
});
|
|
|
|
await fetchLookerRuntimeBundle({
|
|
pullConfig: {
|
|
lookerConnectionId: connectionId,
|
|
connectionMappings: { b2b_sandbox_bq: warehouseConnectionId },
|
|
connectionTypes: { b2b_sandbox_bq: 'BIGQUERY' },
|
|
parsedTargetTables: {
|
|
'b2b.sales_pipeline': {
|
|
ok: true,
|
|
catalog: 'proj',
|
|
schema: 'dataset',
|
|
name: 'opportunities',
|
|
canonicalTable: 'proj.dataset.opportunities',
|
|
},
|
|
'b2b.sales_pipeline.accounts': {
|
|
ok: true,
|
|
catalog: 'proj',
|
|
schema: 'dataset',
|
|
name: 'accounts',
|
|
canonicalTable: 'proj.dataset.accounts',
|
|
},
|
|
},
|
|
},
|
|
stagedDir,
|
|
ctx: { connectionId, sourceKey: 'looker' },
|
|
clientFactory: { createClient: vi.fn().mockResolvedValue(client) },
|
|
now: () => new Date('2026-04-30T12:30:00.000Z'),
|
|
});
|
|
|
|
const salesPipeline = JSON.parse(await readFile(join(stagedDir, 'explores/b2b/sales_pipeline.json'), 'utf-8'));
|
|
expect(salesPipeline).toMatchObject({
|
|
connectionName: 'b2b_sandbox_bq',
|
|
targetWarehouseConnectionId: warehouseConnectionId,
|
|
targetTable: {
|
|
ok: true,
|
|
catalog: 'proj',
|
|
schema: 'dataset',
|
|
name: 'opportunities',
|
|
canonicalTable: 'proj.dataset.opportunities',
|
|
},
|
|
joins: [
|
|
{
|
|
name: 'accounts',
|
|
targetTable: {
|
|
ok: true,
|
|
catalog: 'proj',
|
|
schema: 'dataset',
|
|
name: 'accounts',
|
|
canonicalTable: 'proj.dataset.accounts',
|
|
},
|
|
},
|
|
],
|
|
});
|
|
|
|
const marketing = JSON.parse(await readFile(join(stagedDir, 'explores/b2b/marketing.json'), 'utf-8'));
|
|
expect(marketing).toMatchObject({
|
|
connectionName: 'missing_mapping',
|
|
targetWarehouseConnectionId: null,
|
|
targetTable: {
|
|
ok: false,
|
|
reason: 'no_connection_mapping',
|
|
},
|
|
});
|
|
|
|
const report = JSON.parse(await readFile(join(stagedDir, 'looker-fetch-report.json'), 'utf-8'));
|
|
expect(report.status).toBe('partial');
|
|
expect(report.skipped).toEqual([]);
|
|
expect(report.warnings).toEqual([
|
|
{
|
|
rawPath: 'looker_connection_mappings/missing_mapping',
|
|
entityType: 'looker_connection_mapping',
|
|
entityId: 'missing_mapping',
|
|
severity: 'warning',
|
|
statusCode: null,
|
|
message: 'Looker connection missing_mapping is not mapped to a warehouse connection; 1 explore will be wiki-only.',
|
|
retryRecommended: false,
|
|
kind: 'unmapped_looker_connection',
|
|
details: {
|
|
lookerConnectionName: 'missing_mapping',
|
|
affectedExplores: ['b2b.marketing'],
|
|
},
|
|
},
|
|
]);
|
|
});
|
|
|
|
it('reports parsed target table failures without retrying the Looker fetch', async () => {
|
|
const client = makeClient();
|
|
const warehouseConnectionId = '22222222-2222-4222-8222-222222222222';
|
|
vi.mocked(client.getExplore).mockResolvedValue({
|
|
modelName: 'b2b',
|
|
exploreName: 'sales_pipeline',
|
|
label: 'Sales Pipeline',
|
|
description: null,
|
|
rawSqlTableName: '$' + '{derived.SQL_TABLE_NAME}',
|
|
connectionName: 'b2b_sandbox_bq',
|
|
viewName: 'opportunities',
|
|
fields: {
|
|
dimensions: [{ name: 'opportunities.id', label: null, type: null, sql: null, description: null }],
|
|
measures: [{ name: 'opportunities.arr', label: null, type: null, sql: null, description: null }],
|
|
},
|
|
joins: [],
|
|
targetWarehouseConnectionId: null,
|
|
targetTable: null,
|
|
});
|
|
|
|
await fetchLookerRuntimeBundle({
|
|
pullConfig: {
|
|
lookerConnectionId: connectionId,
|
|
connectionMappings: { b2b_sandbox_bq: warehouseConnectionId },
|
|
connectionTypes: { b2b_sandbox_bq: 'BIGQUERY' },
|
|
parsedTargetTables: {
|
|
'b2b.sales_pipeline': {
|
|
ok: false,
|
|
reason: 'looker_template_unresolved',
|
|
detail: 'Looker template markers cannot be resolved before parsing.',
|
|
},
|
|
},
|
|
},
|
|
stagedDir,
|
|
ctx: { connectionId, sourceKey: 'looker' },
|
|
clientFactory: { createClient: vi.fn().mockResolvedValue(client) },
|
|
now: () => new Date('2026-04-30T12:30:00.000Z'),
|
|
});
|
|
|
|
const explore = JSON.parse(await readFile(join(stagedDir, 'explores/b2b/sales_pipeline.json'), 'utf-8'));
|
|
expect(explore).toMatchObject({
|
|
targetWarehouseConnectionId: warehouseConnectionId,
|
|
targetTable: {
|
|
ok: false,
|
|
reason: 'looker_template_unresolved',
|
|
},
|
|
});
|
|
|
|
const report = JSON.parse(await readFile(join(stagedDir, 'looker-fetch-report.json'), 'utf-8'));
|
|
expect(report).toMatchObject({
|
|
status: 'partial',
|
|
retryRecommended: false,
|
|
skipped: [],
|
|
warnings: [
|
|
{
|
|
rawPath: 'looker_connection_mappings/b2b_sandbox_bq',
|
|
entityType: 'looker_connection_mapping',
|
|
entityId: 'b2b_sandbox_bq',
|
|
severity: 'warning',
|
|
statusCode: null,
|
|
message:
|
|
'Looker explore b2b.sales_pipeline has sql_table_name that cannot be mapped to a physical warehouse table: looker_template_unresolved.',
|
|
retryRecommended: false,
|
|
kind: 'looker_template_unresolved',
|
|
details: {
|
|
lookerConnectionName: 'b2b_sandbox_bq',
|
|
rawSqlTableName: '$' + '{derived.SQL_TABLE_NAME}',
|
|
reason: 'looker_template_unresolved',
|
|
},
|
|
},
|
|
],
|
|
});
|
|
});
|
|
|
|
it('propagates parent explore warehouse targets onto Dashboard tile and Look queries', async () => {
|
|
const client = makeClient();
|
|
const warehouseConnectionId = '22222222-2222-4222-8222-222222222222';
|
|
vi.mocked(client.getExplore).mockResolvedValue({
|
|
modelName: 'b2b',
|
|
exploreName: 'sales_pipeline',
|
|
label: 'Sales Pipeline',
|
|
description: null,
|
|
rawSqlTableName: 'proj.dataset.opportunities AS opportunities',
|
|
connectionName: 'b2b_sandbox_bq',
|
|
viewName: 'opportunities',
|
|
fields: {
|
|
dimensions: [{ name: 'opportunities.id', label: null, type: null, sql: null, description: null }],
|
|
measures: [{ name: 'opportunities.arr', label: null, type: null, sql: null, description: null }],
|
|
},
|
|
joins: [],
|
|
targetWarehouseConnectionId: null,
|
|
targetTable: null,
|
|
});
|
|
|
|
await fetchLookerRuntimeBundle({
|
|
pullConfig: {
|
|
lookerConnectionId: connectionId,
|
|
connectionMappings: { b2b_sandbox_bq: warehouseConnectionId },
|
|
connectionTypes: { b2b_sandbox_bq: 'BIGQUERY' },
|
|
parsedTargetTables: {
|
|
'b2b.sales_pipeline': {
|
|
ok: true,
|
|
catalog: 'proj',
|
|
schema: 'dataset',
|
|
name: 'opportunities',
|
|
canonicalTable: 'proj.dataset.opportunities',
|
|
},
|
|
},
|
|
},
|
|
stagedDir,
|
|
ctx: { connectionId, sourceKey: 'looker' },
|
|
clientFactory: { createClient: vi.fn().mockResolvedValue(client) },
|
|
now: () => new Date('2026-04-30T12:30:00.000Z'),
|
|
});
|
|
|
|
const dashboard = JSON.parse(await readFile(join(stagedDir, 'dashboards/10.json'), 'utf-8'));
|
|
expect(dashboard.tiles[0].query).toMatchObject({
|
|
model: 'b2b',
|
|
view: 'sales_pipeline',
|
|
targetWarehouseConnectionId: warehouseConnectionId,
|
|
targetTable: {
|
|
ok: true,
|
|
catalog: 'proj',
|
|
schema: 'dataset',
|
|
name: 'opportunities',
|
|
canonicalTable: 'proj.dataset.opportunities',
|
|
},
|
|
});
|
|
|
|
const look = JSON.parse(await readFile(join(stagedDir, 'looks/20.json'), 'utf-8'));
|
|
expect(look.query).toMatchObject({
|
|
model: 'b2b',
|
|
view: 'sales_pipeline',
|
|
targetWarehouseConnectionId: warehouseConnectionId,
|
|
targetTable: {
|
|
ok: true,
|
|
catalog: 'proj',
|
|
schema: 'dataset',
|
|
name: 'opportunities',
|
|
canonicalTable: 'proj.dataset.opportunities',
|
|
},
|
|
});
|
|
});
|
|
|
|
it('records skipped detail entities and keeps cursors pinned for affected entity types', async () => {
|
|
const client = makeClient();
|
|
vi.mocked(client.listDashboards).mockResolvedValue([
|
|
{ id: '10', updatedAt: '2026-04-30T12:00:00.000Z' },
|
|
{ id: '11', updatedAt: '2026-04-30T12:10:00.000Z' },
|
|
]);
|
|
vi.mocked(client.getDashboard).mockImplementation(async (id: string) => {
|
|
if (id === '11') {
|
|
const error = new Error('Looker API rate limit remained after retry');
|
|
Object.assign(error, { statusCode: 429 });
|
|
throw error;
|
|
}
|
|
return {
|
|
lookerId: id,
|
|
title: `Dashboard ${id}`,
|
|
description: null,
|
|
folderId: '7',
|
|
ownerId: '3',
|
|
updatedAt: '2026-04-30T12:00:00.000Z',
|
|
tiles: [],
|
|
};
|
|
});
|
|
vi.mocked(client.listLooks).mockResolvedValue([{ id: '20', updatedAt: '2026-04-30T11:15:00.000Z' }]);
|
|
vi.mocked(client.getLook).mockResolvedValue({
|
|
lookerId: '20',
|
|
title: 'Look 20',
|
|
description: null,
|
|
folderId: '7',
|
|
ownerId: '3',
|
|
updatedAt: '2026-04-30T11:15:00.000Z',
|
|
query: null,
|
|
});
|
|
|
|
await fetchLookerRuntimeBundle({
|
|
pullConfig: {
|
|
lookerConnectionId: connectionId,
|
|
dashboardUpdatedSince: '2026-04-30T12:00:00.000Z',
|
|
lookUpdatedSince: '2026-04-30T11:00:00.000Z',
|
|
},
|
|
stagedDir,
|
|
ctx: { connectionId, sourceKey: 'looker' },
|
|
clientFactory: { createClient: vi.fn().mockResolvedValue(client) },
|
|
now: () => new Date('2026-04-30T12:30:00.000Z'),
|
|
});
|
|
|
|
await expect(readdir(join(stagedDir, 'dashboards'))).rejects.toMatchObject({ code: 'ENOENT' });
|
|
await expect(readdir(join(stagedDir, 'looks'))).resolves.toEqual(['20.json']);
|
|
|
|
const syncConfig = JSON.parse(await readFile(join(stagedDir, 'sync-config.json'), 'utf-8'));
|
|
expect(syncConfig.nextCursors).toEqual({
|
|
dashboardsLastSyncedAt: '2026-04-30T12:00:00.000Z',
|
|
looksLastSyncedAt: '2026-04-30T11:15:00.000Z',
|
|
});
|
|
|
|
const report = JSON.parse(await readFile(join(stagedDir, 'looker-fetch-report.json'), 'utf-8'));
|
|
expect(report).toEqual({
|
|
status: 'partial',
|
|
retryRecommended: true,
|
|
skipped: [
|
|
{
|
|
rawPath: 'dashboards/11.json',
|
|
entityType: 'dashboard',
|
|
entityId: '11',
|
|
severity: 'error',
|
|
statusCode: 429,
|
|
message: 'Looker API rate limit remained after retry',
|
|
retryRecommended: true,
|
|
},
|
|
],
|
|
warnings: [],
|
|
});
|
|
});
|
|
|
|
it('continues without explore bootstrap when LookML model listing is denied', async () => {
|
|
const client = makeClient();
|
|
const error = new Error('LookML model access denied');
|
|
Object.assign(error, { statusCode: 403 });
|
|
vi.mocked(client.listLookmlModels).mockRejectedValue(error);
|
|
|
|
await fetchLookerRuntimeBundle({
|
|
pullConfig: { lookerConnectionId: connectionId },
|
|
stagedDir,
|
|
ctx: { connectionId, sourceKey: 'looker' },
|
|
clientFactory: { createClient: vi.fn().mockResolvedValue(client) },
|
|
now: () => new Date('2026-04-30T12:30:00.000Z'),
|
|
});
|
|
|
|
await expect(readdir(join(stagedDir, 'dashboards'))).resolves.toEqual(['10.json']);
|
|
await expect(readdir(join(stagedDir, 'looks'))).resolves.toEqual(['20.json']);
|
|
await expect(readFile(join(stagedDir, 'lookml_models.json'), 'utf-8')).resolves.toBe('{\n "models": []\n}\n');
|
|
await expect(readdir(join(stagedDir, 'explores'))).rejects.toMatchObject({ code: 'ENOENT' });
|
|
expect(client.getExplore).not.toHaveBeenCalled();
|
|
|
|
const report = JSON.parse(await readFile(join(stagedDir, 'looker-fetch-report.json'), 'utf-8'));
|
|
expect(report).toEqual({
|
|
status: 'success',
|
|
retryRecommended: false,
|
|
skipped: [],
|
|
warnings: [
|
|
{
|
|
rawPath: 'lookml_models.json',
|
|
entityType: 'lookml_models',
|
|
entityId: null,
|
|
severity: 'warning',
|
|
statusCode: 403,
|
|
message: 'LookML model access denied',
|
|
retryRecommended: false,
|
|
},
|
|
],
|
|
});
|
|
|
|
const chunked = await chunkLookerStagedDir(stagedDir);
|
|
expect(chunked.workUnits.map((wu) => wu.unitKey).sort()).toEqual(['looker-dashboard-10', 'looker-look-20']);
|
|
expect(chunked.workUnits.flatMap((wu) => wu.dependencyPaths)).not.toContain('explores/b2b/sales_pipeline.json');
|
|
});
|
|
|
|
it('cleans up the Looker client after a successful fetch', async () => {
|
|
const client = makeClient();
|
|
|
|
await fetchLookerRuntimeBundle({
|
|
pullConfig: { lookerConnectionId: connectionId },
|
|
stagedDir,
|
|
ctx: { connectionId, sourceKey: 'looker' },
|
|
clientFactory: { createClient: vi.fn().mockResolvedValue(client) },
|
|
now: () => new Date('2026-04-30T12:30:00.000Z'),
|
|
});
|
|
|
|
expect(client.cleanup).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it('cleans up the Looker client when fetch throws', async () => {
|
|
const client = makeClient();
|
|
vi.mocked(client.listDashboards).mockRejectedValue(new Error('Looker API unavailable'));
|
|
|
|
await expect(
|
|
fetchLookerRuntimeBundle({
|
|
pullConfig: { lookerConnectionId: connectionId },
|
|
stagedDir,
|
|
ctx: { connectionId, sourceKey: 'looker' },
|
|
clientFactory: { createClient: vi.fn().mockResolvedValue(client) },
|
|
now: () => new Date('2026-04-30T12:30:00.000Z'),
|
|
}),
|
|
).rejects.toThrow('Looker API unavailable');
|
|
|
|
expect(client.cleanup).toHaveBeenCalledTimes(1);
|
|
});
|
|
});
|