Merge branch 'main' into copy-claude-code-backend-spec

docs/superpowers/plans/2026-05-15-semantic-layer-docs.md

@@ -0,0 +1,328 @@
+# Semantic Layer Docs Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [x]`) syntax for tracking.
+
+**Goal:** Add a standalone, scannable Concepts page that explains the semantic-layer internals while positioning KTX as a broader context layer.
+
+**Architecture:** Implement this as docs-only MDX content inside the existing Fumadocs tree. The new page uses inline MDX diagrams and Fumadocs color tokens, matching the custom diagram pattern already used in `the-context-layer.mdx`.
+
+**Tech Stack:** MDX, Fumadocs content, Next.js docs site, pnpm workspace commands.
+
+---
+
+### Task 1: Add Concepts Navigation Entry
+
+**Files:**
+- Modify: `docs-site/content/docs/concepts/meta.json`
+
+- [x] **Step 1: Update the Concepts page order**
+
+Replace the `pages` array with:
+
+```json
+{
+  "title": "Concepts",
+  "defaultOpen": true,
+  "pages": ["the-context-layer", "semantic-layer-internals", "context-as-code"]
+}
+```
+
+- [x] **Step 2: Verify JSON parses**
+
+Run:
+
+```bash
+node -e "JSON.parse(require('node:fs').readFileSync('docs-site/content/docs/concepts/meta.json', 'utf8')); console.log('concepts meta ok')"
+```
+
+Expected output:
+
+```text
+concepts meta ok
+```
+
+### Task 2: Create the Semantic Layer Internals Page
+
+**Files:**
+- Create: `docs-site/content/docs/concepts/semantic-layer-internals.mdx`
+
+- [x] **Step 1: Add frontmatter and opening positioning**
+
+Create the page with this frontmatter and opening section:
+
+```mdx
+---
+title: Semantic Layer Internals
+description: How KTX uses join graphs, grain, and relationship metadata to turn context into safe SQL.
+---
+
+KTX is a context layer for agents. Its semantic layer is the query-planning core
+that turns reviewed context into safe SQL.
+
+Use this page to understand the mechanics behind KTX's semantic execution:
+the join graph, how KTX builds and maintains it, and how that graph prevents
+classic analytics errors like fan-out and ambiguous join paths.
+
+| KTX is | KTX is not just |
+|---|---|
+| A context layer for agents | A metric definition store |
+| A system for ingesting, reviewing, and serving analytics context | A markdown saver |
+| A semantic execution layer plus wiki pages, scans, provenance, and agent workflows | A replacement for every BI semantic layer |
+```
+
+- [x] **Step 2: Add the system-fit diagram**
+
+Add a `Where the semantic layer fits` section with a custom `not-prose` diagram.
+The diagram must show:
+
+```text
+Context inputs -> Semantic layer engine -> Agent workflows
+```
+
+The semantic-layer box must be visually prominent and list:
+
+```text
+join graph
+grain
+measures
+relationships
+safe query planning
+```
+
+- [x] **Step 3: Add the join graph section**
+
+Add `## The join graph` with:
+
+- one short paragraph defining nodes and edges;
+- bullets for why the graph matters;
+- an inline diagram using `orders`, `customers`, `order_items`, and `refunds`.
+
+The section must include this claim in plain language:
+
+```text
+The graph lets KTX choose valid paths, reject unsafe paths, and reason about
+whether a join preserves or multiplies rows before SQL is generated.
+```
+
+- [x] **Step 4: Add build and maintenance sections**
+
+Add `## How KTX builds it` and `## How KTX maintains it`.
+
+`How KTX builds it` must cover these inputs:
+
+```text
+declared primary keys
+declared foreign keys
+inferred relationships
+dbt, MetricFlow, and LookML imports
+query history
+analyst review
+```
+
+`How KTX maintains it` must show this loop:
+
+```text
+ingest evidence -> YAML diff -> validation -> analyst review -> agent use -> corrections
+```
+
+- [x] **Step 5: Add the fan-out and safe execution sections**
+
+Add `## Why grain and relationships matter` with a fan-out example comparing
+orders joined to order items. Include a compact table with columns:
+
+```text
+Problem
+What happens
+How KTX avoids it
+```
+
+Add `## How the execution engine uses the graph` with a before/after table:
+
+```text
+Naive SQL shape
+Semantic-layer SQL shape
+```
+
+The safe path must mention:
+
+```text
+pre-aggregates fact measures at their own grain before joining dimensions
+```
+
+- [x] **Step 6: Add agent outcome links**
+
+Add a closing `## What this means for agents` section with bullets explaining
+that agents can:
+
+```text
+search semantic sources
+compile SQL through ktx sl query
+validate changes before review
+patch YAML and Markdown files in git
+explain provenance and metric meaning
+```
+
+End with links to:
+
+```mdx
+[Writing Context](/docs/guides/writing-context)
+[ktx sl](/docs/cli-reference/ktx-sl)
+```
+
+### Task 3: Add the Cross-Link from The Context Layer
+
+**Files:**
+- Modify: `docs-site/content/docs/concepts/the-context-layer.mdx`
+
+- [x] **Step 1: Replace the semantic sources paragraph with a scannable block**
+
+Find the `**Semantic sources**` paragraph under `KTX organizes context into four pillars`.
+Replace the long paragraph with:
+
+```mdx
+**Semantic sources** are YAML definitions that describe your data in terms
+agents can reason about:
+
+- source tables or SQL queries;
+- row grain;
+- typed columns;
+- valid joins;
+- named measures, filters, and segments.
+
+This is where "revenue means `sum(amount)` excluding refunds" lives. For the
+join graph, fan-out protections, and execution mechanics, read
+[Semantic Layer Internals](/docs/concepts/semantic-layer-internals).
+```
+
+- [x] **Step 2: Confirm the page still owns the product positioning**
+
+Search the edited file:
+
+```bash
+rg -n "context layer|Semantic Layer Internals|semantic layer - that's a critical component" docs-site/content/docs/concepts/the-context-layer.mdx
+```
+
+Expected: output includes the existing context-layer framing and the new internals link.
+
+### Task 4: Fix Mobile Docs Header Overflow
+
+**Files:**
+- Modify: `docs-site/app/docs/[[...slug]]/page.tsx`
+
+- [x] **Step 1: Stack title actions on narrow screens**
+
+Replace the non-hero page header wrapper:
+
+```tsx
+<div className="flex items-start justify-between gap-4">
+```
+
+with:
+
+```tsx
+<div className="flex flex-col gap-3 sm:flex-row sm:items-start sm:justify-between sm:gap-4">
+```
+
+This keeps desktop layout unchanged while preventing the action buttons from
+forcing horizontal overflow on mobile.
+
+- [x] **Step 2: Allow the docs article to shrink in the layout grid**
+
+Update the `DocsPage` and `DocsBody` wrappers:
+
+```tsx
+<DocsPage
+  toc={page.data.toc}
+  className="!mx-0 min-w-0 !max-w-[calc(100vw-2rem)] md:!mx-auto md:!max-w-[900px]"
+>
+```
+
+```tsx
+<DocsBody className="min-w-0 max-w-full">
+```
+
+This prevents tables, code blocks, and custom diagrams from forcing the
+Fumadocs main article column wider than the mobile viewport, overrides the
+library's built-in max-width rule on mobile, aligns the article to the left on
+mobile, and preserves the normal centered desktop max width.
+
+If long words still clip under mobile viewport capture, add the same wrapping
+behavior used by the Fumadocs sidebar:
+
+```tsx
+<DocsDescription className="wrap-anywhere">
+  {page.data.description}
+</DocsDescription>
+```
+
+```tsx
+<DocsBody className="min-w-0 max-w-full wrap-anywhere">
+```
+
+- [x] **Step 3: Recheck mobile render**
+
+Capture or inspect a 390px-wide render of:
+
+```text
+http://127.0.0.1:3000/docs/concepts/semantic-layer-internals
+```
+
+Expected: the title, description, action buttons, and positioning block stay
+within the viewport.
+
+### Task 5: Verify Docs Content and Build
+
+**Files:**
+- Check: `docs-site/content/docs/concepts/semantic-layer-internals.mdx`
+- Check: `docs-site/content/docs/concepts/the-context-layer.mdx`
+- Check: `docs-site/content/docs/concepts/meta.json`
+- Check: `docs-site/app/docs/[[...slug]]/page.tsx`
+
+- [x] **Step 1: Run content checks**
+
+Run:
+
+```bash
+rg -n "KTX is a context layer|markdown saver|fan-out|join graph|pre-aggregates|Semantic Layer Internals" docs-site/content/docs/concepts
+```
+
+Expected: matches appear in the new page and the cross-link appears in
+`the-context-layer.mdx`.
+
+- [x] **Step 2: Build the docs site**
+
+Run:
+
+```bash
+pnpm --filter ktx-docs build
+```
+
+Expected: build exits 0.
+
+- [x] **Step 3: Preview locally**
+
+Run:
+
+```bash
+pnpm --filter ktx-docs dev
+```
+
+Open:
+
+```text
+http://localhost:3000/docs/concepts/semantic-layer-internals
+```
+
+Inspect desktop and mobile widths. The opening should clearly position KTX as a
+context layer, the Concepts navigation should list the new page, and diagrams
+should not overlap or produce unreadable text.
+
+- [x] **Step 4: Commit implementation**
+
+Run:
+
+```bash
+git status --short
+git add docs-site/content/docs/concepts/meta.json docs-site/content/docs/concepts/semantic-layer-internals.mdx docs-site/content/docs/concepts/the-context-layer.mdx docs-site/app/docs/[[...slug]]/page.tsx docs/superpowers/plans/2026-05-15-semantic-layer-docs.md
+git commit -m "docs: add semantic layer internals concept"
+```

docs/superpowers/specs/2026-05-15-semantic-layer-docs-design.md

@@ -0,0 +1,166 @@
+# Semantic Layer Docs Design
+
+**Date:** 2026-05-15
+**Status:** Design - pending implementation plan
+
+## Goal
+
+Add a concise Concepts page that explains the semantic layer as the query
+planning engine inside KTX's broader context layer.
+
+The page should make the technical depth visible to skeptical data users
+without positioning KTX as only a semantic-layer product. Success means a reader
+understands:
+
+- KTX is a context layer for agents.
+- The semantic layer is one core subsystem inside that context layer.
+- The join graph, grain declarations, and relationship metadata are what make
+  generated SQL safer than schema-only or markdown-only approaches.
+- KTX maintains this semantic layer through ingest, validation, analyst edits,
+  and reviewable files.
+
+## Current State
+
+The docs currently explain semantic sources in two places:
+
+- `docs-site/content/docs/concepts/the-context-layer.mdx` describes semantic
+  sources as one pillar of KTX context.
+- `docs-site/content/docs/guides/writing-context.mdx` documents the YAML fields
+  for sources, measures, joins, grain, validation, and common errors.
+
+That content is useful, but the differentiator is not visually obvious. The
+semantic layer is embedded in longer narrative pages, so readers can miss the
+hard parts: join graph construction, fan-out prevention, chasm traps, and query
+planning.
+
+## Positioning
+
+Create a standalone Concepts page with a guarded title such as
+`Semantic Layer Internals` or `The Semantic Engine Inside KTX`.
+
+The first screen must frame the product clearly:
+
+> KTX is a context layer. Its semantic layer is the query-planning core that
+> turns reviewed context into safe SQL.
+
+The page should avoid a title like `Semantic Layer` by itself because that can
+make KTX look like a narrow semantic-layer tool. The page should repeatedly show
+the semantic layer between the broader context inputs and the agent workflows it
+supports.
+
+Add a short cross-link from `the-context-layer.mdx` so the existing overview
+keeps owning the product category. That section should say the semantic layer is
+one critical pillar, then link to the internals page for readers who want the
+mechanics.
+
+## Page Structure
+
+Add `docs-site/content/docs/concepts/semantic-layer-internals.mdx` and include
+it in `docs-site/content/docs/concepts/meta.json` after `the-context-layer`.
+
+Recommended sections:
+
+1. `What this page explains`
+   - One short paragraph.
+   - A two-column `KTX is / KTX is not just` table.
+
+2. `Where the semantic layer fits`
+   - A visual block showing:
+     `context inputs -> semantic layer engine -> agent workflows`.
+   - Inputs include semantic YAML, wiki pages, scans, and provenance.
+   - Outputs include search, SQL generation, explanations, edits, and review.
+
+3. `The join graph`
+   - Explain nodes as semantic sources and edges as validated joins.
+   - Show a small graph with `orders`, `customers`, `order_items`, and
+     `refunds`.
+   - Keep text to one or two short paragraphs plus bullets.
+
+4. `How KTX builds it`
+   - Show a pipeline from database evidence and imported modeling tools to
+     reviewable YAML.
+   - Mention declared keys, inferred relationships, dbt/MetricFlow/LookML
+     imports, query history, validation, and analyst review.
+
+5. `How KTX maintains it`
+   - Show a feedback loop:
+     ingest evidence -> YAML diff -> validation -> analyst review -> agent use
+     -> corrections.
+   - Emphasize that files remain the source of truth.
+
+6. `Why grain and relationships matter`
+   - Use the fan-out problem as the central example.
+   - Compare a naive join against a safe semantic-layer plan.
+   - Explain many-to-one, one-to-many, many-to-many, chasm traps, and ambiguous
+     paths in compact bullets.
+
+7. `How the execution engine uses the graph`
+   - Explain path selection, unsafe path rejection, pre-aggregation into CTEs,
+     filter placement, and dialect transpilation.
+   - Include a small before/after SQL-shape diagram or table.
+
+8. `What this means for agents`
+   - Summarize why this is more than saving markdown:
+     agents can inspect, query, validate, edit, and review the same semantic
+     files.
+   - Link to `Writing Context` and `ktx sl`.
+
+## Scannability Rules
+
+The implementation should shorten long prose blocks across the touched pages.
+
+- Keep most text blocks to one or two paragraphs.
+- Prefer bullets, tables, diagrams, and compact callout blocks between prose.
+- Avoid four-paragraph narrative runs.
+- Use diagrams before dense explanations when the concept is spatial.
+- Keep examples concrete and copy-pasteable.
+
+## Visual Direction
+
+Use the existing docs-site MDX style rather than a new design system. The current
+`the-context-layer.mdx` page already uses custom `not-prose` MDX diagrams with
+Fumadocs color tokens; the new page should follow that pattern.
+
+The diagrams should feel like technical product documentation:
+
+- restrained, dense, and readable;
+- high contrast for the semantic-layer engine box;
+- visible arrows or adjacency that make flow obvious;
+- tables for classification and comparison;
+- no marketing hero, decorative gradients, or generic card-heavy layout.
+
+## Non-goals
+
+- Do not redesign the whole docs site.
+- Do not rename KTX concepts, packages, commands, or directories.
+- Do not claim KTX replaces every BI or semantic-layer system.
+- Do not add implementation details that are not true in the current codebase.
+- Do not expand the page into a long reference for every YAML field; keep that
+  in `Writing Context`.
+
+## Verification
+
+Because this is docs-only work, verification should focus on the docs site:
+
+- Run the docs build or the narrowest available docs-site type/build check.
+- Run formatting or lint checks if the docs package exposes them.
+- Preview the page locally and inspect desktop and mobile widths.
+- Confirm the page is listed in Concepts navigation.
+- Confirm the opening section clearly says KTX is a context layer, not just a
+  semantic-layer tool.
+
+If implementation changes only MDX and metadata, TypeScript workspace tests are
+not required unless the page introduces shared components.
+
+## Acceptance Criteria
+
+- A standalone Concepts page explains the semantic-layer internals.
+- The Context Layer page links to the new internals page without making the
+  overview longer.
+- The new page includes diagrams for the system fit, join graph, maintenance
+  loop, and fan-out-safe execution path.
+- Long prose is broken into scannable sections with bullets, tables, and visual
+  interruptions.
+- The positioning consistently says KTX is a context layer with a semantic
+  execution core.
+- Docs-site verification passes or any skipped check is reported with a reason.