ktx/docs/superpowers/specs/2026-05-15-semantic-layer-docs-design.md

# Semantic Layer Docs Design

**Date:** 2026-05-15
**Status:** Design - pending implementation plan

## Goal

Add a concise Concepts page that explains the semantic layer as the query
planning engine inside KTX's broader context layer.

The page should make the technical depth visible to skeptical data users
without positioning KTX as only a semantic-layer product. Success means a reader
understands:

- KTX is a context layer for agents.
- The semantic layer is one core subsystem inside that context layer.
- The join graph, grain declarations, and relationship metadata are what make
  generated SQL safer than schema-only or markdown-only approaches.
- KTX maintains this semantic layer through ingest, validation, analyst edits,
  and reviewable files.

## Current State

The docs currently explain semantic sources in two places:

- `docs-site/content/docs/concepts/the-context-layer.mdx` describes semantic
  sources as one pillar of KTX context.
- `docs-site/content/docs/guides/writing-context.mdx` documents the YAML fields
  for sources, measures, joins, grain, validation, and common errors.

That content is useful, but the differentiator is not visually obvious. The
semantic layer is embedded in longer narrative pages, so readers can miss the
hard parts: join graph construction, fan-out prevention, chasm traps, and query
planning.

## Positioning

Create a standalone Concepts page with a guarded title such as
`Semantic Layer Internals` or `The Semantic Engine Inside KTX`.

The first screen must frame the product clearly:

> KTX is a context layer. Its semantic layer is the query-planning core that
> turns reviewed context into safe SQL.

The page should avoid a title like `Semantic Layer` by itself because that can
make KTX look like a narrow semantic-layer tool. The page should repeatedly show
the semantic layer between the broader context inputs and the agent workflows it
supports.

Add a short cross-link from `the-context-layer.mdx` so the existing overview
keeps owning the product category. That section should say the semantic layer is
one critical pillar, then link to the internals page for readers who want the
mechanics.

## Page Structure

Add `docs-site/content/docs/concepts/semantic-layer-internals.mdx` and include
it in `docs-site/content/docs/concepts/meta.json` after `the-context-layer`.

Recommended sections:

1. `What this page explains`
   - One short paragraph.
   - A two-column `KTX is / KTX is not just` table.

2. `Where the semantic layer fits`
   - A visual block showing:
     `context inputs -> semantic layer engine -> agent workflows`.
   - Inputs include semantic YAML, wiki pages, scans, and provenance.
   - Outputs include search, SQL generation, explanations, edits, and review.

3. `The join graph`
   - Explain nodes as semantic sources and edges as validated joins.
   - Show a small graph with `orders`, `customers`, `order_items`, and
     `refunds`.
   - Keep text to one or two short paragraphs plus bullets.

4. `How KTX builds it`
   - Show a pipeline from database evidence and imported modeling tools to
     reviewable YAML.
   - Mention declared keys, inferred relationships, dbt/MetricFlow/LookML
     imports, query history, validation, and analyst review.

5. `How KTX maintains it`
   - Show a feedback loop:
     ingest evidence -> YAML diff -> validation -> analyst review -> agent use
     -> corrections.
   - Emphasize that files remain the source of truth.

6. `Why grain and relationships matter`
   - Use the fan-out problem as the central example.
   - Compare a naive join against a safe semantic-layer plan.
   - Explain many-to-one, one-to-many, many-to-many, chasm traps, and ambiguous
     paths in compact bullets.

7. `How the execution engine uses the graph`
   - Explain path selection, unsafe path rejection, pre-aggregation into CTEs,
     filter placement, and dialect transpilation.
   - Include a small before/after SQL-shape diagram or table.

8. `What this means for agents`
   - Summarize why this is more than saving markdown:
     agents can inspect, query, validate, edit, and review the same semantic
     files.
   - Link to `Writing Context` and `ktx sl`.

## Scannability Rules

The implementation should shorten long prose blocks across the touched pages.

- Keep most text blocks to one or two paragraphs.
- Prefer bullets, tables, diagrams, and compact callout blocks between prose.
- Avoid four-paragraph narrative runs.
- Use diagrams before dense explanations when the concept is spatial.
- Keep examples concrete and copy-pasteable.

## Visual Direction

Use the existing docs-site MDX style rather than a new design system. The current
`the-context-layer.mdx` page already uses custom `not-prose` MDX diagrams with
Fumadocs color tokens; the new page should follow that pattern.

The diagrams should feel like technical product documentation:

- restrained, dense, and readable;
- high contrast for the semantic-layer engine box;
- visible arrows or adjacency that make flow obvious;
- tables for classification and comparison;
- no marketing hero, decorative gradients, or generic card-heavy layout.

## Non-goals

- Do not redesign the whole docs site.
- Do not rename KTX concepts, packages, commands, or directories.
- Do not claim KTX replaces every BI or semantic-layer system.
- Do not add implementation details that are not true in the current codebase.
- Do not expand the page into a long reference for every YAML field; keep that
  in `Writing Context`.

## Verification

Because this is docs-only work, verification should focus on the docs site:

- Run the docs build or the narrowest available docs-site type/build check.
- Run formatting or lint checks if the docs package exposes them.
- Preview the page locally and inspect desktop and mobile widths.
- Confirm the page is listed in Concepts navigation.
- Confirm the opening section clearly says KTX is a context layer, not just a
  semantic-layer tool.

If implementation changes only MDX and metadata, TypeScript workspace tests are
not required unless the page introduces shared components.

## Acceptance Criteria

- A standalone Concepts page explains the semantic-layer internals.
- The Context Layer page links to the new internals page without making the
  overview longer.
- The new page includes diagrams for the system fit, join graph, maintenance
  loop, and fan-out-safe execution path.
- Long prose is broken into scannable sections with bullets, tables, and visual
  interruptions.
- The positioning consistently says KTX is a context layer with a semantic
  execution core.
- Docs-site verification passes or any skipped check is reported with a reason.