Update API docs for 2.4 (#960)

- Update API specs for 2.4 (#960)
- Update API docs
- Regenerate Python docs

specs/api/README.md

@@ -2,6 +2,55 @@
 
 This directory contains the modular OpenAPI 3.1 specification for the TrustGraph REST API Gateway.
 
+## Authentication
+
+Clients authenticate by passing an opaque bearer token in the
+`Authorization` header.  The gateway resolves the token to an
+authenticated identity and an associated workspace.  Tokens are
+obtained via the IAM service (e.g. `tg-login` or `tg-create-api-key`).
+
+## Service Tiers
+
+API services are organized into three tiers based on their scoping:
+
+### Global services
+
+These services are not scoped to a workspace.  They manage
+system-wide resources.
+
+- **IAM** — user management, authentication, API key lifecycle
+
+### Workspace-scoped services
+
+These services operate within the workspace associated with the
+authenticated token.  The workspace is resolved by the gateway from
+the bearer token — it is not passed as an explicit parameter.
+
+- **Config** — configuration management (prompts, token costs, etc.)
+- **Librarian** — document library management
+- **Knowledge** — knowledge graph core management
+- **Collection Management** — collection metadata
+- **Flow** — flow lifecycle and blueprint management
+
+### Flow-scoped services
+
+These services require a `flow` parameter identifying the processing
+flow to use, in addition to the workspace context from the token.
+
+- **Agent** — agentic AI interactions
+- **Document RAG** — retrieval-augmented generation over documents
+- **Graph RAG** — retrieval-augmented generation over knowledge graphs
+- **Text Completion** — LLM text completion
+- **Prompt** — prompt template expansion
+- **Embeddings** — vector embedding generation
+- **SPARQL Query** — SPARQL queries against the knowledge graph
+- **Graph Embeddings** — knowledge graph embedding queries
+- **Document Embeddings** — document embedding queries
+- **Structured Query** — structured data queries
+- **Row Embeddings** — structured data embedding queries
+- **Rows Query** — row-level data queries
+- **Triples Query** — knowledge graph triple queries
+
 ## Structure
 
 ```

specs/api/components/schemas/collection/CollectionRequest.yaml

@@ -14,7 +14,7 @@ properties:
       - delete-collection
     description: |
       Collection operation:
-      - `list-collections`: List collections in workspace
+      - `list-collections`: List collections in the current workspace (resolved from token)
       - `update-collection`: Create or update collection metadata
       - `delete-collection`: Delete collection
   collection:

specs/api/components/schemas/knowledge/KnowledgeRequest.yaml

@@ -18,7 +18,7 @@ properties:
       - unload-kg-core
     description: |
       Knowledge core operation:
-      - `list-kg-cores`: List knowledge cores in workspace
+      - `list-kg-cores`: List knowledge cores in the current workspace (resolved from token)
       - `get-kg-core`: Get knowledge core by ID
       - `put-kg-core`: Store triples and/or embeddings
       - `delete-kg-core`: Delete knowledge core by ID

specs/api/openapi.yaml

@@ -2,21 +2,44 @@ openapi: 3.1.0
 
 info:
   title: TrustGraph API Gateway
-  version: "2.2"
+  version: "2.4"
   description: |
     REST API for TrustGraph - an AI-powered knowledge graph and RAG system.
 
     ## Overview
 
     The API provides access to:
-    - **Global Services**: Configuration, flow management, knowledge storage, library management
-    - **Flow-Hosted Services**: AI services like RAG, text completion, embeddings (require running flow)
+    - **Global Services**: IAM (user management, authentication)
+    - **Workspace-Scoped Services**: Configuration, flow management, knowledge storage, library management
+    - **Flow-Scoped Services**: AI services like RAG, text completion, embeddings (require running flow)
     - **Import/Export**: Bulk data operations for triples, embeddings, entity contexts
     - **WebSocket**: Multiplexed interface for all services
 
-    ## Service Types
+    ## Authentication
+
+    Clients authenticate by passing an opaque bearer token in the
+    `Authorization` header.  The token is obtained via the IAM service
+    (e.g. `tg-login` or `tg-create-api-key`).
+
+    ```
+    Authorization: Bearer <token>
+    ```
+
+    The gateway resolves the token to an authenticated identity and an
+    associated workspace.  The token is an opaque string — clients must
+    not make assumptions about its internal structure.
+
+    ## Service Tiers
 
     ### Global Services
+    System-wide services with no workspace scoping:
+    - `iam` - User management, authentication, API key lifecycle
+
+    ### Workspace-Scoped Services
+    Operate within the workspace associated with the authenticated
+    token.  The workspace is resolved by the gateway — it is not
+    passed as an explicit parameter.
+
     Fixed endpoints accessible via `/api/v1/{kind}`:
     - `config` - Configuration management
     - `flow` - Flow lifecycle and blueprints
@@ -24,24 +47,17 @@ info:
     - `knowledge` - Knowledge graph core management
     - `collection-management` - Collection metadata
 
-    ### Flow-Hosted Services
-    Require running flow instance, accessed via `/api/v1/flow/{flow}/service/{kind}`:
+    ### Flow-Scoped Services
+    Require a `flow` parameter identifying the processing flow to use.
+    Workspace context comes from the authenticated token.
+
+    Accessed via `/api/v1/flow/{flow}/service/{kind}`:
     - AI services: agent, text-completion, prompt, RAG (document/graph)
     - Embeddings: embeddings, graph-embeddings, document-embeddings
     - Query: triples, rows, nlp-query, structured-query, sparql-query, row-embeddings
     - Data loading: text-load, document-load
     - Utilities: mcp-tool, structured-diag
 
-    ## Authentication
-
-    Bearer token authentication when `GATEWAY_SECRET` environment variable is set.
-    Include token in Authorization header:
-    ```
-    Authorization: Bearer <token>
-    ```
-
-    If `GATEWAY_SECRET` is not set, API runs without authentication (development mode).
-
     ## Field Naming
 
     All JSON fields use **kebab-case**: `flow-id`, `blueprint-name`, `doc-limit`, etc.
@@ -74,17 +90,17 @@ security:
 
 tags:
   - name: Config
-    description: Configuration management (global service)
+    description: Configuration management (workspace-scoped)
   - name: Flow
-    description: Flow lifecycle and blueprint management (global service)
+    description: Flow lifecycle and blueprint management (workspace-scoped)
   - name: Librarian
-    description: Document library management (global service)
+    description: Document library management (workspace-scoped)
   - name: Knowledge
-    description: Knowledge graph core management (global service)
+    description: Knowledge graph core management (workspace-scoped)
   - name: Collection
-    description: Collection metadata management (global service)
+    description: Collection metadata management (workspace-scoped)
   - name: Flow Services
-    description: Services hosted within flow instances
+    description: AI and query services hosted within flow instances (flow-scoped)
   - name: Import/Export
     description: Bulk data import and export
   - name: WebSocket

specs/api/paths/collection-management.yaml

@@ -1,10 +1,13 @@
 post:
   tags:
     - Collection
-  summary: Collection metadata management
+  summary: Collection metadata management (workspace-scoped)
   description: |
     Manage collection metadata for organizing documents and knowledge.
 
+    This is a **workspace-scoped** service. All operations apply to the
+    workspace associated with the authenticated bearer token.
+
     ## Collections
 
     Collections are organizational units for grouping:

specs/api/paths/config.yaml

@@ -1,9 +1,13 @@
 post:
   tags:
     - Config
-  summary: Configuration service
+  summary: Configuration service (workspace-scoped)
   description: |
-    Manage TrustGraph configuration including flows, prompts, token costs, parameter types, and more.
+    Manage TrustGraph configuration including flows, prompts, token costs,
+    parameter types, and more.
+
+    This is a **workspace-scoped** service. All operations apply to the
+    workspace associated with the authenticated bearer token.
 
     ## Operations
 

specs/api/paths/flow.yaml

@@ -1,10 +1,13 @@
 post:
   tags:
     - Flow
-  summary: Flow lifecycle and blueprint management
+  summary: Flow lifecycle and blueprint management (workspace-scoped)
   description: |
     Manage flow instances and blueprints.
 
+    This is a **workspace-scoped** service. All operations apply to the
+    workspace associated with the authenticated bearer token.
+
     ## Important Distinction
 
     The **flow service** manages *running flow instances*.

specs/api/paths/flow/agent.yaml

@@ -5,6 +5,10 @@ post:
   description: |
     AI agent that can understand questions, reason about them, and take actions.
 
+    This is a **flow-scoped** service. It requires a flow instance
+    and operates within the workspace associated with the
+    authenticated bearer token.
+
     ## Agent Overview
 
     The agent service provides a conversational AI that:

specs/api/paths/flow/document-embeddings.yaml

@@ -5,6 +5,10 @@ post:
   description: |
     Query document embeddings to find similar text chunks by vector similarity.
 
+    This is a **flow-scoped** service. It requires a flow instance
+    and operates within the workspace associated with the
+    authenticated bearer token.
+
     ## Document Embeddings Query Overview
 
     Find document chunks semantically similar to a query vector:

specs/api/paths/flow/document-load.yaml

@@ -5,6 +5,10 @@ post:
   description: |
     Load binary documents (PDF, Word, etc.) into processing pipeline.
 
+    This is a **flow-scoped** service. It requires a flow instance
+    and operates within the workspace associated with the
+    authenticated bearer token.
+
     ## Document Load Overview
 
     Fire-and-forget binary document loading:

specs/api/paths/flow/document-rag.yaml

@@ -5,6 +5,10 @@ post:
   description: |
     Retrieval-Augmented Generation over document embeddings.
 
+    This is a **flow-scoped** service. It requires a flow instance
+    and operates within the workspace associated with the
+    authenticated bearer token.
+
     ## Document RAG Overview
 
     Document RAG combines:

specs/api/paths/flow/embeddings.yaml

@@ -5,6 +5,10 @@ post:
   description: |
     Convert text to embedding vectors for semantic similarity search.
 
+    This is a **flow-scoped** service. It requires a flow instance
+    and operates within the workspace associated with the
+    authenticated bearer token.
+
     ## Embeddings Overview
 
     Embeddings transform text into dense vector representations that:

specs/api/paths/flow/graph-embeddings.yaml

@@ -5,6 +5,10 @@ post:
   description: |
     Query graph embeddings to find similar entities by vector similarity.
 
+    This is a **flow-scoped** service. It requires a flow instance
+    and operates within the workspace associated with the
+    authenticated bearer token.
+
     ## Graph Embeddings Query Overview
 
     Find entities semantically similar to a query vector:

specs/api/paths/flow/graph-rag.yaml

@@ -5,6 +5,10 @@ post:
   description: |
     Retrieval-Augmented Generation over knowledge graph.
 
+    This is a **flow-scoped** service. It requires a flow instance
+    and operates within the workspace associated with the
+    authenticated bearer token.
+
     ## Graph RAG Overview
 
     Graph RAG combines:

specs/api/paths/flow/mcp-tool.yaml

@@ -5,6 +5,10 @@ post:
   description: |
     Execute MCP (Model Context Protocol) tools for agent capabilities.
 
+    This is a **flow-scoped** service. It requires a flow instance
+    and operates within the workspace associated with the
+    authenticated bearer token.
+
     ## MCP Tool Overview
 
     MCP tools provide agent capabilities through standardized protocol:

specs/api/paths/flow/nlp-query.yaml

@@ -5,6 +5,10 @@ post:
   description: |
     Convert natural language questions to structured GraphQL queries.
 
+    This is a **flow-scoped** service. It requires a flow instance
+    and operates within the workspace associated with the
+    authenticated bearer token.
+
     ## NLP Query Overview
 
     Transforms user questions into executable GraphQL:

specs/api/paths/flow/prompt.yaml

@@ -5,6 +5,10 @@ post:
   description: |
     Execute stored prompt templates with variable substitution.
 
+    This is a **flow-scoped** service. It requires a flow instance
+    and operates within the workspace associated with the
+    authenticated bearer token.
+
     ## Prompt Service Overview
 
     The prompt service enables:

specs/api/paths/flow/row-embeddings.yaml

@@ -4,6 +4,11 @@ post:
   summary: Row Embeddings Query - semantic search on structured data
   description: |
     Query row embeddings to find similar rows by vector similarity on indexed fields.
+
+    This is a **flow-scoped** service. It requires a flow instance
+    and operates within the workspace associated with the
+    authenticated bearer token.
+
     Enables fuzzy/semantic matching on structured data.
 
     ## Row Embeddings Query Overview

specs/api/paths/flow/rows.yaml

@@ -5,6 +5,10 @@ post:
   description: |
     Query structured data using GraphQL for row-oriented data access.
 
+    This is a **flow-scoped** service. It requires a flow instance
+    and operates within the workspace associated with the
+    authenticated bearer token.
+
     ## Rows Query Overview
 
     GraphQL interface to structured data:

specs/api/paths/flow/sparql-query.yaml

@@ -5,6 +5,10 @@ post:
   description: |
     Execute a SPARQL 1.1 query against the knowledge graph.
 
+    This is a **flow-scoped** service. It requires a flow instance
+    and operates within the workspace associated with the
+    authenticated bearer token.
+
     ## Supported Query Types
 
     - **SELECT**: Returns variable bindings as a table of results

specs/api/paths/flow/structured-diag.yaml

@@ -5,6 +5,10 @@ post:
   description: |
     Analyze and understand structured data (CSV, JSON, XML).
 
+    This is a **flow-scoped** service. It requires a flow instance
+    and operates within the workspace associated with the
+    authenticated bearer token.
+
     ## Structured Diag Overview
 
     Helps process unknown structured data:

specs/api/paths/flow/structured-query.yaml

@@ -5,6 +5,10 @@ post:
   description: |
     Ask natural language questions and get results directly.
 
+    This is a **flow-scoped** service. It requires a flow instance
+    and operates within the workspace associated with the
+    authenticated bearer token.
+
     ## Structured Query Overview
 
     Combines two operations in one call:

specs/api/paths/flow/text-completion.yaml

@@ -5,6 +5,10 @@ post:
   description: |
     Direct text completion using LLM without retrieval augmentation.
 
+    This is a **flow-scoped** service. It requires a flow instance
+    and operates within the workspace associated with the
+    authenticated bearer token.
+
     ## Text Completion Overview
 
     Pure LLM generation for:

specs/api/paths/flow/text-load.yaml

@@ -5,6 +5,10 @@ post:
   description: |
     Load text documents into processing pipeline for indexing and embedding.
 
+    This is a **flow-scoped** service. It requires a flow instance
+    and operates within the workspace associated with the
+    authenticated bearer token.
+
     ## Text Load Overview
 
     Fire-and-forget document loading:

specs/api/paths/flow/triples.yaml

@@ -5,6 +5,10 @@ post:
   description: |
     Query knowledge graph using subject-predicate-object patterns.
 
+    This is a **flow-scoped** service. It requires a flow instance
+    and operates within the workspace associated with the
+    authenticated bearer token.
+
     ## Triples Query Overview
 
     Query RDF triples with flexible pattern matching:

specs/api/paths/knowledge.yaml

@@ -1,9 +1,13 @@
 post:
   tags:
     - Knowledge
-  summary: Knowledge graph core management
+  summary: Knowledge graph core management (workspace-scoped)
   description: |
-    Manage knowledge graph cores - persistent storage of triples and embeddings.
+    Manage knowledge graph cores - persistent storage of triples and
+    embeddings.
+
+    This is a **workspace-scoped** service. All operations apply to the
+    workspace associated with the authenticated bearer token.
 
     ## Knowledge Cores
 

specs/api/paths/librarian.yaml

@@ -1,9 +1,13 @@
 post:
   tags:
     - Librarian
-  summary: Document library management
+  summary: Document library management (workspace-scoped)
   description: |
-    Manage document library: add, remove, list documents, and control processing.
+    Manage document library: add, remove, list documents, and control
+    processing.
+
+    This is a **workspace-scoped** service. All operations apply to the
+    workspace associated with the authenticated bearer token.
 
     ## Document Library
 

specs/api/paths/websocket.yaml

@@ -26,7 +26,7 @@ get:
 
     ### Request Message Format
 
-    **Global Service Request** (no flow parameter):
+    **Workspace-Scoped Service Request** (no flow parameter):
     ```json
     {
       "id": "req-123",
@@ -38,7 +38,7 @@ get:
     }
     ```
 
-    **Flow-Hosted Service Request** (with flow parameter):
+    **Flow-Scoped Service Request** (with flow parameter):
     ```json
     {
       "id": "req-456",
@@ -54,7 +54,7 @@ get:
     **Request Fields**:
     - `id` (string, required): Client-generated unique identifier for this request within the session. Used to match responses to requests.
     - `service` (string, required): Service identifier (e.g., "config", "agent", "document-rag"). Same as `{kind}` in REST URLs.
-    - `flow` (string, optional): Flow ID for flow-hosted services. Omit for global services.
+    - `flow` (string, optional): Flow ID for flow-scoped services. Omit for workspace-scoped and global services.
     - `request` (object, required): Service-specific request payload. Same structure as REST API request body.
 
     ### Response Message Format
@@ -96,14 +96,14 @@ get:
     | `POST /api/v1/config` | `{"service": "config"}` |
     | `POST /api/v1/flow/{flow}/service/agent` | `{"service": "agent", "flow": "my-flow"}` |
 
-    **Global Services** (no `flow` parameter):
+    **Workspace-Scoped Services** (no `flow` parameter, workspace from token):
     - `config` - Configuration management
     - `flow` - Flow lifecycle and blueprints
     - `librarian` - Document library management
     - `knowledge` - Knowledge graph core management
     - `collection-management` - Collection metadata
 
-    **Flow-Hosted Services** (require `flow` parameter):
+    **Flow-Scoped Services** (require `flow` parameter, workspace from token):
     - AI services: `agent`, `text-completion`, `prompt`, `document-rag`, `graph-rag`
     - Embeddings: `embeddings`, `graph-embeddings`, `document-embeddings`
     - Query: `triples`, `objects`, `nlp-query`, `structured-query`
@@ -146,9 +146,11 @@ get:
 
     ## Authentication
 
-    When `GATEWAY_SECRET` is set, include bearer token:
-    - As query parameter: `ws://localhost:8088/api/v1/socket?token=<token>`
-    - Or in WebSocket subprotocol header
+    The `/api/v1/socket` endpoint uses in-band authentication.
+    The WebSocket handshake is accepted unconditionally.  After
+    connecting, the client sends a bearer token as the first frame.
+    The gateway resolves the token to an identity and workspace.
+    All subsequent requests operate within that workspace context.
 
     ## Benefits Over REST
 

specs/api/security/bearerAuth.yaml

@@ -3,10 +3,19 @@ scheme: bearer
 description: |
   Bearer token authentication.
 
-  Set via `GATEWAY_SECRET` environment variable on the gateway.
-  If `GATEWAY_SECRET` is not set, authentication is disabled (development mode).
+  Clients authenticate by passing an opaque token in the
+  `Authorization` header. The token is treated as an opaque string by
+  clients — its internal structure is a gateway implementation detail
+  and must not be relied upon.
+
+  The gateway resolves the token to an authenticated identity and an
+  associated workspace. All workspace-scoped and flow-scoped operations
+  then execute within that workspace context.
+
+  Tokens are obtained via the IAM service (e.g. `tg-login` or
+  `tg-create-api-key`).
 
   Example:
   ```
-  Authorization: Bearer your-secret-token
+  Authorization: Bearer <token>
   ```