From ab002e6edbe805b8b4de8b1b336ee06618ec3c4f Mon Sep 17 00:00:00 2001 From: Cyber MacGeddon Date: Tue, 17 Mar 2026 13:15:33 +0000 Subject: [PATCH] Updating API specs for 2.1 --- docs/api-gateway-changes-v1.8-to-v2.1.md | 108 ++++++++++++++++++++++ docs/cli-changes-v1.8-to-v2.1.md | 112 +++++++++++++++++++++++ 2 files changed, 220 insertions(+) create mode 100644 docs/api-gateway-changes-v1.8-to-v2.1.md create mode 100644 docs/cli-changes-v1.8-to-v2.1.md diff --git a/docs/api-gateway-changes-v1.8-to-v2.1.md b/docs/api-gateway-changes-v1.8-to-v2.1.md new file mode 100644 index 00000000..099dadb0 --- /dev/null +++ b/docs/api-gateway-changes-v1.8-to-v2.1.md @@ -0,0 +1,108 @@ +# API Gateway Changes: v1.8 to v2.1 + +## Summary + +The API gateway gained new WebSocket service dispatchers for embeddings +queries, a new REST streaming endpoint for document content, and underwent +a significant wire format change from `Value` to `Term`. The "objects" +service was renamed to "rows". + +--- + +## New WebSocket Service Dispatchers + +These are new request/response services available through the WebSocket +multiplexer at `/api/v1/socket` (flow-scoped): + +| Service Key | Description | +|-------------|-------------| +| `document-embeddings` | Queries document chunks by text similarity. Request/response uses `DocumentEmbeddingsRequest`/`DocumentEmbeddingsResponse` schemas. | +| `row-embeddings` | Queries structured data rows by text similarity on indexed fields. Request/response uses `RowEmbeddingsRequest`/`RowEmbeddingsResponse` schemas. | + +These join the existing `graph-embeddings` dispatcher (which was already +present in v1.8 but may have been updated). + +### Full list of WebSocket flow service dispatchers (v2.1) + +Request/response services (via `/api/v1/flow/{flow}/service/{kind}` or +WebSocket mux): + +- `agent`, `text-completion`, `prompt`, `mcp-tool` +- `graph-rag`, `document-rag` +- `embeddings`, `graph-embeddings`, `document-embeddings` +- `triples`, `rows`, `nlp-query`, `structured-query`, `structured-diag` +- `row-embeddings` + +--- + +## New REST Endpoint + +| Method | Path | Description | +|--------|------|-------------| +| `GET` | `/api/v1/document-stream` | Streams document content from the library as raw bytes. Query parameters: `user` (required), `document-id` (required), `chunk-size` (optional, default 1MB). Returns the document content in chunked transfer encoding, decoded from base64 internally. | + +--- + +## Renamed Service: "objects" to "rows" + +| v1.8 | v2.1 | Notes | +|------|------|-------| +| `objects_query.py` / `ObjectsQueryRequestor` | `rows_query.py` / `RowsQueryRequestor` | Schema changed from `ObjectsQueryRequest`/`ObjectsQueryResponse` to `RowsQueryRequest`/`RowsQueryResponse`. | +| `objects_import.py` / `ObjectsImport` | `rows_import.py` / `RowsImport` | Import dispatcher for structured data. | + +The WebSocket service key changed from `"objects"` to `"rows"`, and the +import dispatcher key similarly changed from `"objects"` to `"rows"`. + +--- + +## Wire Format Change: Value to Term + +The serialization layer (`serialize.py`) was rewritten to use the new `Term` +type instead of the old `Value` type. + +### Old format (v1.8 — `Value`) + +```json +{"v": "http://example.org/entity", "e": true} +``` + +- `v`: the value (string) +- `e`: boolean flag indicating whether the value is a URI + +### New format (v2.1 — `Term`) + +IRIs: +```json +{"t": "i", "i": "http://example.org/entity"} +``` + +Literals: +```json +{"t": "l", "v": "some text", "d": "datatype-uri", "l": "en"} +``` + +Quoted triples (RDF-star): +```json +{"t": "r", "r": {"s": {...}, "p": {...}, "o": {...}}} +``` + +- `t`: type discriminator — `"i"` (IRI), `"l"` (literal), `"r"` (quoted triple), `"b"` (blank node) +- Serialization now delegates to `TermTranslator` and `TripleTranslator` from `trustgraph.messaging.translators.primitives` + +### Other serialization changes + +| Field | v1.8 | v2.1 | +|-------|------|------| +| Metadata | `metadata.metadata` (subgraph) | `metadata.root` (simple value) | +| Graph embeddings entity | `entity.vectors` (plural) | `entity.vector` (singular) | +| Document embeddings chunk | `chunk.vectors` + `chunk.chunk` (text) | `chunk.vector` + `chunk.chunk_id` (ID reference) | + +--- + +## Breaking Changes + +- **`Value` to `Term` wire format**: All clients sending/receiving triples, embeddings, or entity contexts through the gateway must update to the new Term format. +- **`objects` to `rows` rename**: WebSocket service key and import key changed. +- **Metadata field change**: `metadata.metadata` (a serialized subgraph) replaced by `metadata.root` (a simple value). +- **Embeddings field changes**: `vectors` (plural) became `vector` (singular); document embeddings now reference `chunk_id` instead of inline `chunk` text. +- **New `/api/v1/document-stream` endpoint**: Additive, not breaking. diff --git a/docs/cli-changes-v1.8-to-v2.1.md b/docs/cli-changes-v1.8-to-v2.1.md new file mode 100644 index 00000000..9045b1b4 --- /dev/null +++ b/docs/cli-changes-v1.8-to-v2.1.md @@ -0,0 +1,112 @@ +# CLI Changes: v1.8 to v2.1 + +## Summary + +The CLI (`trustgraph-cli`) has significant additions focused on three themes: +**explainability/provenance**, **embeddings access**, and **graph querying**. +Two legacy tools were removed, one was renamed, and several existing tools +gained new capabilities. + +--- + +## New CLI Tools + +### Explainability & Provenance + +| Command | Description | +|---------|-------------| +| `tg-list-explain-traces` | Lists all explainability sessions (GraphRAG and Agent) in a collection, showing session IDs, type, question text, and timestamps. | +| `tg-show-explain-trace` | Displays the full explainability trace for a session. For GraphRAG: Question, Exploration, Focus, Synthesis stages. For Agent: Session, Iterations (thought/action/observation), Final Answer. Auto-detects trace type. Supports `--show-provenance` to trace edges back to source documents. | +| `tg-show-extraction-provenance` | Given a document ID, traverses the provenance chain: Document -> Pages -> Chunks -> Edges, using `prov:wasDerivedFrom` relationships. Supports `--show-content` and `--max-content` options. | + +### Embeddings + +| Command | Description | +|---------|-------------| +| `tg-invoke-embeddings` | Converts text to a vector embedding via the embeddings service. Accepts one or more text inputs, returns vectors as lists of floats. | +| `tg-invoke-graph-embeddings` | Queries graph entities by text similarity using vector embeddings. Returns matching entities with similarity scores. | +| `tg-invoke-document-embeddings` | Queries document chunks by text similarity using vector embeddings. Returns matching chunk IDs with similarity scores. | +| `tg-invoke-row-embeddings` | Queries structured data rows by text similarity on indexed fields. Returns matching rows with index values and scores. Requires `--schema-name` and supports `--index-name`. | + +### Graph Querying + +| Command | Description | +|---------|-------------| +| `tg-query-graph` | Pattern-based triple store query. Unlike `tg-show-graph` (which dumps everything), this allows selective queries by any combination of subject, predicate, object, and graph. Auto-detects value types: IRIs (`http://...`, `urn:...`, `<...>`), quoted triples (`<>`), and literals. | +| `tg-get-document-content` | Retrieves document content from the library by document ID. Can output to file or stdout, handles both text and binary content. | + +--- + +## Removed CLI Tools + +| Command | Notes | +|---------|-------| +| `tg-load-pdf` | Removed. Document loading is now handled through the library/processing pipeline. | +| `tg-load-text` | Removed. Document loading is now handled through the library/processing pipeline. | + +--- + +## Renamed CLI Tools + +| Old Name | New Name | Notes | +|----------|----------|-------| +| `tg-invoke-objects-query` | `tg-invoke-rows-query` | Reflects the terminology rename from "objects" to "rows" for structured data. | + +--- + +## Significant Changes to Existing Tools + +### `tg-invoke-graph-rag` + +- **Explainability support**: Now supports a 4-stage explainability pipeline (Question, Grounding/Exploration, Focus, Synthesis) with inline provenance event display. +- **Streaming**: Uses WebSocket streaming for real-time output. +- **Provenance tracing**: Can trace selected edges back to source documents via reification and `prov:wasDerivedFrom` chains. +- Grew from ~30 lines to ~760 lines to accommodate the full explainability pipeline. + +### `tg-invoke-document-rag` + +- **Explainability support**: Added `question_explainable()` mode that streams Document RAG responses with inline provenance events (Question, Grounding, Exploration, Synthesis stages). + +### `tg-invoke-agent` + +- **Explainability support**: Added `question_explainable()` mode showing provenance events inline during agent execution (Question, Analysis, Conclusion, AgentThought, AgentObservation, AgentAnswer). +- Verbose mode shows thought/observation streams with emoji prefixes. + +### `tg-show-graph` + +- **Streaming mode**: Now uses `triples_query_stream()` with configurable batch sizes for lower time-to-first-result and reduced memory overhead. +- **Named graph support**: New `--graph` filter option. Recognises named graphs: + - Default graph (empty): Core knowledge facts + - `urn:graph:source`: Extraction provenance + - `urn:graph:retrieval`: Query-time explainability +- **Show graph column**: New `--show-graph` flag to display the named graph for each triple. +- **Configurable limits**: New `--limit` and `--batch-size` options. + +### `tg-graph-to-turtle` + +- **RDF-star support**: Now handles quoted triples (RDF-star reification). +- **Streaming mode**: Uses streaming for lower time-to-first-processing. +- **Wire format handling**: Updated to use the new term wire format (`{"t": "i", "i": uri}` for IRIs, `{"t": "l", "v": value}` for literals, `{"t": "r", "r": {...}}` for quoted triples). +- **Named graph support**: New `--graph` filter option. + +### `tg-set-tool` + +- **New tool type**: `row-embeddings-query` for semantic search on structured data indexes. +- **New options**: `--schema-name`, `--index-name`, `--limit` for configuring row embeddings query tools. + +### `tg-show-tools` + +- Displays the new `row-embeddings-query` tool type with its `schema-name`, `index-name`, and `limit` fields. + +### `tg-load-knowledge` + +- **Progress reporting**: Now counts and reports triples and entity contexts loaded per file and in total. +- **Term format update**: Entity contexts now use the new Term format (`{"t": "i", "i": uri}`) instead of the old Value format (`{"v": entity, "e": True}`). + +--- + +## Breaking Changes + +- **Terminology rename**: The `Value` schema was renamed to `Term` across the system (PR #622). This affects the wire format used by CLI tools that interact with the graph store. The new format uses `{"t": "i", "i": uri}` for IRIs and `{"t": "l", "v": value}` for literals, replacing the old `{"v": ..., "e": ...}` format. +- **`tg-invoke-objects-query` renamed** to `tg-invoke-rows-query`. +- **`tg-load-pdf` and `tg-load-text` removed**.