Adds batching to structured data row imports. Previously each row was
sent as a separate WebSocket message, causing one embeddings service
call per row. The bulk import clients now accumulate rows into batches
(default 40) before sending, and the --batch-size CLI argument is
wired through from tg-load-structured-data to the import call.
Splits the schema "indexes" field into two separate configurations:
- "query-indexes": fields for Cassandra exact-match retrieval
- "vector-indexes": fields for vector embedding and semantic search
This avoids wasting compute embedding primary key integers that only
need exact lookup, while allowing each field to independently opt in
to either or both retrieval patterns.
Adds row_id (the primary key value) as a clustering column in the
Cassandra rows table so multiple rows sharing the same index value
no longer silently overwrite each other.
Tests updated for new Cassandra schema.
Add trustgraph-docling package providing a document decoder powered by
IBM's Docling library. Supports PDF, DOCX, XLSX, PPTX, HTML, Markdown,
and CSV with two chunking modes:
- page mode (default): groups output by page for page-based formats,
emits TextDocument messages for the existing downstream chunker
- hybrid mode: uses Docling's HybridChunker for structure-aware
semantic chunking, emits Chunk messages directly
Tables are preserved as HTML markup. Images are stored in the librarian
with provenance but excluded from the text pipeline. Full provenance
chain is maintained for explainability tracing.
Includes Containerfile, CI/CD integration (PR checks and release
workflows), Makefile targets, and tech spec documentation.
Added unit tests for the package.
Update test fixtures to use librarian_clients dict keyed by workspace
instead of the removed librarian kwarg. Adjust mock references and
add empty list-children response for recursive _stream_doc_tree.
Literal objects (e.g. rdfs:label values) were not being inserted into the
quads_by_entity table with role='O', so queries like `tg-query-graph -o
MIL-HDBK-516D` returned no results even though the triple existed.
Removed the otype guard that skipped entity row insertion for literals in
both insert() and async_insert(). Also updated delete_collection() and
async_delete_collection() to include literal objects in entity partition
cleanup.
Existing data requires re-ingestion for old literal objects to become
queryable by object value.
Thread existing JSON schemas from prompt definitions through the
text-completion service to LLM backends' native structured output
APIs. When a prompt has response-type "json" and a strict-mode
compatible schema, the LLM constrains token selection at the logit
level to guarantee schema-valid output.
Wire-level changes:
- Add response_format and schema fields to TextCompletionRequest
- Update translator to encode/decode new fields
- Pass new fields through LlmService, TextCompletionClient, and
PromptManager
Runtime schema compatibility checker:
- New is_strict_mode_compatible() utility validates schemas against
LLM provider constraints (additionalProperties, required fields,
no unsupported constraints, no open-ended objects)
- Per-prompt eligibility decision: compliant schemas use structured
output, non-compliant schemas fall back to free-text + post-hoc
validation
LLM backend implementations:
- OpenAI: response_format with json_schema, variant-aware top-level
array rejection (openai variant blocks, llama/vllm variants allow)
- New vllm variant for the OpenAI backend
- vLLM (dedicated): response_format in raw HTTP body
- Ollama: format=<schema> parameter
- Claude: tool-use trick (forced tool call with schema as input_schema)
- Mistral: native json_schema response_format
- Llamafile, LM Studio: OpenAI SDK response_format
- Azure OpenAI: AzureOpenAI SDK response_format
- Azure serverless: response_format in raw HTTP body
- TGI: response_format in raw HTTP body
- VertexAI Gemini: response_mime_type + response_schema
- VertexAI Claude: tool-use trick
- Google AI Studio: response_mime_type + response_schema
- Bedrock, Cohere: signature-only (no structured output yet)
Post-hoc jsonschema.validate() retained as defence-in-depth.
Tech spec added: docs/tech-specs/structured-output.md
Update tests
The entity URI normalizer stripped every non-ASCII character via the
ASCII-only patterns `[^a-z0-9\-.]` and `[^a-z0-9\-]`. For non-Latin
entity names (e.g. Chinese), this deleted the entire name, so distinct
entities collapsed onto the same URI (".../<ontology>/-"), silently
merging unrelated nodes and breaking OntoRAG for non-English content.
Switch both filters to `[^\w\-.]` / `[^\w\-]`. In Python 3, `\w` is
Unicode-aware for str patterns, so non-ASCII letters (CJK, etc.) are
preserved while punctuation and symbols are still removed. ASCII
behaviour is unchanged: names are lowercased and underscores converted
to hyphens before the filter runs, so for ASCII input the patterns are
equivalent to the originals.
Adds tests covering ASCII regression guards and non-ASCII names/types,
including the core case that two distinct Chinese entities must not
collapse onto the same URI.
Co-authored-by: arthurxuwei <5179840+arthurxuwei@users.noreply.github.com>
Adds a sparse keyword retrieval path beside the existing vector path in
document-RAG, fused by weighted Reciprocal Rank Fusion on chunk_id, behind
--retrieval-mode (vector | keyword | hybrid, default vector).
The keyword index is a new pluggable service (KeywordIndexService /
KeywordIndexClientSpec); the first backend is SQLite FTS5, consuming Chunk
messages off the ingestion stream and answering BM25 queries from one
process, since the index is a single local file. Query text is sanitized
into per-term quoted phrases (raw text is not valid FTS5 syntax), which
also makes dotted clause numbers and error codes exact-match without a
trigram index. Indexes are scoped per (workspace, collection) and dropped
on collection deletion.
The keyword-index client spec is only registered when the sparse path is
enabled, so existing flow definitions without keyword-index queues are
untouched; with retrieval_mode=vector the retrieval path is unchanged. In
hybrid mode a keyword-path failure degrades to vector-only.
* feat: streaming N-Quads serializer for wire-format triples
Groundwork for Phase 2 of #877 (knowledge export). Hand-rolled
N-Triples term encoding: rdflib's term.n3() emits Turtle-style forms
(numeric shorthand, unescaped newlines) that are invalid in
line-oriented N-Quads, so literals are escaped per the ECHAR grammar
and IRIs validated for representability. Round-trip tests parse the
output back with rdflib's nquads parser and compare term-for-term.
* feat: export/import workspace knowledge in .tgx bundles (#877)
Phase 2 of the workspace bundle commands: tg-export-workspace now
includes the workspace's knowledge by default — per-collection
knowledge-graph triples as N-Quads (the collection names the graph,
streamed through a tempfile so memory stays flat regardless of
knowledge-base size) and the document library (metadata plus content,
fetched one document at a time). --config-only skips knowledge on both
sides; --triples-limit bounds very large graphs; -f/--flow-id selects
the flow the triples services run through.
tg-import-workspace streams triples back through the bulk import per
collection and recreates library documents (children after parents).
Knowledge import is additive, unlike config's skip-existing semantics.
Embedding vectors are not carried in bundles: --process re-runs
imported documents through the flow, which regenerates extraction
output and embeddings; --process-collection targets it.
Round-trip covered by unit tests over real archives: export with a
mocked Api, re-import, and assert the bulk triples stream and library
add calls reproduce the original values (including datatyped literals
via the N-Quads path).
Add a complete audit event pipeline that emits structured, machine-
parseable events for every gateway request and IAM decision.
Schema and publisher:
- AuditEvent dataclass and notify-class queue (audit_events_queue)
- AuditPublisher utility: fire-and-forget emission with envelope
(schema_version, event_id, event_type, timestamp, producer)
- New request_id and client_ip fields on IamRequest for correlation
Gateway (gateway.request events):
- aiohttp middleware assigns request_id, captures timing/status/sizes
and emits an event after every HTTP request completes
- IamAuth.authenticate annotates the request with identity
- Main endpoint handlers annotate capability and workspace
- request_id and client_ip forwarded to IAM on authenticate/authorise
IAM service (iam.authenticate, iam.authorise, iam.management events):
- Emits iam.authenticate for resolve-api-key, login, anonymous auth
- Emits iam.authorise for authorise and authorise-many decisions
- Emits iam.management for user/workspace/key mutations
- All events include request_id for correlation with gateway events
Design: events land on a pub/sub notify topic — non-persistent,
per-subscriber delivery. If no audit consumer is deployed, events
are silently discarded. Storage, retention, and alerting are
consumer-side concerns outside this boundary.
Added unit tests for the publisher and gateway middleware, unit
tests for IAM audit emission, and a contract test for the AuditEvent
schema.
Tech spec: docs/tech-specs/audit-events.md
- Filter out RDF/RDFS/OWL schema predicates (rdfs:domain, owl:inverseOf,
etc.) from hop traversal, keeping rdf:type for data signal
- Skip edges where reranker-visible components are unlabeled IRIs, since
the cross-encoder cannot meaningfully score raw URIs
- Add max-reranker-input safety cap (default 350) to prevent overloading
the reranker, applied after filtering for maximum useful candidates
- Expose max-reranker-input as per-request parameter through schema,
translator, REST API, socket client, CLI, and OpenAPI spec
- Update tests
- Update tech spec
Phase 1 (config only): export a workspace's full configuration as a
portable .tgx bundle (gzipped tar with manifest.json and one
pretty-printed, self-describing JSON file per config key under
config/<type>/), and import it into another deployment or workspace.
Import defaults to WorkspaceInit's re-run semantics (existing keys kept,
missing keys added; --overwrite replaces), supports --workspace rename,
--dry-run, and --config-only, and refuses to silently drop knowledge
data from future Phase-2 bundles it cannot import yet.
Convention in the tests is to just import the libraries as production
code would. This is fragile, and could possibly be used to inject
malicious code in the CI environment.
The reranker document text now reflects the traversal direction,
showing only the new information relative to the frontier entity:
- From S (subject is frontier): text = "{predicate} {object}"
- From O (object is frontier): text = "{subject} {predicate}"
- From P (predicate is frontier): text = "{subject} {object}"
This eliminates duplicate reranker texts when traversing inward
from shared object nodes (e.g. 18 CPUs all producing identical
"hasSubcategory Processors" text when the subject was dropped).
execute_batch_triple_queries now returns (triple, direction)
tuples so hop_and_filter can select the appropriate text format.
Updates tech spec to document the direction-aware approach.
Adds unit tests for direction tracking and reranker text
construction.
Wire the FlashRank reranker subsystem from #1005 into Document-RAG: after
vector retrieval, over-fetch a wider candidate pool, rerank with the
cross-encoder, and keep the top doc_limit chunks for synthesis.
Per maintainer review, the fetch and select sizes are two caller-controlled
limits rather than one internal heuristic:
- doc_limit: chunks selected into the synthesis prompt (unchanged meaning).
- fetch_limit: candidate pool pulled from the vector store before reranking.
0 = derive (OVERFETCH_FACTOR x doc_limit); values below doc_limit are
raised to it. Lets the caller control how hard the reranker has to work.
Details:
- schema: DocumentRagQuery.fetch_limit (additive, backward compatible).
- document_rag.py / rag.py: fetch_limit resolved in the processor (mirrors
doc_limit); the core applies the heuristic default and derives synthesis
provenance from the chunk-selection focus when reranking ran.
- provenance: tg:ChunkSelection focus stage (mirrors tg:EdgeSelection).
- request translator + client SDKs + CLI: fetch-limit / --fetch-limit,
threaded exactly like doc_limit and the GraphRAG limits.
- tests: no-op identity, over-fetch/narrow, explicit fetch_limit, heuristic
default, floor-at-doc_limit, provenance lineage, cross-repo topic wiring.
Reranking is skipped byte-identically when no reranker role is wired.
Requires the companion trustgraph-templates change wiring the reranker
topics into the document-rag flow (mirrors #279 for GraphRAG).
Tests using MagicMock processors need the variant, thinking mode,
and _build_kwargs/_extract_content methods bound to work with the
new variant-based API kwargs construction.
Replace the three-prompt LLM scoring pipeline (kg-edge-scoring,
kg-edge-reasoning, kg-edge-selection) with a cross-encoder reranker
service backed by FlashRank. The new hop_and_filter() method performs
iterative graph traversal with semantic scoring at each hop, replacing
the previous follow_edges/get_subgraph approach.
- Add reranker service (trustgraph-base client/service, FlashRank processor)
- Add gateway dispatch for reranker via API and WebSocket
- Rewrite GraphRAG pipeline: hop_and_filter() with per-hop cross-encoder scoring
- Remove kg_prompt() and edge_score_limit from prompt client
- Update provenance: add tg:EdgeSelection type, tg:concept, tg:score predicates
- Update CLIs (tg-invoke-graph-rag, tg-show-explain-trace) for new metadata
- Add tg-invoke-reranker CLI tool
- Add tech spec and UX developer guidance
- Update all unit and integration tests
* feat: make bootstrapper initialiser timeouts configurable
DefaultFlowStart and WorkspaceInit hardcoded the request timeouts for
their flow-svc and IAM calls, leaving operators no way to tune them for
high-latency environments (#874).
Expose them as constructor parameters threaded through the existing
initialiser `params:` mechanism, defaulting to the current values so
behaviour is unchanged unless explicitly overridden:
- DefaultFlowStart: list_timeout=10 (list-flows), start_timeout=30 (start-flow)
- WorkspaceInit: iam_timeout=10 (create-workspace)
Add unit tests for the defaults, override storage, and that configured
values reach the underlying request calls.
* test: mark async bootstrap test with @pytest.mark.asyncio
Addresses review feedback on PR #999: add the explicit
@pytest.mark.asyncio decorator to test_run_forwards_configured_timeouts
so it does not rely on asyncio_mode=auto and stays consistent with the
rest of the suite.
Users are global entities, not scoped to workspaces. This change:
Track A — Global usernames:
- Change iam_users_by_username to PRIMARY KEY (username), removing
workspace from the lookup key
- Login looks up username globally, no workspace required
- Username uniqueness is enforced globally, not per-workspace
- Login -w now overrides the JWT workspace (session workspace)
rather than selecting which user registry to search
Track B — Rename workspace to default_workspace:
- UserRecord.workspace → UserRecord.default_workspace
- Identity.workspace → Identity.default_workspace
- JWT claim "workspace" → "default_workspace"
- IamResponse.resolved_workspace → resolved_default_workspace
- WebSocket auth-ok frame field → default_workspace
- Socket clients read default_workspace from auth-ok
- _user_record_to_dict wire key → default_workspace
- CLI help text and output updated throughout
- Test files updated for renamed fields
The query service now uses async_execute_paged (indexed path) and
async_scan (scan path) instead of async_execute. Tests were mocking
the old function, causing them to hang indefinitely.
resolve_cassandra_config did not accept replication_factor as a kwarg,
so cassandra_replication_factor from YAML params was silently ignored
by all 6 callers. Add the kwarg and pass it from every caller.
Same fix for Qdrant: 3 writers now pass qdrant_replication_factor and
qdrant_shard_number from params.
Add tests covering the params path for both helpers.
- Add centralised qdrant_config.py helper with env-var fallback for
QDRANT_URL, QDRANT_API_KEY, QDRANT_REPLICATION_FACTOR, QDRANT_SHARD_NUMBER
- Update all 6 Qdrant processors to use the helper; writers pass
replication_factor and shard_number to create_collection
- Fix hardcoded Cassandra replication_factor=1 in cassandra_kg.py,
write.py, and sparql_cassandra.py to respect CASSANDRA_REPLICATION_FACTOR
- Upgrade Cassandra TLS from deprecated PROTOCOL_TLSv1_2 to
ssl.create_default_context() across all connectors
Implements all three changes from the knowledge-core-completeness tech spec:
1. Named graph field preserved through Cassandra storage (7-element tuple),
enabling provenance triples to retain their graph URIs on round-trip.
2. Provenance triples already arrive on triples-input — no routing change
needed; Change 1 was sufficient.
3. Source material (library documents) streamed alongside triples and
embeddings during core download/upload. The knowledge manager fetches
the document hierarchy from the librarian on download and recreates it
on upload, preserving the full provenance chain across instances.
Bulk clients (sync and async) were not forwarding the workspace parameter,
causing all bulk operations to hit the default workspace regardless of the
Api instance's workspace setting. Also fixes the gateway socket endpoint to
pass query parameters (including workspace) to the dispatcher, and prevents
the auth handshake from overwriting an explicitly set workspace.
Updates knowledge table store tests for paged query interface.
The tests were patching
trustgraph.embeddings.hf.hf.HuggingFaceEmbeddings - a module-level
attribute that doesn't exist because HuggingFaceEmbeddings is
imported locally inside _load_model. Changed all 8 occurrences to
patch langchain_huggingface.HuggingFaceEmbeddings, which is the
actual import source the code uses at runtime.
Replace hallucinated relative imports with correct absolute imports
across the ontology query package, and fix OntologyMatcher reference
to match the actual class name OntologyMatcherForQueries. Simplify
test to use standard imports instead of importlib hack.
Cosmetic, but simpler imports provides undeterministic imports in a dev
environment, and also means we're properly testing linkage
Convert the SPARQL algebra evaluator from eager list-based evaluation to
lazy async generators so results stream incrementally. This lets Slice
terminate early (via generator cleanup) and avoids materialising full
result sets for streamable operators like Project, Filter, Union, and
Extend. Blocking operators (Join, LeftJoin, OrderBy, Group) materialise
at their boundary then yield.
Add bind join optimization for Join nodes where one side is small
(VALUES/ToMultiSet): instead of materialising both sides independently
and hash-joining, iterate the small side's bindings and evaluate the
large side with those bindings pre-seeded. This turns wildcard BGP
queries into selective ones — e.g. VALUES ?x { <uri> } joined with a
BGP now queries the triple store with ?x bound rather than fetching
all triples.
Add TriplesClient.query_gen() async generator that wraps the existing
streaming callback API via an asyncio.Queue bridge, yielding individual
Triple objects as batches arrive.
Add streaming request path in the SPARQL query service that batches
solutions from the live async generator and sends them as they fill.
Fix FILTER IN/NOT IN: rdflib represents these as RelationalExpression
nodes with op="IN", not as Builtin_IN — handle both representations.
Fix Builtin_IN/Builtin_NOTIN dispatch ordering so the specific handlers
are checked before the generic Builtin_ prefix match.
Fix VALUES handling for rdflib's two representations: positional
(var/value) and dict-based (res).
Add 30+ SPARQL 1.1 built-in functions and the MINUS algebra operator to the
custom SPARQL query backend.
String functions:
- SUBSTR (2-arg and 3-arg forms), STRBEFORE, STRAFTER
- REPLACE (regex with flags), ENCODE_FOR_URI
Numeric functions:
- FLOOR, CEIL, ROUND, ABS
Date/time accessors:
- YEAR, MONTH, DAY, HOURS, MINUTES, SECONDS
- NOW, TZ
Hash functions:
- MD5, SHA1, SHA256, SHA512
Term constructors:
- IRI/URI, BNODE, UUID, STRUUID
Other functions:
- LANGMATCHES, RAND
- EXISTS / NOT EXISTS (with async pre-evaluation to bridge the
sync expression evaluator and async algebra evaluator)
Algebra:
- MINUS set-difference operator
- HAVING already works via rdflib's Filter mapping (verified)
Fix SPARQL ORDER handling
Includes 653 lines of new unit tests covering all added functionality
across expressions, solutions, and algebra layers.
service.py:
- Constructor takes **config (same pattern as api-gateway) instead
of individual args
- Creates IamAuth and calls await self.auth.start() before the
message loop
- Passes auth to both ConfigReceiver and MessageDispatcher
- Uses add_pubsub_args / add_logging_args instead of hand-rolled
Pulsar args
- Passes timeout through
dispatcher.py:
- Accepts auth and timeout parameters
- Passes both to DispatcherManager — fixes the missing auth argument
that would have crashed on startup
The remote end's requests now go through the same IAM authentication
path as api-gateway. Token validation, workspace resolution, and
permissions all work identically regardless of which direction
initiated the connection.
Fixed tests — the test now passes auth and timeout to MessageDispatcher
and verifies they're forwarded to DispatcherManager.
Update rev gateway dispatcher to align with IAM. A "token" parameter
must be passed with each message.
Fix websocket relay to align with rev-gateway changes, conforms to
the api-gateway protocol.
Revert consumer receive timeout from 100ms back to the original
2000ms. The 100ms change was based on a misunderstanding — receive()
is a blocking call that returns immediately when a message arrives,
so the timeout only affects how quickly a consumer checks the shutdown
flag during idle periods. 100ms generated ~200 WARN lines/sec from
the C++ client with no latency benefit.
Also set the Pulsar C++ client logger to Error level so residual
timeout warnings from the subscriber (250ms) don't produce noise.
Update poll timeout test to match reverted 2000ms value
Adds `no-auth-svc`, a lightweight IAM service that permits all access
unconditionally — no database, no bootstrap, no signing keys. Deploy
it in place of `iam-svc` for development, demos, and single-user
setups where authentication overhead is unwanted.
The gateway no longer hard-codes a 401 on missing credentials.
Instead it asks the IAM regime via a new `authenticate-anonymous`
operation whether token-free access is allowed. This keeps the
gateway regime-agnostic: `iam-svc` rejects anonymous auth (preserving
existing security), while `no-auth-svc` grants it with a configurable
default user and workspace.
Includes a tech spec (docs/tech-specs/no-auth-regime.md) and tests
that pin the safety boundary — malformed tokens never fall through
to the anonymous path, and a contract test ensures the full iam-svc
always rejects `authenticate-anonymous`.
- Align similarity_threshold default to 0.3 everywhere (class signature
had stale 0.7). Fix matching contradiction in tech-spec.
- Add bypass_selector_below parameter (default 5) to skip vector
similarity selection when ontology element count is small enough.
- Enforce domain/range constraints in TripleConverter for object
properties and datatype properties, with subclass hierarchy support.
Properties with no declared domain/range pass through unchanged.
- Add unit tests for domain/range validation, subclass acceptance,
polymorphic pass-through, and selector bypass.
Fixes#908, #920
Fix 5 cascading bugs in the Library API wrapper that prevented
the get_documents → update_document round-trip from working:
- Tolerate missing title field in document metadata (use .get())
- Use attribute access on Triple objects instead of subscript
- Serialize datetime to int seconds for JSON compatibility
- Handle empty server response on successful update
- Send both id and document-id keys in update request
Added library API tests
Cassandra triples services were using syncronous EntityCentricKnowledgeGraph
methods from async contexts, and connection state was managed with
threading.local which is wrong for asyncio coroutines sharing a single
thread. Qdrant services had no async wrapping at all, blocking the event
loop on every network call. Rows services had unprotected shared state
mutations across concurrent coroutines.
- Add async methods to EntityCentricKnowledgeGraph (async_insert,
async_get_s/p/o/sp/po/os/spo/all, async_collection_exists,
async_create_collection, async_delete_collection) using the existing
cassandra_async.async_execute bridge
- Rewrite triples write + query services: replace threading.local with
asyncio.Lock + dict cache for per-workspace connections, use async
ECKG methods for all data operations, keep asyncio.to_thread only for
one-time blocking ECKG construction
- Wrap all Qdrant calls in asyncio.to_thread across all 6 services
(doc/graph/row embeddings write + query), add asyncio.Lock + set cache
for collection existence checks
- Add asyncio.Lock to rows write + query services to protect shared
state (schemas, sessions, config caches) from concurrent mutation
- Update all affected tests to match new async patterns
Fix threading of workspace paramater:
- The SPARQL algebra evaluator was threading a workspace parameter
through every function and passing it to TriplesClient.query(),
which doesn't accept it. Workspace isolation is handled by pub/sub
topic routing — the TriplesClient is already scoped to a
workspace-specific flow, same as GraphRAG. Passing workspace
explicitly was both incorrect and unnecessary.
Update tests:
- tests/unit/test_query/test_sparql_algebra.py (new) — Tests
_query_pattern, _eval_bgp, and evaluate() with various algebra
nodes. Key tests assert workspace is never in tc.query() kwargs,
plus correctness tests for BGP, JOIN, UNION, SLICE, DISTINCT, and
edge cases.
- tests/unit/test_retrieval/test_graph_rag.py — Added
test_triples_query_never_passes_workspace (checks query()) and
test_follow_edges_never_passes_workspace (checks query_stream()).
* Fix publisher resource leak in librarian submit_document (#883)
Wrap pub.start()/pub.send() in try/finally to guarantee pub.stop() is
called on error. Remove unnecessary asyncio.sleep(1) kludge.
* Make Cassandra replication factor configurable (issue #787) (#887)
Add CASSANDRA_REPLICATION_FACTOR environment variable and
--cassandra-replication-factor CLI argument to cassandra_config.py.
Update all four table store constructors (ConfigTableStore,
KnowledgeTableStore, LibraryTableStore, IamTableStore) to accept
an optional replication_factor parameter and use it in keyspace
creation CQL queries.
Thread the replication factor through all service constructors:
Configuration, KnowledgeManager, Librarian, IamService, and
knowledge store Processor.
* Update tests
---------
Co-authored-by: gittihub-jpg <rico@springer-mail.net>
- Fixed document-rag workspace problem
- OpenAI text-completion processor now puts 'not-set' in the token
if no token is set (new OpenAI library requires it to be set to
something.
- Update tests
Replace singleton LibrarianClient with per-flow instances via the new
LibrarianSpec, giving each flow its own librarian tied to the
workspace-scoped request/response queues from the blueprint.
Move all workspace-scoped services (config, flow, librarian, knowledge)
from a single base-queue response producer to per-workspace response
producers created alongside the existing per-workspace request
consumers. Update the gateway dispatcher and bootstrapper flow client
to subscribe to the matching workspace-scoped response queues.
Fix WorkspaceInit to register workspaces through the IAM
create-workspace API so they appear in __workspaces__ and are visible
to the gateway. Simplify the bootstrapper gate to only check
config-svc reachability.
Updated tests accordingly.
Workspace identity is now determined by queue infrastructure instead of
message body fields, closing a privilege-escalation vector where a caller
could spoof workspace in the request payload.
- Add WorkspaceProcessor base class: discovers workspaces from config at
startup, creates per-workspace consumers (queue:workspace), and manages
consumer lifecycle on workspace create/delete events
- Roll out to librarian, flow-svc, knowledge cores, and config-svc
- Config service gets a dual-queue regime: a system queue for
cross-workspace ops (getvalues-all-ws, bootstrapper writes to
__workspaces__) and per-workspace queues for tenant-scoped ops, with
workspace discovery from its own Cassandra store
- Remove workspace field from request schemas (FlowRequest,
LibrarianRequest, KnowledgeRequest, CollectionManagementRequest) and
from DocumentMetadata / ProcessingMetadata — table stores now accept
workspace as an explicit parameter
- Strip workspace encode/decode from all message translators and gateway
serializers
- Gateway enforces workspace existence: reject requests targeting
non-existent workspaces instead of routing to queues with no consumer
- Config service provisions new workspaces from __template__ on creation
- Add workspace lifecycle hooks to AsyncProcessor so any processor can
react to workspace create/delete without subclassing WorkspaceProcessor
Fixes#826. Addresses all five points the maintainer called out in
the follow-up to #825.
Source change (trustgraph-flow/trustgraph/extract/kg/ontology/extract.py):
- Added `_is_subclass_of(cls, target, ontology_subset, max_depth=100)`
helper with visited-set cycle detection + a defensive depth cap.
LLM-generated ontologies may emit cycles (A subclass_of B,
B subclass_of A); the prior while-loop would infinite-loop on that.
- Replaced both near-identical domain and range subclass walks in
`is_valid_triple` with a single call to the new helper. Net is
-20 duplicated lines + 26-line helper.
Tests (tests/unit/test_extract/test_ontology/test_prompt_and_extraction.py):
- test_is_valid_triple_subclass_is_accepted: domain expects Recipe,
actual type is Cake (subclass), validates.
- test_is_valid_triple_handles_subclass_cycle_without_infinite_loop:
A subclass_of B, B subclass_of A; call returns False within the
depth cap rather than hanging.
- test_parse_and_validate_triples_collects_entity_types_from_rdf_type:
end-to-end path: rdf:type triples build the entity_types dict,
subsequent domain-check triples validate against it.
- test_is_valid_triple_entity_types_none_default: the None default
path now has explicit coverage.
156 existing tests in tests/unit/test_extract/test_ontology still pass.
The gateway no longer holds any policy state — capability sets, role
definitions, workspace scope rules. Per the IAM contract it asks the
regime "may this identity perform this capability on this resource?"
per request. That moves the OSS role-based regime entirely into
iam-svc, which can be replaced (SSO, ABAC, ReBAC) without changing
the gateway, the wire protocol, or backend services.
Contract:
- authenticate(credential) -> Identity (handle, workspace,
principal_id, source). No roles, claims, or policy state surface
to the gateway.
- authorise(identity, capability, resource, parameters) -> (allow,
ttl). Cached per-decision (regime TTL clamped above; fail-closed
on regime errors).
- authorise_many available as a fan-out variant.
Operation registry drives every authorisation decision:
- /api/v1/iam -> IamEndpoint, looks up bare op name (create-user,
list-workspaces, ...).
- /api/v1/{kind} -> RegistryRoutedVariableEndpoint, <kind>:<op>
(config:get, flow:list-blueprints, librarian:add-document, ...).
- /api/v1/flow/{flow}/service/{kind} -> flow-service:<kind>.
- /api/v1/flow/{flow}/{import,export}/{kind} ->
flow-{import,export}:<kind>.
- WS Mux per-frame -> flow-service:<kind>; closes a gap where
authenticated users could hit any service kind.
85 operations registered across the surface.
JWT carries identity only — sub + workspace. The roles claim is gone;
the gateway never reads policy state from a credential.
The three coarse *_KIND_CAPABILITY maps are removed. The registry is
the only source of truth for the capability + resource shape of an
operation. Tests migrated to the new Identity shape and to
authorise()-mocked auth doubles.
Specs updated: docs/tech-specs/iam-contract.md (Identity surface,
caching, registry-naming conventions), iam.md (JWT shape, gateway
flow, role section reframed as OSS-regime detail), iam-protocol.md
(positioned as one implementation of the contract).
