- Increase per-request config timeout from 10s to 60s
- Parallelize per-workspace config fetches with asyncio.gather
- Track applied workspaces so retries resume instead of restarting
With 50 workspaces, sequential fetches with 10s timeouts caused a
cascade: timeouts triggered retries that re-fetched everything,
resulting in 10+ minute startup times.
The API gateway URL (port 8088) is not publicly accessible in most
deployments since the UI and gateway share the same URL space. Use
the UI URL for all checks and remove the separate --api-url and
--pulsar-url parameters along with the unused check_pulsar function.
With many workspaces and flows, the prometheus metrics expand
greatly. Many significant metrics (e.g. latency) make more sense
measured globally. This change introduces a consolidation.
Histograms (request_latency, text_completion_duration, chunk_size, etc.)
are now global with only processor-level labels, eliminating per-workspace
time series multiplication. Counters and enums retain workspace and flow
labels for per-workspace debugging visibility.
Renamed metric labels for clarity: name -> consumer/producer/subscriber,
and added explicit workspace label alongside flow. Fixed bug where
request_response_spec labelled response subscriber with request_name
instead of response_name. Removed unused Histogram imports from model
and service files.
With many workspaces, the getvalues-all-ws response for types with
large values (e.g. prompt templates) exceeds Pulsar's max message size.
Add a getkeys-all-ws operation that returns workspace/key pairs without
values. Processors now use this to discover which workspaces have config
of a given type, then fetch values per-workspace. Each individual
response stays within message size limits.
The send method had two sequential while loops: one for connecting and
one for sending. After a send failure the producer was closed and set
to None, but control stayed in the send loop rather than returning to
the connect loop. The next iteration called .send() on None, then
.close() on None in the except block, propagating an AttributeError
that masked the original error (e.g. MessageTooBig).
Nest the connect loop inside the send loop so failures trigger
reconnection, and guard the .close() call against None.
The indexed query path only used a single index for the Cassandra lookup
and ignored any remaining filters. This caused multi-field GraphQL
queries to return results matching only the first indexed filter.
Post-filter all results against the full filter set after the index
lookup, and apply the limit after filtering to avoid short results.
Adds batching to structured data row imports. Previously each row was
sent as a separate WebSocket message, causing one embeddings service
call per row. The bulk import clients now accumulate rows into batches
(default 40) before sending, and the --batch-size CLI argument is
wired through from tg-load-structured-data to the import call.
Splits the schema "indexes" field into two separate configurations:
- "query-indexes": fields for Cassandra exact-match retrieval
- "vector-indexes": fields for vector embedding and semantic search
This avoids wasting compute embedding primary key integers that only
need exact lookup, while allowing each field to independently opt in
to either or both retrieval patterns.
Adds row_id (the primary key value) as a clustering column in the
Cassandra rows table so multiple rows sharing the same index value
no longer silently overwrite each other.
Tests updated for new Cassandra schema.
Add trustgraph-docling package providing a document decoder powered by
IBM's Docling library. Supports PDF, DOCX, XLSX, PPTX, HTML, Markdown,
and CSV with two chunking modes:
- page mode (default): groups output by page for page-based formats,
emits TextDocument messages for the existing downstream chunker
- hybrid mode: uses Docling's HybridChunker for structure-aware
semantic chunking, emits Chunk messages directly
Tables are preserved as HTML markup. Images are stored in the librarian
with provenance but excluded from the text pipeline. Full provenance
chain is maintained for explainability tracing.
Includes Containerfile, CI/CD integration (PR checks and release
workflows), Makefile targets, and tech spec documentation.
Added unit tests for the package.
Update test fixtures to use librarian_clients dict keyed by workspace
instead of the removed librarian kwarg. Adjust mock references and
add empty list-children response for recursive _stream_doc_tree.
Change librarian from a single shared client to per-workspace clients,
ensuring knowledge core get/put operations use workspace-scoped queues.
LibrarianClient is now created in on_workspace_created and cleaned up
in on_workspace_deleted.
Fix term_to_tuple/tuple_to_term to handle TRIPLE and BLANK term types.
Previously only IRI and LITERAL were supported, causing ~1000 edges
(ns/contains with quoted triple objects) to be silently lost during
Cassandra storage. TRIPLE terms are now JSON-serialized with a
<<TRIPLE>> marker prefix, BLANK terms with <<BLANK>>.
Make document tree export recursive via _stream_doc_tree so that
grandchild documents (chunks under pages) are included in knowledge
core export. Without this, explainability source links break after
a get/put/load round-trip because chunk documents are not restored.
Add error response encoding to KnowledgeResponseTranslator.encode()
and fix encode_with_completion to treat error responses as non-final.
Change KnowledgeResponse.ids from list[str] with default factory to
Optional[list[str]] so list responses can be distinguished from empty
responses. Add error display to get_kg_core CLI.
Language tags and XSD datatypes on RDF literals were being silently
dropped at multiple points in the pipeline, making SPARQL FILTER(LANG())
queries and multilingual datasets non-functional.
Ingestion fixes:
- load_knowledge.py and load_turtle.py now extract .language and
.datatype from rdflib Literal objects instead of discarding them
- Triple dataclass gains optional o_datatype and o_language fields
- _string_to_term() emits "dt" and "ln" in the wire format
- AsyncBulkClient uses the same term serialization as BulkClient
Query fix:
- TriplesClient.query_gen() preserves Term objects directly instead of
converting through Uri/Literal (which are str subclasses with no room
for language metadata), enabling SPARQL LANG() filters to work
Literal objects (e.g. rdfs:label values) were not being inserted into the
quads_by_entity table with role='O', so queries like `tg-query-graph -o
MIL-HDBK-516D` returned no results even though the triple existed.
Removed the otype guard that skipped entity row insertion for literals in
both insert() and async_insert(). Also updated delete_collection() and
async_delete_collection() to include literal objects in entity partition
cleanup.
Existing data requires re-ingestion for old literal objects to become
queryable by object value.
Thread existing JSON schemas from prompt definitions through the
text-completion service to LLM backends' native structured output
APIs. When a prompt has response-type "json" and a strict-mode
compatible schema, the LLM constrains token selection at the logit
level to guarantee schema-valid output.
Wire-level changes:
- Add response_format and schema fields to TextCompletionRequest
- Update translator to encode/decode new fields
- Pass new fields through LlmService, TextCompletionClient, and
PromptManager
Runtime schema compatibility checker:
- New is_strict_mode_compatible() utility validates schemas against
LLM provider constraints (additionalProperties, required fields,
no unsupported constraints, no open-ended objects)
- Per-prompt eligibility decision: compliant schemas use structured
output, non-compliant schemas fall back to free-text + post-hoc
validation
LLM backend implementations:
- OpenAI: response_format with json_schema, variant-aware top-level
array rejection (openai variant blocks, llama/vllm variants allow)
- New vllm variant for the OpenAI backend
- vLLM (dedicated): response_format in raw HTTP body
- Ollama: format=<schema> parameter
- Claude: tool-use trick (forced tool call with schema as input_schema)
- Mistral: native json_schema response_format
- Llamafile, LM Studio: OpenAI SDK response_format
- Azure OpenAI: AzureOpenAI SDK response_format
- Azure serverless: response_format in raw HTTP body
- TGI: response_format in raw HTTP body
- VertexAI Gemini: response_mime_type + response_schema
- VertexAI Claude: tool-use trick
- Google AI Studio: response_mime_type + response_schema
- Bedrock, Cohere: signature-only (no structured output yet)
Post-hoc jsonschema.validate() retained as defence-in-depth.
Tech spec added: docs/tech-specs/structured-output.md
Update tests
The entity URI normalizer stripped every non-ASCII character via the
ASCII-only patterns `[^a-z0-9\-.]` and `[^a-z0-9\-]`. For non-Latin
entity names (e.g. Chinese), this deleted the entire name, so distinct
entities collapsed onto the same URI (".../<ontology>/-"), silently
merging unrelated nodes and breaking OntoRAG for non-English content.
Switch both filters to `[^\w\-.]` / `[^\w\-]`. In Python 3, `\w` is
Unicode-aware for str patterns, so non-ASCII letters (CJK, etc.) are
preserved while punctuation and symbols are still removed. ASCII
behaviour is unchanged: names are lowercased and underscores converted
to hyphens before the filter runs, so for ASCII input the patterns are
equivalent to the originals.
Adds tests covering ASCII regression guards and non-ASCII names/types,
including the core case that two distinct Chinese entities must not
collapse onto the same URI.
Co-authored-by: arthurxuwei <5179840+arthurxuwei@users.noreply.github.com>
If the gateway started before IAM was ready and exhausted its startup
retries, _signing_public_pem stayed None and all JWT authentication
permanently 401'd with no recovery path short of a restart. Now
_verify_jwt retries the fetch once on demand, so the gateway self-heals
as soon as IAM becomes available.
Adds a sparse keyword retrieval path beside the existing vector path in
document-RAG, fused by weighted Reciprocal Rank Fusion on chunk_id, behind
--retrieval-mode (vector | keyword | hybrid, default vector).
The keyword index is a new pluggable service (KeywordIndexService /
KeywordIndexClientSpec); the first backend is SQLite FTS5, consuming Chunk
messages off the ingestion stream and answering BM25 queries from one
process, since the index is a single local file. Query text is sanitized
into per-term quoted phrases (raw text is not valid FTS5 syntax), which
also makes dotted clause numbers and error codes exact-match without a
trigram index. Indexes are scoped per (workspace, collection) and dropped
on collection deletion.
The keyword-index client spec is only registered when the sparse path is
enabled, so existing flow definitions without keyword-index queues are
untouched; with retrieval_mode=vector the retrieval path is unchanged. In
hybrid mode a keyword-path failure degrades to vector-only.
Documents and processing records without a stored time field caused a
KeyError crash in get_documents and list_children. Use optional access
so time resolves to None instead of failing.
* feat: streaming N-Quads serializer for wire-format triples
Groundwork for Phase 2 of #877 (knowledge export). Hand-rolled
N-Triples term encoding: rdflib's term.n3() emits Turtle-style forms
(numeric shorthand, unescaped newlines) that are invalid in
line-oriented N-Quads, so literals are escaped per the ECHAR grammar
and IRIs validated for representability. Round-trip tests parse the
output back with rdflib's nquads parser and compare term-for-term.
* feat: export/import workspace knowledge in .tgx bundles (#877)
Phase 2 of the workspace bundle commands: tg-export-workspace now
includes the workspace's knowledge by default — per-collection
knowledge-graph triples as N-Quads (the collection names the graph,
streamed through a tempfile so memory stays flat regardless of
knowledge-base size) and the document library (metadata plus content,
fetched one document at a time). --config-only skips knowledge on both
sides; --triples-limit bounds very large graphs; -f/--flow-id selects
the flow the triples services run through.
tg-import-workspace streams triples back through the bulk import per
collection and recreates library documents (children after parents).
Knowledge import is additive, unlike config's skip-existing semantics.
Embedding vectors are not carried in bundles: --process re-runs
imported documents through the flow, which regenerates extraction
output and embeddings; --process-collection targets it.
Round-trip covered by unit tests over real archives: export with a
mocked Api, re-import, and assert the bulk triples stream and library
add calls reproduce the original values (including datatyped literals
via the N-Quads path).
Add a complete audit event pipeline that emits structured, machine-
parseable events for every gateway request and IAM decision.
Schema and publisher:
- AuditEvent dataclass and notify-class queue (audit_events_queue)
- AuditPublisher utility: fire-and-forget emission with envelope
(schema_version, event_id, event_type, timestamp, producer)
- New request_id and client_ip fields on IamRequest for correlation
Gateway (gateway.request events):
- aiohttp middleware assigns request_id, captures timing/status/sizes
and emits an event after every HTTP request completes
- IamAuth.authenticate annotates the request with identity
- Main endpoint handlers annotate capability and workspace
- request_id and client_ip forwarded to IAM on authenticate/authorise
IAM service (iam.authenticate, iam.authorise, iam.management events):
- Emits iam.authenticate for resolve-api-key, login, anonymous auth
- Emits iam.authorise for authorise and authorise-many decisions
- Emits iam.management for user/workspace/key mutations
- All events include request_id for correlation with gateway events
Design: events land on a pub/sub notify topic — non-persistent,
per-subscriber delivery. If no audit consumer is deployed, events
are silently discarded. Storage, retention, and alerting are
consumer-side concerns outside this boundary.
Added unit tests for the publisher and gateway middleware, unit
tests for IAM audit emission, and a contract test for the AuditEvent
schema.
Tech spec: docs/tech-specs/audit-events.md
Documents and processing records without a stored time field caused a
KeyError crash in get_documents and list_children. Use optional access
so time resolves to None instead of failing.
* feat: streaming N-Quads serializer for wire-format triples
Groundwork for Phase 2 of #877 (knowledge export). Hand-rolled
N-Triples term encoding: rdflib's term.n3() emits Turtle-style forms
(numeric shorthand, unescaped newlines) that are invalid in
line-oriented N-Quads, so literals are escaped per the ECHAR grammar
and IRIs validated for representability. Round-trip tests parse the
output back with rdflib's nquads parser and compare term-for-term.
* feat: export/import workspace knowledge in .tgx bundles (#877)
Phase 2 of the workspace bundle commands: tg-export-workspace now
includes the workspace's knowledge by default — per-collection
knowledge-graph triples as N-Quads (the collection names the graph,
streamed through a tempfile so memory stays flat regardless of
knowledge-base size) and the document library (metadata plus content,
fetched one document at a time). --config-only skips knowledge on both
sides; --triples-limit bounds very large graphs; -f/--flow-id selects
the flow the triples services run through.
tg-import-workspace streams triples back through the bulk import per
collection and recreates library documents (children after parents).
Knowledge import is additive, unlike config's skip-existing semantics.
Embedding vectors are not carried in bundles: --process re-runs
imported documents through the flow, which regenerates extraction
output and embeddings; --process-collection targets it.
Round-trip covered by unit tests over real archives: export with a
mocked Api, re-import, and assert the bulk triples stream and library
add calls reproduce the original values (including datatyped literals
via the N-Quads path).
- Filter out RDF/RDFS/OWL schema predicates (rdfs:domain, owl:inverseOf,
etc.) from hop traversal, keeping rdf:type for data signal
- Skip edges where reranker-visible components are unlabeled IRIs, since
the cross-encoder cannot meaningfully score raw URIs
- Add max-reranker-input safety cap (default 350) to prevent overloading
the reranker, applied after filtering for maximum useful candidates
- Expose max-reranker-input as per-request parameter through schema,
translator, REST API, socket client, CLI, and OpenAPI spec
- Update tests
- Update tech spec
Phase 1 (config only): export a workspace's full configuration as a
portable .tgx bundle (gzipped tar with manifest.json and one
pretty-printed, self-describing JSON file per config key under
config/<type>/), and import it into another deployment or workspace.
Import defaults to WorkspaceInit's re-run semantics (existing keys kept,
missing keys added; --overwrite replaces), supports --workspace rename,
--dry-run, and --config-only, and refuses to silently drop knowledge
data from future Phase-2 bundles it cannot import yet.
Convention in the tests is to just import the libraries as production
code would. This is fragile, and could possibly be used to inject
malicious code in the CI environment.
The reranker document text now reflects the traversal direction,
showing only the new information relative to the frontier entity:
- From S (subject is frontier): text = "{predicate} {object}"
- From O (object is frontier): text = "{subject} {predicate}"
- From P (predicate is frontier): text = "{subject} {object}"
This eliminates duplicate reranker texts when traversing inward
from shared object nodes (e.g. 18 CPUs all producing identical
"hasSubcategory Processors" text when the subject was dropped).
execute_batch_triple_queries now returns (triple, direction)
tuples so hop_and_filter can select the appropriate text format.
Updates tech spec to document the direction-aware approach.
Adds unit tests for direction tracking and reranker text
construction.
Wire the FlashRank reranker subsystem from #1005 into Document-RAG: after
vector retrieval, over-fetch a wider candidate pool, rerank with the
cross-encoder, and keep the top doc_limit chunks for synthesis.
Per maintainer review, the fetch and select sizes are two caller-controlled
limits rather than one internal heuristic:
- doc_limit: chunks selected into the synthesis prompt (unchanged meaning).
- fetch_limit: candidate pool pulled from the vector store before reranking.
0 = derive (OVERFETCH_FACTOR x doc_limit); values below doc_limit are
raised to it. Lets the caller control how hard the reranker has to work.
Details:
- schema: DocumentRagQuery.fetch_limit (additive, backward compatible).
- document_rag.py / rag.py: fetch_limit resolved in the processor (mirrors
doc_limit); the core applies the heuristic default and derives synthesis
provenance from the chunk-selection focus when reranking ran.
- provenance: tg:ChunkSelection focus stage (mirrors tg:EdgeSelection).
- request translator + client SDKs + CLI: fetch-limit / --fetch-limit,
threaded exactly like doc_limit and the GraphRAG limits.
- tests: no-op identity, over-fetch/narrow, explicit fetch_limit, heuristic
default, floor-at-doc_limit, provenance lineage, cross-repo topic wiring.
Reranking is skipped byte-identically when no reranker role is wired.
Requires the companion trustgraph-templates change wiring the reranker
topics into the document-rag flow (mirrors #279 for GraphRAG).
Replace the client.post()/httpx bypass with standard SDK extra_body,
confirmed working against DashScope. Make DashScope the base variant
with Qwen as a subclass alias. Route all API calls through variant
create_completion/create_completion_stream methods.
Interactive bash installer (install_trustgraph.sh) that detects hardware,
recommends an LLM mode (OpenAI or Ollama), installs missing prerequisites
via Homebrew, sets up a Python venv, runs the test suite, generates a
deployment via npx @trustgraph/config, starts the Docker Compose stack,
health-checks the API gateway, and opens the Workbench UI.
Includes README.dev-install.md with usage documentation covering CLI
options, environment variables, LLM mode selection, non-interactive/CI
usage, uninstall, and troubleshooting. Currently macOS only.
Tests using MagicMock processors need the variant, thinking mode,
and _build_kwargs/_extract_content methods bound to work with the
new variant-based API kwargs construction.
Add a --variant flag (openai, deepseek, qwen, mistral, llama) that
encapsulates provider-specific API differences: output token parameter
names, thinking/reasoning toggles, temperature rules, and thinking
output extraction. Add --thinking flag (off, low, medium, high) to
control reasoning effort.
Replace the three-prompt LLM scoring pipeline (kg-edge-scoring,
kg-edge-reasoning, kg-edge-selection) with a cross-encoder reranker
service backed by FlashRank. The new hop_and_filter() method performs
iterative graph traversal with semantic scoring at each hop, replacing
the previous follow_edges/get_subgraph approach.
- Add reranker service (trustgraph-base client/service, FlashRank processor)
- Add gateway dispatch for reranker via API and WebSocket
- Rewrite GraphRAG pipeline: hop_and_filter() with per-hop cross-encoder scoring
- Remove kg_prompt() and edge_score_limit from prompt client
- Update provenance: add tg:EdgeSelection type, tg:concept, tg:score predicates
- Update CLIs (tg-invoke-graph-rag, tg-show-explain-trace) for new metadata
- Add tg-invoke-reranker CLI tool
- Add tech spec and UX developer guidance
- Update all unit and integration tests
