Convention in the tests is to just import the libraries as production
code would. This is fragile, and could possibly be used to inject
malicious code in the CI environment.
The reranker document text now reflects the traversal direction,
showing only the new information relative to the frontier entity:
- From S (subject is frontier): text = "{predicate} {object}"
- From O (object is frontier): text = "{subject} {predicate}"
- From P (predicate is frontier): text = "{subject} {object}"
This eliminates duplicate reranker texts when traversing inward
from shared object nodes (e.g. 18 CPUs all producing identical
"hasSubcategory Processors" text when the subject was dropped).
execute_batch_triple_queries now returns (triple, direction)
tuples so hop_and_filter can select the appropriate text format.
Updates tech spec to document the direction-aware approach.
Adds unit tests for direction tracking and reranker text
construction.
Wire the FlashRank reranker subsystem from #1005 into Document-RAG: after
vector retrieval, over-fetch a wider candidate pool, rerank with the
cross-encoder, and keep the top doc_limit chunks for synthesis.
Per maintainer review, the fetch and select sizes are two caller-controlled
limits rather than one internal heuristic:
- doc_limit: chunks selected into the synthesis prompt (unchanged meaning).
- fetch_limit: candidate pool pulled from the vector store before reranking.
0 = derive (OVERFETCH_FACTOR x doc_limit); values below doc_limit are
raised to it. Lets the caller control how hard the reranker has to work.
Details:
- schema: DocumentRagQuery.fetch_limit (additive, backward compatible).
- document_rag.py / rag.py: fetch_limit resolved in the processor (mirrors
doc_limit); the core applies the heuristic default and derives synthesis
provenance from the chunk-selection focus when reranking ran.
- provenance: tg:ChunkSelection focus stage (mirrors tg:EdgeSelection).
- request translator + client SDKs + CLI: fetch-limit / --fetch-limit,
threaded exactly like doc_limit and the GraphRAG limits.
- tests: no-op identity, over-fetch/narrow, explicit fetch_limit, heuristic
default, floor-at-doc_limit, provenance lineage, cross-repo topic wiring.
Reranking is skipped byte-identically when no reranker role is wired.
Requires the companion trustgraph-templates change wiring the reranker
topics into the document-rag flow (mirrors #279 for GraphRAG).
Replace the client.post()/httpx bypass with standard SDK extra_body,
confirmed working against DashScope. Make DashScope the base variant
with Qwen as a subclass alias. Route all API calls through variant
create_completion/create_completion_stream methods.
Interactive bash installer (install_trustgraph.sh) that detects hardware,
recommends an LLM mode (OpenAI or Ollama), installs missing prerequisites
via Homebrew, sets up a Python venv, runs the test suite, generates a
deployment via npx @trustgraph/config, starts the Docker Compose stack,
health-checks the API gateway, and opens the Workbench UI.
Includes README.dev-install.md with usage documentation covering CLI
options, environment variables, LLM mode selection, non-interactive/CI
usage, uninstall, and troubleshooting. Currently macOS only.
Tests using MagicMock processors need the variant, thinking mode,
and _build_kwargs/_extract_content methods bound to work with the
new variant-based API kwargs construction.
Add a --variant flag (openai, deepseek, qwen, mistral, llama) that
encapsulates provider-specific API differences: output token parameter
names, thinking/reasoning toggles, temperature rules, and thinking
output extraction. Add --thinking flag (off, low, medium, high) to
control reasoning effort.
Replace the three-prompt LLM scoring pipeline (kg-edge-scoring,
kg-edge-reasoning, kg-edge-selection) with a cross-encoder reranker
service backed by FlashRank. The new hop_and_filter() method performs
iterative graph traversal with semantic scoring at each hop, replacing
the previous follow_edges/get_subgraph approach.
- Add reranker service (trustgraph-base client/service, FlashRank processor)
- Add gateway dispatch for reranker via API and WebSocket
- Rewrite GraphRAG pipeline: hop_and_filter() with per-hop cross-encoder scoring
- Remove kg_prompt() and edge_score_limit from prompt client
- Update provenance: add tg:EdgeSelection type, tg:concept, tg:score predicates
- Update CLIs (tg-invoke-graph-rag, tg-show-explain-trace) for new metadata
- Add tg-invoke-reranker CLI tool
- Add tech spec and UX developer guidance
- Update all unit and integration tests
* feat: make bootstrapper initialiser timeouts configurable
DefaultFlowStart and WorkspaceInit hardcoded the request timeouts for
their flow-svc and IAM calls, leaving operators no way to tune them for
high-latency environments (#874).
Expose them as constructor parameters threaded through the existing
initialiser `params:` mechanism, defaulting to the current values so
behaviour is unchanged unless explicitly overridden:
- DefaultFlowStart: list_timeout=10 (list-flows), start_timeout=30 (start-flow)
- WorkspaceInit: iam_timeout=10 (create-workspace)
Add unit tests for the defaults, override storage, and that configured
values reach the underlying request calls.
* test: mark async bootstrap test with @pytest.mark.asyncio
Addresses review feedback on PR #999: add the explicit
@pytest.mark.asyncio decorator to test_run_forwards_configured_timeouts
so it does not rely on asyncio_mode=auto and stays consistent with the
rest of the suite.
Users are global entities, not scoped to workspaces. This change:
Track A — Global usernames:
- Change iam_users_by_username to PRIMARY KEY (username), removing
workspace from the lookup key
- Login looks up username globally, no workspace required
- Username uniqueness is enforced globally, not per-workspace
- Login -w now overrides the JWT workspace (session workspace)
rather than selecting which user registry to search
Track B — Rename workspace to default_workspace:
- UserRecord.workspace → UserRecord.default_workspace
- Identity.workspace → Identity.default_workspace
- JWT claim "workspace" → "default_workspace"
- IamResponse.resolved_workspace → resolved_default_workspace
- WebSocket auth-ok frame field → default_workspace
- Socket clients read default_workspace from auth-ok
- _user_record_to_dict wire key → default_workspace
- CLI help text and output updated throughout
- Test files updated for renamed fields
The mux was extracting the authorisation resource workspace from the
inner request body via registry extractors. But workspace-scoped
services (config, flow, librarian, etc.) receive workspace from the
queue identity, not the message body — the inner workspace field is
a dead field that no service handler reads.
This caused access-denied errors when the inner body's workspace
(e.g. CLI default "default") disagreed with the caller's assigned
workspace, even though the envelope workspace was correct.
Fix: resolve workspace from the envelope only. Split the non-flow
authorisation path by resource level — WORKSPACE ops use the envelope
workspace directly; SYSTEM ops (IAM) still use registry extractors
since they legitimately read operation-specific body fields.
Updated the README to clarify the concept of holons and the functionality of TrustGraph. Improved the structure and flow of information regarding context management and agent explainability.
Replace the broken GATEWAY_SECRET auth (token was sent as a query
parameter, silently ignored by the gateway) with end-to-end Bearer
token forwarding. Each MCP caller gets a dedicated WebSocket
authenticated via the gateway's in-band first-frame protocol, with
whoami verification on first connect.
Also fix and extend the tool surface:
- embeddings: accept list of texts (was single string)
- triples_query: use Term wire format with compact keys (was legacy
Value format), add collection and graph parameters
- sparql_query: new tool for SPARQL SELECT/ASK/CONSTRUCT/DESCRIBE
- graphql_query: new tool for structured data (rows) GraphQL queries
- all tools: add optional workspace parameter
Replace the broken GATEWAY_SECRET auth (token was sent as a query
parameter, silently ignored by the gateway) with end-to-end Bearer
token forwarding. Each MCP caller gets a dedicated WebSocket
authenticated via the gateway's in-band first-frame protocol, with
whoami verification on first connect.
Also fix and extend the tool surface:
- embeddings: accept list of texts (was single string)
- triples_query: use Term wire format with compact keys (was legacy
Value format), add collection and graph parameters
- sparql_query: new tool for SPARQL SELECT/ASK/CONSTRUCT/DESCRIBE
- graphql_query: new tool for structured data (rows) GraphQL queries
- all tools: add optional workspace parameter
The query service now uses async_execute_paged (indexed path) and
async_scan (scan path) instead of async_execute. Tests were mocking
the old function, causing them to hang indefinitely.
- Pass auth token to schema discovery and descriptor generation in
tg-load-structured-data, fixing 401 errors with IAM enabled
- Fix row query pagination: replace single-page async_execute with
async_scan that streams pages and applies filters without
materialising the full result set (OOM on large datasets)
- Add missing filter operators (not, startsWith, endsWith, not_in)
to row query post-filter matching
- Fall back to scan path when an indexed field is queried with an
empty string value, since empty index values are not stored
- Revert top-level indexes array support — the current table schema
overwrites rows with duplicate index values, so only primary_key
fields are safe to index until the schema is redesigned
resolve_cassandra_config did not accept replication_factor as a kwarg,
so cassandra_replication_factor from YAML params was silently ignored
by all 6 callers. Add the kwarg and pass it from every caller.
Same fix for Qdrant: 3 writers now pass qdrant_replication_factor and
qdrant_shard_number from params.
Add tests covering the params path for both helpers.
- Add centralised qdrant_config.py helper with env-var fallback for
QDRANT_URL, QDRANT_API_KEY, QDRANT_REPLICATION_FACTOR, QDRANT_SHARD_NUMBER
- Update all 6 Qdrant processors to use the helper; writers pass
replication_factor and shard_number to create_collection
- Fix hardcoded Cassandra replication_factor=1 in cassandra_kg.py,
write.py, and sparql_cassandra.py to respect CASSANDRA_REPLICATION_FACTOR
- Upgrade Cassandra TLS from deprecated PROTOCOL_TLSv1_2 to
ssl.create_default_context() across all connectors
The librarian now reads OBJECT_STORE_ENDPOINT, OBJECT_STORE_ACCESS_KEY,
OBJECT_STORE_SECRET_KEY, OBJECT_STORE_REGION, and OBJECT_STORE_USE_SSL
from the environment when not set via params. This lets K8s Secrets
supply credentials without them appearing in launch.yaml.
Implements all three changes from the knowledge-core-completeness tech spec:
1. Named graph field preserved through Cassandra storage (7-element tuple),
enabling provenance triples to retain their graph URIs on round-trip.
2. Provenance triples already arrive on triples-input — no routing change
needed; Change 1 was sufficient.
3. Source material (library documents) streamed alongside triples and
embeddings during core download/upload. The knowledge manager fetches
the document hierarchy from the librarian on download and recreates it
on upload, preserving the full provenance chain across instances.
The mux unconditionally called auth.authorise() for every operation,
passing capability sentinels like AUTHENTICATED ("__authenticated__")
to the IAM regime. Since no role grants "__authenticated__", the regime
denied the request — breaking whoami (and any future AUTHENTICATED-only
operation) over the WebSocket path while the HTTP endpoints worked fine.
Match the guard pattern used by iam_endpoint.py and registry_endpoint.py:
only call authorise() for real capability strings, not sentinels.
Bulk clients (sync and async) were not forwarding the workspace parameter,
causing all bulk operations to hit the default workspace regardless of the
Api instance's workspace setting. Also fixes the gateway socket endpoint to
pass query parameters (including workspace) to the dispatcher, and prevents
the auth handshake from overwriting an explicitly set workspace.
Updates knowledge table store tests for paged query interface.
- Paginate heavy Cassandra reads (triples, graph/document embeddings)
using synchronous session.execute() in run_in_executor with fetch_size
paging, preventing materialization hang on large result sets
- Fix document stream endpoint to use workspace-scoped librarian queues
- Add decoder error handling for PDF/OCR/unstructured processors
- Add WebSocket mux guards for missing auth fields
- Add null check in librarian document streaming
- Rewrite get_document_content CLI to stream via librarian
- Add Poppler dependency to unstructured container
The tests were patching
trustgraph.embeddings.hf.hf.HuggingFaceEmbeddings - a module-level
attribute that doesn't exist because HuggingFaceEmbeddings is
imported locally inside _load_model. Changed all 8 occurrences to
patch langchain_huggingface.HuggingFaceEmbeddings, which is the
actual import source the code uses at runtime.
The auth-ok response includes the token's bound workspace, and
AsyncSocketClient was unconditionally adopting it — clobbering any
workspace the caller explicitly requested via the constructor.
Several CLI commands silently routed requests to the default workspace
regardless of the -w flag: show-flows, show-flow-blueprints,
show-parameter-types, set-prompt --system, and load-structured-data.
The workspace was sent in the inner request body but not on the
WebSocket envelope or API client constructor, so the gateway always
dispatched to the default workspace queue.
