Documents and processing records without a stored time field caused a
KeyError crash in get_documents and list_children. Use optional access
so time resolves to None instead of failing.
* feat: streaming N-Quads serializer for wire-format triples
Groundwork for Phase 2 of #877 (knowledge export). Hand-rolled
N-Triples term encoding: rdflib's term.n3() emits Turtle-style forms
(numeric shorthand, unescaped newlines) that are invalid in
line-oriented N-Quads, so literals are escaped per the ECHAR grammar
and IRIs validated for representability. Round-trip tests parse the
output back with rdflib's nquads parser and compare term-for-term.
* feat: export/import workspace knowledge in .tgx bundles (#877)
Phase 2 of the workspace bundle commands: tg-export-workspace now
includes the workspace's knowledge by default — per-collection
knowledge-graph triples as N-Quads (the collection names the graph,
streamed through a tempfile so memory stays flat regardless of
knowledge-base size) and the document library (metadata plus content,
fetched one document at a time). --config-only skips knowledge on both
sides; --triples-limit bounds very large graphs; -f/--flow-id selects
the flow the triples services run through.
tg-import-workspace streams triples back through the bulk import per
collection and recreates library documents (children after parents).
Knowledge import is additive, unlike config's skip-existing semantics.
Embedding vectors are not carried in bundles: --process re-runs
imported documents through the flow, which regenerates extraction
output and embeddings; --process-collection targets it.
Round-trip covered by unit tests over real archives: export with a
mocked Api, re-import, and assert the bulk triples stream and library
add calls reproduce the original values (including datatyped literals
via the N-Quads path).
Add a complete audit event pipeline that emits structured, machine-
parseable events for every gateway request and IAM decision.
Schema and publisher:
- AuditEvent dataclass and notify-class queue (audit_events_queue)
- AuditPublisher utility: fire-and-forget emission with envelope
(schema_version, event_id, event_type, timestamp, producer)
- New request_id and client_ip fields on IamRequest for correlation
Gateway (gateway.request events):
- aiohttp middleware assigns request_id, captures timing/status/sizes
and emits an event after every HTTP request completes
- IamAuth.authenticate annotates the request with identity
- Main endpoint handlers annotate capability and workspace
- request_id and client_ip forwarded to IAM on authenticate/authorise
IAM service (iam.authenticate, iam.authorise, iam.management events):
- Emits iam.authenticate for resolve-api-key, login, anonymous auth
- Emits iam.authorise for authorise and authorise-many decisions
- Emits iam.management for user/workspace/key mutations
- All events include request_id for correlation with gateway events
Design: events land on a pub/sub notify topic — non-persistent,
per-subscriber delivery. If no audit consumer is deployed, events
are silently discarded. Storage, retention, and alerting are
consumer-side concerns outside this boundary.
Added unit tests for the publisher and gateway middleware, unit
tests for IAM audit emission, and a contract test for the AuditEvent
schema.
Tech spec: docs/tech-specs/audit-events.md
Documents and processing records without a stored time field caused a
KeyError crash in get_documents and list_children. Use optional access
so time resolves to None instead of failing.
* feat: streaming N-Quads serializer for wire-format triples
Groundwork for Phase 2 of #877 (knowledge export). Hand-rolled
N-Triples term encoding: rdflib's term.n3() emits Turtle-style forms
(numeric shorthand, unescaped newlines) that are invalid in
line-oriented N-Quads, so literals are escaped per the ECHAR grammar
and IRIs validated for representability. Round-trip tests parse the
output back with rdflib's nquads parser and compare term-for-term.
* feat: export/import workspace knowledge in .tgx bundles (#877)
Phase 2 of the workspace bundle commands: tg-export-workspace now
includes the workspace's knowledge by default — per-collection
knowledge-graph triples as N-Quads (the collection names the graph,
streamed through a tempfile so memory stays flat regardless of
knowledge-base size) and the document library (metadata plus content,
fetched one document at a time). --config-only skips knowledge on both
sides; --triples-limit bounds very large graphs; -f/--flow-id selects
the flow the triples services run through.
tg-import-workspace streams triples back through the bulk import per
collection and recreates library documents (children after parents).
Knowledge import is additive, unlike config's skip-existing semantics.
Embedding vectors are not carried in bundles: --process re-runs
imported documents through the flow, which regenerates extraction
output and embeddings; --process-collection targets it.
Round-trip covered by unit tests over real archives: export with a
mocked Api, re-import, and assert the bulk triples stream and library
add calls reproduce the original values (including datatyped literals
via the N-Quads path).
- Filter out RDF/RDFS/OWL schema predicates (rdfs:domain, owl:inverseOf,
etc.) from hop traversal, keeping rdf:type for data signal
- Skip edges where reranker-visible components are unlabeled IRIs, since
the cross-encoder cannot meaningfully score raw URIs
- Add max-reranker-input safety cap (default 350) to prevent overloading
the reranker, applied after filtering for maximum useful candidates
- Expose max-reranker-input as per-request parameter through schema,
translator, REST API, socket client, CLI, and OpenAPI spec
- Update tests
- Update tech spec
Phase 1 (config only): export a workspace's full configuration as a
portable .tgx bundle (gzipped tar with manifest.json and one
pretty-printed, self-describing JSON file per config key under
config/<type>/), and import it into another deployment or workspace.
Import defaults to WorkspaceInit's re-run semantics (existing keys kept,
missing keys added; --overwrite replaces), supports --workspace rename,
--dry-run, and --config-only, and refuses to silently drop knowledge
data from future Phase-2 bundles it cannot import yet.
Convention in the tests is to just import the libraries as production
code would. This is fragile, and could possibly be used to inject
malicious code in the CI environment.
The reranker document text now reflects the traversal direction,
showing only the new information relative to the frontier entity:
- From S (subject is frontier): text = "{predicate} {object}"
- From O (object is frontier): text = "{subject} {predicate}"
- From P (predicate is frontier): text = "{subject} {object}"
This eliminates duplicate reranker texts when traversing inward
from shared object nodes (e.g. 18 CPUs all producing identical
"hasSubcategory Processors" text when the subject was dropped).
execute_batch_triple_queries now returns (triple, direction)
tuples so hop_and_filter can select the appropriate text format.
Updates tech spec to document the direction-aware approach.
Adds unit tests for direction tracking and reranker text
construction.
Wire the FlashRank reranker subsystem from #1005 into Document-RAG: after
vector retrieval, over-fetch a wider candidate pool, rerank with the
cross-encoder, and keep the top doc_limit chunks for synthesis.
Per maintainer review, the fetch and select sizes are two caller-controlled
limits rather than one internal heuristic:
- doc_limit: chunks selected into the synthesis prompt (unchanged meaning).
- fetch_limit: candidate pool pulled from the vector store before reranking.
0 = derive (OVERFETCH_FACTOR x doc_limit); values below doc_limit are
raised to it. Lets the caller control how hard the reranker has to work.
Details:
- schema: DocumentRagQuery.fetch_limit (additive, backward compatible).
- document_rag.py / rag.py: fetch_limit resolved in the processor (mirrors
doc_limit); the core applies the heuristic default and derives synthesis
provenance from the chunk-selection focus when reranking ran.
- provenance: tg:ChunkSelection focus stage (mirrors tg:EdgeSelection).
- request translator + client SDKs + CLI: fetch-limit / --fetch-limit,
threaded exactly like doc_limit and the GraphRAG limits.
- tests: no-op identity, over-fetch/narrow, explicit fetch_limit, heuristic
default, floor-at-doc_limit, provenance lineage, cross-repo topic wiring.
Reranking is skipped byte-identically when no reranker role is wired.
Requires the companion trustgraph-templates change wiring the reranker
topics into the document-rag flow (mirrors #279 for GraphRAG).
Replace the client.post()/httpx bypass with standard SDK extra_body,
confirmed working against DashScope. Make DashScope the base variant
with Qwen as a subclass alias. Route all API calls through variant
create_completion/create_completion_stream methods.
Interactive bash installer (install_trustgraph.sh) that detects hardware,
recommends an LLM mode (OpenAI or Ollama), installs missing prerequisites
via Homebrew, sets up a Python venv, runs the test suite, generates a
deployment via npx @trustgraph/config, starts the Docker Compose stack,
health-checks the API gateway, and opens the Workbench UI.
Includes README.dev-install.md with usage documentation covering CLI
options, environment variables, LLM mode selection, non-interactive/CI
usage, uninstall, and troubleshooting. Currently macOS only.
Tests using MagicMock processors need the variant, thinking mode,
and _build_kwargs/_extract_content methods bound to work with the
new variant-based API kwargs construction.
Add a --variant flag (openai, deepseek, qwen, mistral, llama) that
encapsulates provider-specific API differences: output token parameter
names, thinking/reasoning toggles, temperature rules, and thinking
output extraction. Add --thinking flag (off, low, medium, high) to
control reasoning effort.
Replace the three-prompt LLM scoring pipeline (kg-edge-scoring,
kg-edge-reasoning, kg-edge-selection) with a cross-encoder reranker
service backed by FlashRank. The new hop_and_filter() method performs
iterative graph traversal with semantic scoring at each hop, replacing
the previous follow_edges/get_subgraph approach.
- Add reranker service (trustgraph-base client/service, FlashRank processor)
- Add gateway dispatch for reranker via API and WebSocket
- Rewrite GraphRAG pipeline: hop_and_filter() with per-hop cross-encoder scoring
- Remove kg_prompt() and edge_score_limit from prompt client
- Update provenance: add tg:EdgeSelection type, tg:concept, tg:score predicates
- Update CLIs (tg-invoke-graph-rag, tg-show-explain-trace) for new metadata
- Add tg-invoke-reranker CLI tool
- Add tech spec and UX developer guidance
- Update all unit and integration tests
* feat: make bootstrapper initialiser timeouts configurable
DefaultFlowStart and WorkspaceInit hardcoded the request timeouts for
their flow-svc and IAM calls, leaving operators no way to tune them for
high-latency environments (#874).
Expose them as constructor parameters threaded through the existing
initialiser `params:` mechanism, defaulting to the current values so
behaviour is unchanged unless explicitly overridden:
- DefaultFlowStart: list_timeout=10 (list-flows), start_timeout=30 (start-flow)
- WorkspaceInit: iam_timeout=10 (create-workspace)
Add unit tests for the defaults, override storage, and that configured
values reach the underlying request calls.
* test: mark async bootstrap test with @pytest.mark.asyncio
Addresses review feedback on PR #999: add the explicit
@pytest.mark.asyncio decorator to test_run_forwards_configured_timeouts
so it does not rely on asyncio_mode=auto and stays consistent with the
rest of the suite.
Users are global entities, not scoped to workspaces. This change:
Track A — Global usernames:
- Change iam_users_by_username to PRIMARY KEY (username), removing
workspace from the lookup key
- Login looks up username globally, no workspace required
- Username uniqueness is enforced globally, not per-workspace
- Login -w now overrides the JWT workspace (session workspace)
rather than selecting which user registry to search
Track B — Rename workspace to default_workspace:
- UserRecord.workspace → UserRecord.default_workspace
- Identity.workspace → Identity.default_workspace
- JWT claim "workspace" → "default_workspace"
- IamResponse.resolved_workspace → resolved_default_workspace
- WebSocket auth-ok frame field → default_workspace
- Socket clients read default_workspace from auth-ok
- _user_record_to_dict wire key → default_workspace
- CLI help text and output updated throughout
- Test files updated for renamed fields
The mux was extracting the authorisation resource workspace from the
inner request body via registry extractors. But workspace-scoped
services (config, flow, librarian, etc.) receive workspace from the
queue identity, not the message body — the inner workspace field is
a dead field that no service handler reads.
This caused access-denied errors when the inner body's workspace
(e.g. CLI default "default") disagreed with the caller's assigned
workspace, even though the envelope workspace was correct.
Fix: resolve workspace from the envelope only. Split the non-flow
authorisation path by resource level — WORKSPACE ops use the envelope
workspace directly; SYSTEM ops (IAM) still use registry extractors
since they legitimately read operation-specific body fields.
Updated the README to clarify the concept of holons and the functionality of TrustGraph. Improved the structure and flow of information regarding context management and agent explainability.
Replace the broken GATEWAY_SECRET auth (token was sent as a query
parameter, silently ignored by the gateway) with end-to-end Bearer
token forwarding. Each MCP caller gets a dedicated WebSocket
authenticated via the gateway's in-band first-frame protocol, with
whoami verification on first connect.
Also fix and extend the tool surface:
- embeddings: accept list of texts (was single string)
- triples_query: use Term wire format with compact keys (was legacy
Value format), add collection and graph parameters
- sparql_query: new tool for SPARQL SELECT/ASK/CONSTRUCT/DESCRIBE
- graphql_query: new tool for structured data (rows) GraphQL queries
- all tools: add optional workspace parameter
Replace the broken GATEWAY_SECRET auth (token was sent as a query
parameter, silently ignored by the gateway) with end-to-end Bearer
token forwarding. Each MCP caller gets a dedicated WebSocket
authenticated via the gateway's in-band first-frame protocol, with
whoami verification on first connect.
Also fix and extend the tool surface:
- embeddings: accept list of texts (was single string)
- triples_query: use Term wire format with compact keys (was legacy
Value format), add collection and graph parameters
- sparql_query: new tool for SPARQL SELECT/ASK/CONSTRUCT/DESCRIBE
- graphql_query: new tool for structured data (rows) GraphQL queries
- all tools: add optional workspace parameter
