Derive consumer behaviour from queue class, remove
consumer_type parameter
The queue class prefix (flow, request, response, notify) now
fully determines consumer behaviour in both RabbitMQ and Pulsar
backends. Added 'notify' class for ephemeral broadcast (config
push notifications). Response and notify classes always create
per-subscriber auto-delete queues, eliminating orphaned queues
that accumulated on service restarts.
Change init-trustgraph to set up the 'notify' namespace in
Pulsar instead of old hangover 'state'.
Fixes 'stuck backlog' on RabbitMQ config notification queue.
The triples client returns Uri/Literal (str subclasses), not Term
objects. _quoted_triple() treated all values as IRIs, so literal
objects like skos:definition values were mistyped in focus
provenance events, and trace_source_documents could not match
them in the store.
Added to_term() to convert Uri/Literal back to Term, threaded a
term_map from follow_edges_batch through
get_subgraph/get_labelgraph into uri_map, and updated
_quoted_triple to accept Term objects directly.
Store the initialization Promise in the requestors map synchronously
before yielding, so concurrent callers for the same key await the same
instance — prevents orphaned RequestResponse objects and duplicate NATS
subscriptions. Mirrors upstream fix 8f18ba02.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Subscriber resilience: recreate consumer after connection failure
- Move consumer creation from Subscriber.start() into the run() loop,
matching the pattern used by Consumer. If the connection drops and the
consumer is closed in the finally block, the loop now recreates it on
the next iteration instead of spinning forever on a None consumer.
Consumer thread safety:
- Dedicated ThreadPoolExecutor per consumer so all pika operations
(create, receive, acknowledge, negative_acknowledge) run on the
same thread — pika BlockingConnection is not thread-safe
- Applies to both Consumer and Subscriber classes
Config handler type audit — fix four mismatched type registrations:
- librarian: was ["librarian"] (non-existent type), now ["flow",
"active-flow"] (matches config["flow"] that the handler reads)
- cores/service: was ["kg-core"], now ["flow"] (reads
config["flow"])
- metering/counter: was ["token-costs"], now ["token-cost"]
(singular)
- agent/mcp_tool: was ["mcp-tool"], now ["mcp"] (reads
config["mcp"])
Update tests
Three QA iterations to convergence (zero issues remaining):
Workbench UI:
- Connection badge: amber "Connected (no auth)" for unauthenticated state
- Theme persistence: restore script in index.html + localStorage sync
- Settings About section: add bottom padding so content isn't clipped
- Clear messages: cancel in-flight requests when clearing chat
- Feature switch labels: proper casing + acronym handling (MCP, LLM)
- Token Cost badge: hidden during loading state
- ARIA: role="switch", aria-checked on toggles, aria-labels on buttons
- ConfigApi: null-safe chaining for getPrompts/getSystemPrompt
Grafana dashboards:
- Auto-refresh 30s on all 3 dashboards
- Panel heights reduced to fit viewport without scrolling
- Anonymous role upgraded to Editor for Explore access
Infrastructure:
- Nginx: DNS resolver with variable-based upstream (prevents crash loop)
- Workbench port set to 3002 in .env
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Provenance triples are now included directly in explain messages from
GraphRAG, DocumentRAG, and Agent services, eliminating the need for
follow-up knowledge graph queries to retrieve explainability details.
Each explain message in the response stream now carries:
- explain_id: root URI for this provenance step (unchanged)
- explain_graph: named graph where triples are stored (unchanged)
- explain_triples: the actual provenance triples for this step (new)
Changes across the stack:
- Schema: added explain_triples field to GraphRagResponse,
DocumentRagResponse, and AgentResponse
- Services: all explain message call sites pass triples through
(graph_rag, document_rag, agent react, agent orchestrator)
- Translators: encode explain_triples via TripleTranslator for
gateway wire format
- Python SDK: ProvenanceEvent now includes parsed ExplainEntity
and raw triples; expanded event_type detection
- CLI: invoke_graph_rag, invoke_agent, invoke_document_rag use
inline entity when available, fall back to graph query
- Tech specs updated
Additional explainability test
- Fix agent react and orchestrator services appending bare methods
to config_handlers instead of using register_config_handler() —
caused 'method object is not subscriptable' on config notify
- Add exc_info to config fetch retry logging for proper tracebacks
- Remove debug print statements from collection management
dispatcher and translator
- Disable RabbitMQ heartbeats (heartbeat=0) to prevent broker
closing idle producer connections that can't process heartbeat
frames from BlockingConnection
Add full pipeline test that generates a real PDF, processes it through
the entire pipeline, and verifies knowledge lands in FalkorDB:
- Create test PDF generator using pdf-lib (2-page doc about Acme Corp)
- Add testFullPipeline() to integration tests with store verification
- Fix FalkorDB client connect() — createClient returns unconnected client
in both TriplesStore and TriplesQuery classes
Results: PDF decoded (2 pages) → chunked (2 chunks) → extracted
(4 relationships) → 16 triples stored in FalkorDB including:
alice-johnson → is-a-senior-engineer → acme-corporation
cloudsync → uses-aws-for-hosting → amazon-web-services
provenance: pages → prov:wasDerivedFrom → source document
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two bugs found during end-to-end testing:
1. FlowProcessor never restarted flows when config changed — it only
started them once. Stale NATS JetStream data from previous sessions
caused services to bind to wrong topics. Fix: stop and restart flows
on every config push that includes flow definitions.
2. Gateway publishToTopic sent messages without an id property. Pipeline
FlowProcessor handlers check properties.id and silently return if
missing. Fix: auto-generate a message id when publishing to topics.
Both fixes validated: 13/13 integration tests passing, PDF decoder
correctly receives and processes document messages through the pipeline.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wire up the query and retrieval side of the pipeline so the agent can
answer questions from stored knowledge:
- Triples query service (FalkorDB) — all SPO pattern queries via NATS
- Graph embeddings query service (Qdrant) — entity vector similarity
- Document embeddings query service (Qdrant) — chunk vector similarity
- Graph RAG service — full concept→entity→traverse→score→synthesize pipeline
- Document RAG service — embed→find chunks→synthesize pipeline
- Runner scripts for chunker, extractor, embeddings (missing from Phase 5)
- Add DocumentEmbeddingsRequest/Response schema types
- Add RAG prompt templates (extract-concepts, edge-scoring, synthesize)
- Add graph/doc embeddings query topics to seed config + flow manager
- Add all pipeline/query/retrieval services to docker-compose
- 8 new runner scripts, 8 new pnpm script aliases
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add end-to-end document processing pipeline:
- PDF decoder service (pdfjs-dist) extracts text per page from librarian docs
- Ollama native LLM service for local model inference
- FalkorDB triples store FlowProcessor consumer
- Qdrant graph embeddings store FlowProcessor consumer
- Fix spec name collisions in chunker/extractor (input→chunk-input, etc.)
- Gateway /load endpoint to trigger document processing
- Align flow manager blueprint and seed config with full pipeline topics
- Add runner scripts and test coverage for document load
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Marks FlowProcessor and EmbeddingsService constructors as protected
since these classes should only be instantiated via subclasses.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the config push mechanism that broadcast the full config
blob on a 'state' class pub/sub queue with a lightweight notify
signal containing only the version number and affected config
types. Processors fetch the full config via request/response from
the config service when notified.
This eliminates the need for the pub/sub 'state' queue class and
stateful pub/sub services entirely. The config push queue moves
from 'state' to 'flow' class — a simple transient signal rather
than a retained message. This solves the RabbitMQ
late-subscriber problem where restarting processes never received
the current config because their fresh queue had no historical
messages.
Key changes:
- ConfigPush schema: config dict replaced with types list
- Subscribe-then-fetch startup with retry: processors subscribe
to notify queue, fetch config via request/response, then
process buffered notifies with version comparison to avoid race
conditions
- register_config_handler() accepts optional types parameter so
handlers only fire when their config types change
- Short-lived config request/response clients to avoid subscriber
contention on non-persistent response topics
- Config service passes affected types through put/delete/flow
operations
- Gateway ConfigReceiver rewritten with same notify pattern and
retry loop
Tests updated
New tests:
- register_config_handler: without types, with types, multiple
types, multiple handlers
- on_config_notify: old/same version skipped, irrelevant types
skipped (version still updated), relevant type triggers fetch,
handler without types always called, mixed handler filtering,
empty types invokes all, fetch failure handled gracefully
- fetch_config: returns config+version, raises on error response,
stops client even on exception
- fetch_and_apply_config: applies to all handlers on startup,
retries on failure
* fix: prevent duplicate dispatcher creation race condition in invoke_global_service
Concurrent coroutines could all pass the `if key in self.dispatchers` check
before any of them wrote the result back, because `await dispatcher.start()`
yields to the event loop. This caused multiple Pulsar consumers to be created
on the same shared subscription, distributing responses round-robin and
dropping ~2/3 of them — manifesting as a permanent spinner in the Workbench UI.
Apply a double-checked asyncio.Lock in both `invoke_global_service` and
`invoke_flow_service` so only one dispatcher is ever created per service key.
* test: add concurrent-dispatch tests for race condition fix
Add asyncio.gather-based tests that verify invoke_global_service and
invoke_flow_service create exactly one dispatcher under concurrent calls,
preventing the duplicate Pulsar consumer bug.
* fix: prevent duplicate dispatcher creation race condition in invoke_global_service
Concurrent coroutines could all pass the `if key in self.dispatchers` check
before any of them wrote the result back, because `await dispatcher.start()`
yields to the event loop. This caused multiple Pulsar consumers to be created
on the same shared subscription, distributing responses round-robin and
dropping ~2/3 of them — manifesting as a permanent spinner in the Workbench UI.
Apply a double-checked asyncio.Lock in both `invoke_global_service` and
`invoke_flow_service` so only one dispatcher is ever created per service key.
* test: add concurrent-dispatch tests for race condition fix
Add asyncio.gather-based tests that verify invoke_global_service and
invoke_flow_service create exactly one dispatcher under concurrent calls,
preventing the duplicate Pulsar consumer bug.
Flow Management Service:
- FlowManagerService (AsyncProcessor) handling list/get/start/stop flows
and list/get blueprints via kebab-case wire format
- Default blueprint with all service topic mappings
- Pushes flow config to config service on start/stop
Config Seeding:
- seed-config.ts script pushes prompt templates (extract-relationships,
extract-definitions, document-prompt, kg-prompt) and default flow
definition via gateway REST API
Integration Tests:
- Librarian CRUD: add-document, list-documents, get-content, delete
- Agent query: verifies routing through gateway to agent service
- Skip flags: SKIP_LIBRARIAN=1, SKIP_AGENT=1
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Multi-stage Containerfile for all Node.js services (single image,
different CMD per docker-compose service). ESM entrypoints for gateway,
config, text-completion, prompt, embeddings, agent, and librarian.
Workbench gets a separate Containerfile (nginx:alpine) with SPA routing
and API/WebSocket proxy to gateway.
Docker Compose updated with 6 app services (gateway, config-service,
text-completion, prompt, embeddings, workbench) using shared
trustgraph-ts:local image.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix three critical bugs preventing the NATS message pipeline from working:
- FlowProcessor now subscribes to config-push topic (was missing entirely),
using DeliverPolicy.All to replay config on service restart
- NATS streams use wildcard subjects (tg.flow.>) instead of per-topic
narrow filters that caused 503 errors on publish
- Subscriber dispatch loop has exponential backoff on errors to prevent
tight error loops
Add service runner scripts (gateway, config, LLM) and a 7-test
integration suite that verifies config CRUD, WebSocket round-trip,
and full LLM text-completion through the NATS pipeline.
Fix Docker Compose infra: pin Tempo to v2.6.1, remove deprecated Loki
config fields, add user:0 for volume permissions, remap conflicting
ports (FalkorDB 6380, OTLP 4327/4328).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Minor fixes from linter: readonly modifiers, unused parameter prefixes,
type narrowing in graph-rag BFS traversal and edge scoring.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
