2025-06-20 16:59:55 +01:00
|
|
|
from typing import Dict, Any, Tuple, Optional
|
|
|
|
|
from ...schema import (
|
2026-03-11 10:51:39 +00:00
|
|
|
KnowledgeRequest, KnowledgeResponse, Triples, GraphEmbeddings,
|
2025-06-20 16:59:55 +01:00
|
|
|
Metadata, EntityEmbeddings
|
|
|
|
|
)
|
|
|
|
|
from .base import MessageTranslator
|
|
|
|
|
from .primitives import ValueTranslator, SubgraphTranslator
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
class KnowledgeRequestTranslator(MessageTranslator):
|
|
|
|
|
"""Translator for KnowledgeRequest schema objects"""
|
2026-03-11 10:51:39 +00:00
|
|
|
|
2025-06-20 16:59:55 +01:00
|
|
|
def __init__(self):
|
|
|
|
|
self.value_translator = ValueTranslator()
|
|
|
|
|
self.subgraph_translator = SubgraphTranslator()
|
2026-03-11 10:51:39 +00:00
|
|
|
|
Pub/sub abstraction: decouple from Pulsar (#751)
Remove Pulsar-specific concepts from application code so that
the pub/sub backend is swappable via configuration.
Rename translators:
- to_pulsar/from_pulsar → decode/encode across all translator
classes, dispatch handlers, and tests (55+ files)
- from_response_with_completion → encode_with_completion
- Remove pulsar.schema.Record from translator base class
Queue naming (CLASS:TOPICSPACE:TOPIC):
- Replace topic() helper with queue() using new format:
flow:tg:name, request:tg:name, response:tg:name, state:tg:name
- Queue class implies persistence/TTL (no QoS in names)
- Update Pulsar backend map_topic() to parse new format
- Librarian queues use flow class (persistent, for chunking)
- Config push uses state class (persistent, last-value)
- Remove 15 dead topic imports from schema files
- Update init_trustgraph.py namespace: config → state
Confine Pulsar to pulsar_backend.py:
- Delete legacy PulsarClient class from pubsub.py
- Move add_args to add_pubsub_args() with standalone flag
for CLI tools (defaults to localhost)
- PulsarBackendConsumer.receive() catches _pulsar.Timeout,
raises standard TimeoutError
- Remove Pulsar imports from: async_processor, flow_processor,
log_level, all 11 client files, 4 storage writers, gateway
service, gateway config receiver
- Remove log_level/LoggerLevel from client API
- Rewrite tg-monitor-prompts to use backend abstraction
- Update tg-dump-queues to use add_pubsub_args
Also: pubsub-abstraction.md tech spec covering problem statement,
design goals, as-is requirements, candidate broker assessment,
approach, and implementation order.
2026-04-01 20:16:53 +01:00
|
|
|
def decode(self, data: Dict[str, Any]) -> KnowledgeRequest:
|
2025-06-20 16:59:55 +01:00
|
|
|
triples = None
|
|
|
|
|
if "triples" in data:
|
|
|
|
|
triples = Triples(
|
|
|
|
|
metadata=Metadata(
|
|
|
|
|
id=data["triples"]["metadata"]["id"],
|
2026-03-11 12:16:39 +00:00
|
|
|
root=data["triples"]["metadata"].get("root", ""),
|
2025-06-20 16:59:55 +01:00
|
|
|
collection=data["triples"]["metadata"]["collection"]
|
|
|
|
|
),
|
Pub/sub abstraction: decouple from Pulsar (#751)
Remove Pulsar-specific concepts from application code so that
the pub/sub backend is swappable via configuration.
Rename translators:
- to_pulsar/from_pulsar → decode/encode across all translator
classes, dispatch handlers, and tests (55+ files)
- from_response_with_completion → encode_with_completion
- Remove pulsar.schema.Record from translator base class
Queue naming (CLASS:TOPICSPACE:TOPIC):
- Replace topic() helper with queue() using new format:
flow:tg:name, request:tg:name, response:tg:name, state:tg:name
- Queue class implies persistence/TTL (no QoS in names)
- Update Pulsar backend map_topic() to parse new format
- Librarian queues use flow class (persistent, for chunking)
- Config push uses state class (persistent, last-value)
- Remove 15 dead topic imports from schema files
- Update init_trustgraph.py namespace: config → state
Confine Pulsar to pulsar_backend.py:
- Delete legacy PulsarClient class from pubsub.py
- Move add_args to add_pubsub_args() with standalone flag
for CLI tools (defaults to localhost)
- PulsarBackendConsumer.receive() catches _pulsar.Timeout,
raises standard TimeoutError
- Remove Pulsar imports from: async_processor, flow_processor,
log_level, all 11 client files, 4 storage writers, gateway
service, gateway config receiver
- Remove log_level/LoggerLevel from client API
- Rewrite tg-monitor-prompts to use backend abstraction
- Update tg-dump-queues to use add_pubsub_args
Also: pubsub-abstraction.md tech spec covering problem statement,
design goals, as-is requirements, candidate broker assessment,
approach, and implementation order.
2026-04-01 20:16:53 +01:00
|
|
|
triples=self.subgraph_translator.decode(data["triples"]["triples"]),
|
2025-06-20 16:59:55 +01:00
|
|
|
)
|
2026-03-11 10:51:39 +00:00
|
|
|
|
2025-06-20 16:59:55 +01:00
|
|
|
graph_embeddings = None
|
|
|
|
|
if "graph-embeddings" in data:
|
|
|
|
|
graph_embeddings = GraphEmbeddings(
|
|
|
|
|
metadata=Metadata(
|
|
|
|
|
id=data["graph-embeddings"]["metadata"]["id"],
|
2026-03-11 12:16:39 +00:00
|
|
|
root=data["graph-embeddings"]["metadata"].get("root", ""),
|
2025-06-20 16:59:55 +01:00
|
|
|
collection=data["graph-embeddings"]["metadata"]["collection"]
|
|
|
|
|
),
|
|
|
|
|
entities=[
|
|
|
|
|
EntityEmbeddings(
|
Pub/sub abstraction: decouple from Pulsar (#751)
Remove Pulsar-specific concepts from application code so that
the pub/sub backend is swappable via configuration.
Rename translators:
- to_pulsar/from_pulsar → decode/encode across all translator
classes, dispatch handlers, and tests (55+ files)
- from_response_with_completion → encode_with_completion
- Remove pulsar.schema.Record from translator base class
Queue naming (CLASS:TOPICSPACE:TOPIC):
- Replace topic() helper with queue() using new format:
flow:tg:name, request:tg:name, response:tg:name, state:tg:name
- Queue class implies persistence/TTL (no QoS in names)
- Update Pulsar backend map_topic() to parse new format
- Librarian queues use flow class (persistent, for chunking)
- Config push uses state class (persistent, last-value)
- Remove 15 dead topic imports from schema files
- Update init_trustgraph.py namespace: config → state
Confine Pulsar to pulsar_backend.py:
- Delete legacy PulsarClient class from pubsub.py
- Move add_args to add_pubsub_args() with standalone flag
for CLI tools (defaults to localhost)
- PulsarBackendConsumer.receive() catches _pulsar.Timeout,
raises standard TimeoutError
- Remove Pulsar imports from: async_processor, flow_processor,
log_level, all 11 client files, 4 storage writers, gateway
service, gateway config receiver
- Remove log_level/LoggerLevel from client API
- Rewrite tg-monitor-prompts to use backend abstraction
- Update tg-dump-queues to use add_pubsub_args
Also: pubsub-abstraction.md tech spec covering problem statement,
design goals, as-is requirements, candidate broker assessment,
approach, and implementation order.
2026-04-01 20:16:53 +01:00
|
|
|
entity=self.value_translator.decode(ent["entity"]),
|
Fix Metadata/EntityEmbeddings schema migration tail and add regression tests (#777)
The Metadata dataclass dropped its `metadata: list[Triple]` field
and EntityEmbeddings/ChunkEmbeddings settled on a singular
`vector: list[float]` field, but several call sites kept passing
`Metadata(metadata=...)` and `EntityEmbeddings(vectors=...)`. The
bugs were latent until a websocket client first hit
`/api/v1/flow/default/import/entity-contexts`, at which point the
dispatcher TypeError'd on construction.
Production fixes (5 call sites on the same migration tail):
* trustgraph-flow gateway dispatchers entity_contexts_import.py
and graph_embeddings_import.py — drop the stale
Metadata(metadata=...) kwarg; switch graph_embeddings_import
to the singular `vector` wire key.
* trustgraph-base messaging translators knowledge.py and
document_loading.py — fix decode side to read the singular
`"vector"` key, matching what their own encode sides have
always written.
* trustgraph-flow tables/knowledge.py — fix Cassandra row
deserialiser to construct EntityEmbeddings(vector=...)
instead of vectors=.
* trustgraph-flow gateway core_import/core_export — switch the
kg-core msgpack wire format to the singular `"v"`/`"vector"`
key and drop the dead `m["m"]` envelope field that referenced
the removed Metadata.metadata triples list (it was a
guaranteed KeyError on the export side).
Defense-in-depth regression coverage (32 new tests across 7 files):
* tests/contract/test_schema_field_contracts.py — pin the field
set of Metadata, EntityEmbeddings, ChunkEmbeddings,
EntityContext so any future schema rename fails CI loudly
with a clear diff.
* tests/unit/test_translators/test_knowledge_translator_roundtrip.py
and test_document_embeddings_translator_roundtrip.py -
encode→decode round-trip the affected translators end to end,
locking in the singular `"vector"` wire key.
* tests/unit/test_gateway/test_entity_contexts_import_dispatcher.py
and test_graph_embeddings_import_dispatcher.py — exercise the
websocket dispatchers' receive() path with realistic
payloads, the direct regression test for the original
production crash.
* tests/unit/test_gateway/test_core_import_export_roundtrip.py
— pack/unpack the kg-core msgpack format through the real
dispatcher classes (with KnowledgeRequestor mocked),
including a full export→import round-trip.
* tests/unit/test_tables/test_knowledge_table_store.py —
exercise the Cassandra row → schema conversion via __new__ to
bypass the live cluster connection.
Also fixes an unrelated leaked-coroutine RuntimeWarning in
test_gateway/test_service.py::test_run_method_calls_web_run_app: the
mocked aiohttp.web.run_app now closes the coroutine that Api.run() hands
it, mirroring what the real run_app would do, instead of leaving it for
the GC to complain about.
2026-04-10 20:43:45 +01:00
|
|
|
vector=ent["vector"],
|
2025-06-20 16:59:55 +01:00
|
|
|
)
|
|
|
|
|
for ent in data["graph-embeddings"]["entities"]
|
|
|
|
|
]
|
|
|
|
|
)
|
2026-03-11 10:51:39 +00:00
|
|
|
|
2025-06-20 16:59:55 +01:00
|
|
|
return KnowledgeRequest(
|
|
|
|
|
operation=data.get("operation"),
|
|
|
|
|
id=data.get("id"),
|
|
|
|
|
flow=data.get("flow"),
|
|
|
|
|
collection=data.get("collection"),
|
|
|
|
|
triples=triples,
|
|
|
|
|
graph_embeddings=graph_embeddings,
|
|
|
|
|
)
|
2026-03-11 10:51:39 +00:00
|
|
|
|
Pub/sub abstraction: decouple from Pulsar (#751)
Remove Pulsar-specific concepts from application code so that
the pub/sub backend is swappable via configuration.
Rename translators:
- to_pulsar/from_pulsar → decode/encode across all translator
classes, dispatch handlers, and tests (55+ files)
- from_response_with_completion → encode_with_completion
- Remove pulsar.schema.Record from translator base class
Queue naming (CLASS:TOPICSPACE:TOPIC):
- Replace topic() helper with queue() using new format:
flow:tg:name, request:tg:name, response:tg:name, state:tg:name
- Queue class implies persistence/TTL (no QoS in names)
- Update Pulsar backend map_topic() to parse new format
- Librarian queues use flow class (persistent, for chunking)
- Config push uses state class (persistent, last-value)
- Remove 15 dead topic imports from schema files
- Update init_trustgraph.py namespace: config → state
Confine Pulsar to pulsar_backend.py:
- Delete legacy PulsarClient class from pubsub.py
- Move add_args to add_pubsub_args() with standalone flag
for CLI tools (defaults to localhost)
- PulsarBackendConsumer.receive() catches _pulsar.Timeout,
raises standard TimeoutError
- Remove Pulsar imports from: async_processor, flow_processor,
log_level, all 11 client files, 4 storage writers, gateway
service, gateway config receiver
- Remove log_level/LoggerLevel from client API
- Rewrite tg-monitor-prompts to use backend abstraction
- Update tg-dump-queues to use add_pubsub_args
Also: pubsub-abstraction.md tech spec covering problem statement,
design goals, as-is requirements, candidate broker assessment,
approach, and implementation order.
2026-04-01 20:16:53 +01:00
|
|
|
def encode(self, obj: KnowledgeRequest) -> Dict[str, Any]:
|
2025-06-20 16:59:55 +01:00
|
|
|
result = {}
|
2026-03-11 10:51:39 +00:00
|
|
|
|
2025-06-20 16:59:55 +01:00
|
|
|
if obj.operation:
|
|
|
|
|
result["operation"] = obj.operation
|
|
|
|
|
if obj.id:
|
|
|
|
|
result["id"] = obj.id
|
|
|
|
|
if obj.flow:
|
|
|
|
|
result["flow"] = obj.flow
|
|
|
|
|
if obj.collection:
|
|
|
|
|
result["collection"] = obj.collection
|
2026-03-11 10:51:39 +00:00
|
|
|
|
2025-06-20 16:59:55 +01:00
|
|
|
if obj.triples:
|
|
|
|
|
result["triples"] = {
|
|
|
|
|
"metadata": {
|
|
|
|
|
"id": obj.triples.metadata.id,
|
2026-03-11 12:16:39 +00:00
|
|
|
"root": obj.triples.metadata.root,
|
2025-06-20 16:59:55 +01:00
|
|
|
"collection": obj.triples.metadata.collection,
|
|
|
|
|
},
|
Pub/sub abstraction: decouple from Pulsar (#751)
Remove Pulsar-specific concepts from application code so that
the pub/sub backend is swappable via configuration.
Rename translators:
- to_pulsar/from_pulsar → decode/encode across all translator
classes, dispatch handlers, and tests (55+ files)
- from_response_with_completion → encode_with_completion
- Remove pulsar.schema.Record from translator base class
Queue naming (CLASS:TOPICSPACE:TOPIC):
- Replace topic() helper with queue() using new format:
flow:tg:name, request:tg:name, response:tg:name, state:tg:name
- Queue class implies persistence/TTL (no QoS in names)
- Update Pulsar backend map_topic() to parse new format
- Librarian queues use flow class (persistent, for chunking)
- Config push uses state class (persistent, last-value)
- Remove 15 dead topic imports from schema files
- Update init_trustgraph.py namespace: config → state
Confine Pulsar to pulsar_backend.py:
- Delete legacy PulsarClient class from pubsub.py
- Move add_args to add_pubsub_args() with standalone flag
for CLI tools (defaults to localhost)
- PulsarBackendConsumer.receive() catches _pulsar.Timeout,
raises standard TimeoutError
- Remove Pulsar imports from: async_processor, flow_processor,
log_level, all 11 client files, 4 storage writers, gateway
service, gateway config receiver
- Remove log_level/LoggerLevel from client API
- Rewrite tg-monitor-prompts to use backend abstraction
- Update tg-dump-queues to use add_pubsub_args
Also: pubsub-abstraction.md tech spec covering problem statement,
design goals, as-is requirements, candidate broker assessment,
approach, and implementation order.
2026-04-01 20:16:53 +01:00
|
|
|
"triples": self.subgraph_translator.encode(obj.triples.triples),
|
2025-06-20 16:59:55 +01:00
|
|
|
}
|
2026-03-11 10:51:39 +00:00
|
|
|
|
2025-06-20 16:59:55 +01:00
|
|
|
if obj.graph_embeddings:
|
|
|
|
|
result["graph-embeddings"] = {
|
|
|
|
|
"metadata": {
|
|
|
|
|
"id": obj.graph_embeddings.metadata.id,
|
2026-03-11 12:16:39 +00:00
|
|
|
"root": obj.graph_embeddings.metadata.root,
|
2025-06-20 16:59:55 +01:00
|
|
|
"collection": obj.graph_embeddings.metadata.collection,
|
|
|
|
|
},
|
|
|
|
|
"entities": [
|
|
|
|
|
{
|
2026-03-10 13:28:16 +00:00
|
|
|
"vector": entity.vector,
|
Pub/sub abstraction: decouple from Pulsar (#751)
Remove Pulsar-specific concepts from application code so that
the pub/sub backend is swappable via configuration.
Rename translators:
- to_pulsar/from_pulsar → decode/encode across all translator
classes, dispatch handlers, and tests (55+ files)
- from_response_with_completion → encode_with_completion
- Remove pulsar.schema.Record from translator base class
Queue naming (CLASS:TOPICSPACE:TOPIC):
- Replace topic() helper with queue() using new format:
flow:tg:name, request:tg:name, response:tg:name, state:tg:name
- Queue class implies persistence/TTL (no QoS in names)
- Update Pulsar backend map_topic() to parse new format
- Librarian queues use flow class (persistent, for chunking)
- Config push uses state class (persistent, last-value)
- Remove 15 dead topic imports from schema files
- Update init_trustgraph.py namespace: config → state
Confine Pulsar to pulsar_backend.py:
- Delete legacy PulsarClient class from pubsub.py
- Move add_args to add_pubsub_args() with standalone flag
for CLI tools (defaults to localhost)
- PulsarBackendConsumer.receive() catches _pulsar.Timeout,
raises standard TimeoutError
- Remove Pulsar imports from: async_processor, flow_processor,
log_level, all 11 client files, 4 storage writers, gateway
service, gateway config receiver
- Remove log_level/LoggerLevel from client API
- Rewrite tg-monitor-prompts to use backend abstraction
- Update tg-dump-queues to use add_pubsub_args
Also: pubsub-abstraction.md tech spec covering problem statement,
design goals, as-is requirements, candidate broker assessment,
approach, and implementation order.
2026-04-01 20:16:53 +01:00
|
|
|
"entity": self.value_translator.encode(entity.entity),
|
2025-06-20 16:59:55 +01:00
|
|
|
}
|
|
|
|
|
for entity in obj.graph_embeddings.entities
|
|
|
|
|
],
|
|
|
|
|
}
|
2026-03-11 10:51:39 +00:00
|
|
|
|
2025-06-20 16:59:55 +01:00
|
|
|
return result
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
class KnowledgeResponseTranslator(MessageTranslator):
|
|
|
|
|
"""Translator for KnowledgeResponse schema objects"""
|
2026-03-11 10:51:39 +00:00
|
|
|
|
2025-06-20 16:59:55 +01:00
|
|
|
def __init__(self):
|
|
|
|
|
self.value_translator = ValueTranslator()
|
|
|
|
|
self.subgraph_translator = SubgraphTranslator()
|
2026-03-11 10:51:39 +00:00
|
|
|
|
Pub/sub abstraction: decouple from Pulsar (#751)
Remove Pulsar-specific concepts from application code so that
the pub/sub backend is swappable via configuration.
Rename translators:
- to_pulsar/from_pulsar → decode/encode across all translator
classes, dispatch handlers, and tests (55+ files)
- from_response_with_completion → encode_with_completion
- Remove pulsar.schema.Record from translator base class
Queue naming (CLASS:TOPICSPACE:TOPIC):
- Replace topic() helper with queue() using new format:
flow:tg:name, request:tg:name, response:tg:name, state:tg:name
- Queue class implies persistence/TTL (no QoS in names)
- Update Pulsar backend map_topic() to parse new format
- Librarian queues use flow class (persistent, for chunking)
- Config push uses state class (persistent, last-value)
- Remove 15 dead topic imports from schema files
- Update init_trustgraph.py namespace: config → state
Confine Pulsar to pulsar_backend.py:
- Delete legacy PulsarClient class from pubsub.py
- Move add_args to add_pubsub_args() with standalone flag
for CLI tools (defaults to localhost)
- PulsarBackendConsumer.receive() catches _pulsar.Timeout,
raises standard TimeoutError
- Remove Pulsar imports from: async_processor, flow_processor,
log_level, all 11 client files, 4 storage writers, gateway
service, gateway config receiver
- Remove log_level/LoggerLevel from client API
- Rewrite tg-monitor-prompts to use backend abstraction
- Update tg-dump-queues to use add_pubsub_args
Also: pubsub-abstraction.md tech spec covering problem statement,
design goals, as-is requirements, candidate broker assessment,
approach, and implementation order.
2026-04-01 20:16:53 +01:00
|
|
|
def decode(self, data: Dict[str, Any]) -> KnowledgeResponse:
|
2025-06-20 16:59:55 +01:00
|
|
|
raise NotImplementedError("Response translation to Pulsar not typically needed")
|
2026-03-11 10:51:39 +00:00
|
|
|
|
Pub/sub abstraction: decouple from Pulsar (#751)
Remove Pulsar-specific concepts from application code so that
the pub/sub backend is swappable via configuration.
Rename translators:
- to_pulsar/from_pulsar → decode/encode across all translator
classes, dispatch handlers, and tests (55+ files)
- from_response_with_completion → encode_with_completion
- Remove pulsar.schema.Record from translator base class
Queue naming (CLASS:TOPICSPACE:TOPIC):
- Replace topic() helper with queue() using new format:
flow:tg:name, request:tg:name, response:tg:name, state:tg:name
- Queue class implies persistence/TTL (no QoS in names)
- Update Pulsar backend map_topic() to parse new format
- Librarian queues use flow class (persistent, for chunking)
- Config push uses state class (persistent, last-value)
- Remove 15 dead topic imports from schema files
- Update init_trustgraph.py namespace: config → state
Confine Pulsar to pulsar_backend.py:
- Delete legacy PulsarClient class from pubsub.py
- Move add_args to add_pubsub_args() with standalone flag
for CLI tools (defaults to localhost)
- PulsarBackendConsumer.receive() catches _pulsar.Timeout,
raises standard TimeoutError
- Remove Pulsar imports from: async_processor, flow_processor,
log_level, all 11 client files, 4 storage writers, gateway
service, gateway config receiver
- Remove log_level/LoggerLevel from client API
- Rewrite tg-monitor-prompts to use backend abstraction
- Update tg-dump-queues to use add_pubsub_args
Also: pubsub-abstraction.md tech spec covering problem statement,
design goals, as-is requirements, candidate broker assessment,
approach, and implementation order.
2026-04-01 20:16:53 +01:00
|
|
|
def encode(self, obj: KnowledgeResponse) -> Dict[str, Any]:
|
2025-06-20 16:59:55 +01:00
|
|
|
# Response to list operation
|
|
|
|
|
if obj.ids is not None:
|
|
|
|
|
return {"ids": obj.ids}
|
2026-03-11 10:51:39 +00:00
|
|
|
|
2025-06-20 16:59:55 +01:00
|
|
|
# Streaming triples response
|
|
|
|
|
if obj.triples:
|
|
|
|
|
return {
|
|
|
|
|
"triples": {
|
|
|
|
|
"metadata": {
|
|
|
|
|
"id": obj.triples.metadata.id,
|
2026-03-11 12:16:39 +00:00
|
|
|
"root": obj.triples.metadata.root,
|
2025-06-20 16:59:55 +01:00
|
|
|
"collection": obj.triples.metadata.collection,
|
|
|
|
|
},
|
Pub/sub abstraction: decouple from Pulsar (#751)
Remove Pulsar-specific concepts from application code so that
the pub/sub backend is swappable via configuration.
Rename translators:
- to_pulsar/from_pulsar → decode/encode across all translator
classes, dispatch handlers, and tests (55+ files)
- from_response_with_completion → encode_with_completion
- Remove pulsar.schema.Record from translator base class
Queue naming (CLASS:TOPICSPACE:TOPIC):
- Replace topic() helper with queue() using new format:
flow:tg:name, request:tg:name, response:tg:name, state:tg:name
- Queue class implies persistence/TTL (no QoS in names)
- Update Pulsar backend map_topic() to parse new format
- Librarian queues use flow class (persistent, for chunking)
- Config push uses state class (persistent, last-value)
- Remove 15 dead topic imports from schema files
- Update init_trustgraph.py namespace: config → state
Confine Pulsar to pulsar_backend.py:
- Delete legacy PulsarClient class from pubsub.py
- Move add_args to add_pubsub_args() with standalone flag
for CLI tools (defaults to localhost)
- PulsarBackendConsumer.receive() catches _pulsar.Timeout,
raises standard TimeoutError
- Remove Pulsar imports from: async_processor, flow_processor,
log_level, all 11 client files, 4 storage writers, gateway
service, gateway config receiver
- Remove log_level/LoggerLevel from client API
- Rewrite tg-monitor-prompts to use backend abstraction
- Update tg-dump-queues to use add_pubsub_args
Also: pubsub-abstraction.md tech spec covering problem statement,
design goals, as-is requirements, candidate broker assessment,
approach, and implementation order.
2026-04-01 20:16:53 +01:00
|
|
|
"triples": self.subgraph_translator.encode(obj.triples.triples),
|
2025-06-20 16:59:55 +01:00
|
|
|
}
|
|
|
|
|
}
|
2026-03-11 10:51:39 +00:00
|
|
|
|
2025-06-20 16:59:55 +01:00
|
|
|
# Streaming graph embeddings response
|
|
|
|
|
if obj.graph_embeddings:
|
|
|
|
|
return {
|
|
|
|
|
"graph-embeddings": {
|
|
|
|
|
"metadata": {
|
|
|
|
|
"id": obj.graph_embeddings.metadata.id,
|
2026-03-11 12:16:39 +00:00
|
|
|
"root": obj.graph_embeddings.metadata.root,
|
2025-06-20 16:59:55 +01:00
|
|
|
"collection": obj.graph_embeddings.metadata.collection,
|
|
|
|
|
},
|
|
|
|
|
"entities": [
|
|
|
|
|
{
|
2026-03-10 13:28:16 +00:00
|
|
|
"vector": entity.vector,
|
Pub/sub abstraction: decouple from Pulsar (#751)
Remove Pulsar-specific concepts from application code so that
the pub/sub backend is swappable via configuration.
Rename translators:
- to_pulsar/from_pulsar → decode/encode across all translator
classes, dispatch handlers, and tests (55+ files)
- from_response_with_completion → encode_with_completion
- Remove pulsar.schema.Record from translator base class
Queue naming (CLASS:TOPICSPACE:TOPIC):
- Replace topic() helper with queue() using new format:
flow:tg:name, request:tg:name, response:tg:name, state:tg:name
- Queue class implies persistence/TTL (no QoS in names)
- Update Pulsar backend map_topic() to parse new format
- Librarian queues use flow class (persistent, for chunking)
- Config push uses state class (persistent, last-value)
- Remove 15 dead topic imports from schema files
- Update init_trustgraph.py namespace: config → state
Confine Pulsar to pulsar_backend.py:
- Delete legacy PulsarClient class from pubsub.py
- Move add_args to add_pubsub_args() with standalone flag
for CLI tools (defaults to localhost)
- PulsarBackendConsumer.receive() catches _pulsar.Timeout,
raises standard TimeoutError
- Remove Pulsar imports from: async_processor, flow_processor,
log_level, all 11 client files, 4 storage writers, gateway
service, gateway config receiver
- Remove log_level/LoggerLevel from client API
- Rewrite tg-monitor-prompts to use backend abstraction
- Update tg-dump-queues to use add_pubsub_args
Also: pubsub-abstraction.md tech spec covering problem statement,
design goals, as-is requirements, candidate broker assessment,
approach, and implementation order.
2026-04-01 20:16:53 +01:00
|
|
|
"entity": self.value_translator.encode(entity.entity),
|
2025-06-20 16:59:55 +01:00
|
|
|
}
|
|
|
|
|
for entity in obj.graph_embeddings.entities
|
|
|
|
|
],
|
|
|
|
|
}
|
|
|
|
|
}
|
2026-03-11 10:51:39 +00:00
|
|
|
|
2025-06-20 16:59:55 +01:00
|
|
|
# End of stream marker
|
|
|
|
|
if obj.eos is True:
|
|
|
|
|
return {"eos": True}
|
2026-03-11 10:51:39 +00:00
|
|
|
|
2025-06-20 16:59:55 +01:00
|
|
|
# Empty response (successful delete)
|
|
|
|
|
return {}
|
|
|
|
|
|
Pub/sub abstraction: decouple from Pulsar (#751)
Remove Pulsar-specific concepts from application code so that
the pub/sub backend is swappable via configuration.
Rename translators:
- to_pulsar/from_pulsar → decode/encode across all translator
classes, dispatch handlers, and tests (55+ files)
- from_response_with_completion → encode_with_completion
- Remove pulsar.schema.Record from translator base class
Queue naming (CLASS:TOPICSPACE:TOPIC):
- Replace topic() helper with queue() using new format:
flow:tg:name, request:tg:name, response:tg:name, state:tg:name
- Queue class implies persistence/TTL (no QoS in names)
- Update Pulsar backend map_topic() to parse new format
- Librarian queues use flow class (persistent, for chunking)
- Config push uses state class (persistent, last-value)
- Remove 15 dead topic imports from schema files
- Update init_trustgraph.py namespace: config → state
Confine Pulsar to pulsar_backend.py:
- Delete legacy PulsarClient class from pubsub.py
- Move add_args to add_pubsub_args() with standalone flag
for CLI tools (defaults to localhost)
- PulsarBackendConsumer.receive() catches _pulsar.Timeout,
raises standard TimeoutError
- Remove Pulsar imports from: async_processor, flow_processor,
log_level, all 11 client files, 4 storage writers, gateway
service, gateway config receiver
- Remove log_level/LoggerLevel from client API
- Rewrite tg-monitor-prompts to use backend abstraction
- Update tg-dump-queues to use add_pubsub_args
Also: pubsub-abstraction.md tech spec covering problem statement,
design goals, as-is requirements, candidate broker assessment,
approach, and implementation order.
2026-04-01 20:16:53 +01:00
|
|
|
def encode_with_completion(self, obj: KnowledgeResponse) -> Tuple[Dict[str, Any], bool]:
|
2025-06-20 16:59:55 +01:00
|
|
|
"""Returns (response_dict, is_final)"""
|
Pub/sub abstraction: decouple from Pulsar (#751)
Remove Pulsar-specific concepts from application code so that
the pub/sub backend is swappable via configuration.
Rename translators:
- to_pulsar/from_pulsar → decode/encode across all translator
classes, dispatch handlers, and tests (55+ files)
- from_response_with_completion → encode_with_completion
- Remove pulsar.schema.Record from translator base class
Queue naming (CLASS:TOPICSPACE:TOPIC):
- Replace topic() helper with queue() using new format:
flow:tg:name, request:tg:name, response:tg:name, state:tg:name
- Queue class implies persistence/TTL (no QoS in names)
- Update Pulsar backend map_topic() to parse new format
- Librarian queues use flow class (persistent, for chunking)
- Config push uses state class (persistent, last-value)
- Remove 15 dead topic imports from schema files
- Update init_trustgraph.py namespace: config → state
Confine Pulsar to pulsar_backend.py:
- Delete legacy PulsarClient class from pubsub.py
- Move add_args to add_pubsub_args() with standalone flag
for CLI tools (defaults to localhost)
- PulsarBackendConsumer.receive() catches _pulsar.Timeout,
raises standard TimeoutError
- Remove Pulsar imports from: async_processor, flow_processor,
log_level, all 11 client files, 4 storage writers, gateway
service, gateway config receiver
- Remove log_level/LoggerLevel from client API
- Rewrite tg-monitor-prompts to use backend abstraction
- Update tg-dump-queues to use add_pubsub_args
Also: pubsub-abstraction.md tech spec covering problem statement,
design goals, as-is requirements, candidate broker assessment,
approach, and implementation order.
2026-04-01 20:16:53 +01:00
|
|
|
response = self.encode(obj)
|
2025-06-20 16:59:55 +01:00
|
|
|
|
|
|
|
|
# Check if this is a final response
|
|
|
|
|
is_final = (
|
|
|
|
|
obj.ids is not None or # List response
|
|
|
|
|
obj.eos is True or # End of stream
|
|
|
|
|
(not obj.triples and not obj.graph_embeddings) # Empty response
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
return response, is_final
|