Pub/sub abstraction: decouple from Pulsar (#751)
Remove Pulsar-specific concepts from application code so that
the pub/sub backend is swappable via configuration.
Rename translators:
- to_pulsar/from_pulsar → decode/encode across all translator
classes, dispatch handlers, and tests (55+ files)
- from_response_with_completion → encode_with_completion
- Remove pulsar.schema.Record from translator base class
Queue naming (CLASS:TOPICSPACE:TOPIC):
- Replace topic() helper with queue() using new format:
flow:tg:name, request:tg:name, response:tg:name, state:tg:name
- Queue class implies persistence/TTL (no QoS in names)
- Update Pulsar backend map_topic() to parse new format
- Librarian queues use flow class (persistent, for chunking)
- Config push uses state class (persistent, last-value)
- Remove 15 dead topic imports from schema files
- Update init_trustgraph.py namespace: config → state
Confine Pulsar to pulsar_backend.py:
- Delete legacy PulsarClient class from pubsub.py
- Move add_args to add_pubsub_args() with standalone flag
for CLI tools (defaults to localhost)
- PulsarBackendConsumer.receive() catches _pulsar.Timeout,
raises standard TimeoutError
- Remove Pulsar imports from: async_processor, flow_processor,
log_level, all 11 client files, 4 storage writers, gateway
service, gateway config receiver
- Remove log_level/LoggerLevel from client API
- Rewrite tg-monitor-prompts to use backend abstraction
- Update tg-dump-queues to use add_pubsub_args
Also: pubsub-abstraction.md tech spec covering problem statement,
design goals, as-is requirements, candidate broker assessment,
approach, and implementation order.
2026-04-01 20:16:53 +01:00
|
|
|
"""
|
|
|
|
|
Tests for queue naming and topic mapping.
|
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
import pytest
|
|
|
|
|
import argparse
|
|
|
|
|
|
|
|
|
|
from trustgraph.schema.core.topic import queue
|
|
|
|
|
from trustgraph.base.pubsub import get_pubsub, add_pubsub_args
|
|
|
|
|
from trustgraph.base.pulsar_backend import PulsarBackend
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
class TestQueueFunction:
|
|
|
|
|
|
|
|
|
|
def test_flow_default(self):
|
|
|
|
|
assert queue('text-completion-request') == 'flow:tg:text-completion-request'
|
|
|
|
|
|
|
|
|
|
def test_request_class(self):
|
|
|
|
|
assert queue('config', cls='request') == 'request:tg:config'
|
|
|
|
|
|
|
|
|
|
def test_response_class(self):
|
|
|
|
|
assert queue('config', cls='response') == 'response:tg:config'
|
|
|
|
|
|
2026-04-09 09:55:41 +01:00
|
|
|
def test_notify_class(self):
|
|
|
|
|
assert queue('config', cls='notify') == 'notify:tg:config'
|
Pub/sub abstraction: decouple from Pulsar (#751)
Remove Pulsar-specific concepts from application code so that
the pub/sub backend is swappable via configuration.
Rename translators:
- to_pulsar/from_pulsar → decode/encode across all translator
classes, dispatch handlers, and tests (55+ files)
- from_response_with_completion → encode_with_completion
- Remove pulsar.schema.Record from translator base class
Queue naming (CLASS:TOPICSPACE:TOPIC):
- Replace topic() helper with queue() using new format:
flow:tg:name, request:tg:name, response:tg:name, state:tg:name
- Queue class implies persistence/TTL (no QoS in names)
- Update Pulsar backend map_topic() to parse new format
- Librarian queues use flow class (persistent, for chunking)
- Config push uses state class (persistent, last-value)
- Remove 15 dead topic imports from schema files
- Update init_trustgraph.py namespace: config → state
Confine Pulsar to pulsar_backend.py:
- Delete legacy PulsarClient class from pubsub.py
- Move add_args to add_pubsub_args() with standalone flag
for CLI tools (defaults to localhost)
- PulsarBackendConsumer.receive() catches _pulsar.Timeout,
raises standard TimeoutError
- Remove Pulsar imports from: async_processor, flow_processor,
log_level, all 11 client files, 4 storage writers, gateway
service, gateway config receiver
- Remove log_level/LoggerLevel from client API
- Rewrite tg-monitor-prompts to use backend abstraction
- Update tg-dump-queues to use add_pubsub_args
Also: pubsub-abstraction.md tech spec covering problem statement,
design goals, as-is requirements, candidate broker assessment,
approach, and implementation order.
2026-04-01 20:16:53 +01:00
|
|
|
|
|
|
|
|
def test_custom_topicspace(self):
|
|
|
|
|
assert queue('config', cls='request', topicspace='prod') == 'request:prod:config'
|
|
|
|
|
|
|
|
|
|
def test_default_class_is_flow(self):
|
|
|
|
|
result = queue('something')
|
|
|
|
|
assert result.startswith('flow:')
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
class TestPulsarMapTopic:
|
|
|
|
|
|
|
|
|
|
@pytest.fixture
|
|
|
|
|
def backend(self):
|
|
|
|
|
"""Create a PulsarBackend without connecting."""
|
|
|
|
|
b = object.__new__(PulsarBackend)
|
|
|
|
|
return b
|
|
|
|
|
|
|
|
|
|
def test_flow_maps_to_persistent(self, backend):
|
|
|
|
|
assert backend.map_topic('flow:tg:text-completion-request') == \
|
|
|
|
|
'persistent://tg/flow/text-completion-request'
|
|
|
|
|
|
2026-04-09 09:55:41 +01:00
|
|
|
def test_notify_maps_to_non_persistent(self, backend):
|
|
|
|
|
assert backend.map_topic('notify:tg:config') == \
|
|
|
|
|
'non-persistent://tg/notify/config'
|
Pub/sub abstraction: decouple from Pulsar (#751)
Remove Pulsar-specific concepts from application code so that
the pub/sub backend is swappable via configuration.
Rename translators:
- to_pulsar/from_pulsar → decode/encode across all translator
classes, dispatch handlers, and tests (55+ files)
- from_response_with_completion → encode_with_completion
- Remove pulsar.schema.Record from translator base class
Queue naming (CLASS:TOPICSPACE:TOPIC):
- Replace topic() helper with queue() using new format:
flow:tg:name, request:tg:name, response:tg:name, state:tg:name
- Queue class implies persistence/TTL (no QoS in names)
- Update Pulsar backend map_topic() to parse new format
- Librarian queues use flow class (persistent, for chunking)
- Config push uses state class (persistent, last-value)
- Remove 15 dead topic imports from schema files
- Update init_trustgraph.py namespace: config → state
Confine Pulsar to pulsar_backend.py:
- Delete legacy PulsarClient class from pubsub.py
- Move add_args to add_pubsub_args() with standalone flag
for CLI tools (defaults to localhost)
- PulsarBackendConsumer.receive() catches _pulsar.Timeout,
raises standard TimeoutError
- Remove Pulsar imports from: async_processor, flow_processor,
log_level, all 11 client files, 4 storage writers, gateway
service, gateway config receiver
- Remove log_level/LoggerLevel from client API
- Rewrite tg-monitor-prompts to use backend abstraction
- Update tg-dump-queues to use add_pubsub_args
Also: pubsub-abstraction.md tech spec covering problem statement,
design goals, as-is requirements, candidate broker assessment,
approach, and implementation order.
2026-04-01 20:16:53 +01:00
|
|
|
|
|
|
|
|
def test_request_maps_to_non_persistent(self, backend):
|
|
|
|
|
assert backend.map_topic('request:tg:config') == \
|
|
|
|
|
'non-persistent://tg/request/config'
|
|
|
|
|
|
|
|
|
|
def test_response_maps_to_non_persistent(self, backend):
|
|
|
|
|
assert backend.map_topic('response:tg:librarian') == \
|
|
|
|
|
'non-persistent://tg/response/librarian'
|
|
|
|
|
|
|
|
|
|
def test_passthrough_pulsar_uri(self, backend):
|
|
|
|
|
uri = 'persistent://tg/flow/something'
|
|
|
|
|
assert backend.map_topic(uri) == uri
|
|
|
|
|
|
|
|
|
|
def test_invalid_format_raises(self, backend):
|
|
|
|
|
with pytest.raises(ValueError, match="Invalid queue format"):
|
|
|
|
|
backend.map_topic('bad-format')
|
|
|
|
|
|
|
|
|
|
def test_invalid_class_raises(self, backend):
|
|
|
|
|
with pytest.raises(ValueError, match="Invalid queue class"):
|
|
|
|
|
backend.map_topic('unknown:tg:topic')
|
|
|
|
|
|
|
|
|
|
def test_custom_topicspace(self, backend):
|
|
|
|
|
assert backend.map_topic('flow:prod:my-queue') == \
|
|
|
|
|
'persistent://prod/flow/my-queue'
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
class TestGetPubsubDispatch:
|
|
|
|
|
|
|
|
|
|
def test_unknown_backend_raises(self):
|
|
|
|
|
with pytest.raises(ValueError, match="Unknown pub/sub backend"):
|
|
|
|
|
get_pubsub(pubsub_backend='redis')
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
class TestAddPubsubArgs:
|
|
|
|
|
|
|
|
|
|
def test_standalone_defaults_to_localhost(self):
|
|
|
|
|
parser = argparse.ArgumentParser()
|
|
|
|
|
add_pubsub_args(parser, standalone=True)
|
|
|
|
|
args = parser.parse_args([])
|
|
|
|
|
assert args.pulsar_host == 'pulsar://localhost:6650'
|
|
|
|
|
assert args.pulsar_listener == 'localhost'
|
|
|
|
|
|
|
|
|
|
def test_non_standalone_defaults_to_container(self):
|
|
|
|
|
parser = argparse.ArgumentParser()
|
|
|
|
|
add_pubsub_args(parser, standalone=False)
|
|
|
|
|
args = parser.parse_args([])
|
|
|
|
|
assert 'pulsar:6650' in args.pulsar_host
|
|
|
|
|
assert args.pulsar_listener is None
|
|
|
|
|
|
|
|
|
|
def test_cli_override_respected(self):
|
|
|
|
|
parser = argparse.ArgumentParser()
|
|
|
|
|
add_pubsub_args(parser, standalone=True)
|
|
|
|
|
args = parser.parse_args(['--pulsar-host', 'pulsar://custom:6650'])
|
|
|
|
|
assert args.pulsar_host == 'pulsar://custom:6650'
|
|
|
|
|
|
|
|
|
|
def test_pubsub_backend_default(self):
|
|
|
|
|
parser = argparse.ArgumentParser()
|
|
|
|
|
add_pubsub_args(parser)
|
|
|
|
|
args = parser.parse_args([])
|
|
|
|
|
assert args.pubsub_backend == 'pulsar'
|
|
|
|
|
|
|
|
|
|
|
2026-04-02 12:47:16 +01:00
|
|
|
class TestAddPubsubArgsRabbitMQ:
|
|
|
|
|
|
|
|
|
|
def test_rabbitmq_args_present(self):
|
|
|
|
|
parser = argparse.ArgumentParser()
|
|
|
|
|
add_pubsub_args(parser)
|
|
|
|
|
args = parser.parse_args([
|
|
|
|
|
'--pubsub-backend', 'rabbitmq',
|
|
|
|
|
'--rabbitmq-host', 'myhost',
|
|
|
|
|
'--rabbitmq-port', '5673',
|
|
|
|
|
])
|
|
|
|
|
assert args.pubsub_backend == 'rabbitmq'
|
|
|
|
|
assert args.rabbitmq_host == 'myhost'
|
|
|
|
|
assert args.rabbitmq_port == 5673
|
|
|
|
|
|
|
|
|
|
def test_rabbitmq_defaults_container(self):
|
|
|
|
|
parser = argparse.ArgumentParser()
|
|
|
|
|
add_pubsub_args(parser)
|
|
|
|
|
args = parser.parse_args([])
|
|
|
|
|
assert args.rabbitmq_host == 'rabbitmq'
|
|
|
|
|
assert args.rabbitmq_port == 5672
|
|
|
|
|
assert args.rabbitmq_username == 'guest'
|
|
|
|
|
assert args.rabbitmq_password == 'guest'
|
|
|
|
|
assert args.rabbitmq_vhost == '/'
|
|
|
|
|
|
|
|
|
|
def test_rabbitmq_standalone_defaults_to_localhost(self):
|
|
|
|
|
parser = argparse.ArgumentParser()
|
|
|
|
|
add_pubsub_args(parser, standalone=True)
|
|
|
|
|
args = parser.parse_args([])
|
|
|
|
|
assert args.rabbitmq_host == 'localhost'
|
|
|
|
|
|
|
|
|
|
|
Pub/sub abstraction: decouple from Pulsar (#751)
Remove Pulsar-specific concepts from application code so that
the pub/sub backend is swappable via configuration.
Rename translators:
- to_pulsar/from_pulsar → decode/encode across all translator
classes, dispatch handlers, and tests (55+ files)
- from_response_with_completion → encode_with_completion
- Remove pulsar.schema.Record from translator base class
Queue naming (CLASS:TOPICSPACE:TOPIC):
- Replace topic() helper with queue() using new format:
flow:tg:name, request:tg:name, response:tg:name, state:tg:name
- Queue class implies persistence/TTL (no QoS in names)
- Update Pulsar backend map_topic() to parse new format
- Librarian queues use flow class (persistent, for chunking)
- Config push uses state class (persistent, last-value)
- Remove 15 dead topic imports from schema files
- Update init_trustgraph.py namespace: config → state
Confine Pulsar to pulsar_backend.py:
- Delete legacy PulsarClient class from pubsub.py
- Move add_args to add_pubsub_args() with standalone flag
for CLI tools (defaults to localhost)
- PulsarBackendConsumer.receive() catches _pulsar.Timeout,
raises standard TimeoutError
- Remove Pulsar imports from: async_processor, flow_processor,
log_level, all 11 client files, 4 storage writers, gateway
service, gateway config receiver
- Remove log_level/LoggerLevel from client API
- Rewrite tg-monitor-prompts to use backend abstraction
- Update tg-dump-queues to use add_pubsub_args
Also: pubsub-abstraction.md tech spec covering problem statement,
design goals, as-is requirements, candidate broker assessment,
approach, and implementation order.
2026-04-01 20:16:53 +01:00
|
|
|
class TestQueueDefinitions:
|
|
|
|
|
"""Verify the actual queue constants produce correct names."""
|
|
|
|
|
|
|
|
|
|
def test_config_request(self):
|
|
|
|
|
from trustgraph.schema.services.config import config_request_queue
|
|
|
|
|
assert config_request_queue == 'request:tg:config'
|
|
|
|
|
|
|
|
|
|
def test_config_response(self):
|
|
|
|
|
from trustgraph.schema.services.config import config_response_queue
|
|
|
|
|
assert config_response_queue == 'response:tg:config'
|
|
|
|
|
|
|
|
|
|
def test_config_push(self):
|
|
|
|
|
from trustgraph.schema.services.config import config_push_queue
|
2026-04-09 09:55:41 +01:00
|
|
|
assert config_push_queue == 'notify:tg:config'
|
Pub/sub abstraction: decouple from Pulsar (#751)
Remove Pulsar-specific concepts from application code so that
the pub/sub backend is swappable via configuration.
Rename translators:
- to_pulsar/from_pulsar → decode/encode across all translator
classes, dispatch handlers, and tests (55+ files)
- from_response_with_completion → encode_with_completion
- Remove pulsar.schema.Record from translator base class
Queue naming (CLASS:TOPICSPACE:TOPIC):
- Replace topic() helper with queue() using new format:
flow:tg:name, request:tg:name, response:tg:name, state:tg:name
- Queue class implies persistence/TTL (no QoS in names)
- Update Pulsar backend map_topic() to parse new format
- Librarian queues use flow class (persistent, for chunking)
- Config push uses state class (persistent, last-value)
- Remove 15 dead topic imports from schema files
- Update init_trustgraph.py namespace: config → state
Confine Pulsar to pulsar_backend.py:
- Delete legacy PulsarClient class from pubsub.py
- Move add_args to add_pubsub_args() with standalone flag
for CLI tools (defaults to localhost)
- PulsarBackendConsumer.receive() catches _pulsar.Timeout,
raises standard TimeoutError
- Remove Pulsar imports from: async_processor, flow_processor,
log_level, all 11 client files, 4 storage writers, gateway
service, gateway config receiver
- Remove log_level/LoggerLevel from client API
- Rewrite tg-monitor-prompts to use backend abstraction
- Update tg-dump-queues to use add_pubsub_args
Also: pubsub-abstraction.md tech spec covering problem statement,
design goals, as-is requirements, candidate broker assessment,
approach, and implementation order.
2026-04-01 20:16:53 +01:00
|
|
|
|
2026-04-02 12:47:16 +01:00
|
|
|
def test_librarian_request(self):
|
Pub/sub abstraction: decouple from Pulsar (#751)
Remove Pulsar-specific concepts from application code so that
the pub/sub backend is swappable via configuration.
Rename translators:
- to_pulsar/from_pulsar → decode/encode across all translator
classes, dispatch handlers, and tests (55+ files)
- from_response_with_completion → encode_with_completion
- Remove pulsar.schema.Record from translator base class
Queue naming (CLASS:TOPICSPACE:TOPIC):
- Replace topic() helper with queue() using new format:
flow:tg:name, request:tg:name, response:tg:name, state:tg:name
- Queue class implies persistence/TTL (no QoS in names)
- Update Pulsar backend map_topic() to parse new format
- Librarian queues use flow class (persistent, for chunking)
- Config push uses state class (persistent, last-value)
- Remove 15 dead topic imports from schema files
- Update init_trustgraph.py namespace: config → state
Confine Pulsar to pulsar_backend.py:
- Delete legacy PulsarClient class from pubsub.py
- Move add_args to add_pubsub_args() with standalone flag
for CLI tools (defaults to localhost)
- PulsarBackendConsumer.receive() catches _pulsar.Timeout,
raises standard TimeoutError
- Remove Pulsar imports from: async_processor, flow_processor,
log_level, all 11 client files, 4 storage writers, gateway
service, gateway config receiver
- Remove log_level/LoggerLevel from client API
- Rewrite tg-monitor-prompts to use backend abstraction
- Update tg-dump-queues to use add_pubsub_args
Also: pubsub-abstraction.md tech spec covering problem statement,
design goals, as-is requirements, candidate broker assessment,
approach, and implementation order.
2026-04-01 20:16:53 +01:00
|
|
|
from trustgraph.schema.services.library import librarian_request_queue
|
2026-04-02 12:47:16 +01:00
|
|
|
assert librarian_request_queue == 'request:tg:librarian'
|
Pub/sub abstraction: decouple from Pulsar (#751)
Remove Pulsar-specific concepts from application code so that
the pub/sub backend is swappable via configuration.
Rename translators:
- to_pulsar/from_pulsar → decode/encode across all translator
classes, dispatch handlers, and tests (55+ files)
- from_response_with_completion → encode_with_completion
- Remove pulsar.schema.Record from translator base class
Queue naming (CLASS:TOPICSPACE:TOPIC):
- Replace topic() helper with queue() using new format:
flow:tg:name, request:tg:name, response:tg:name, state:tg:name
- Queue class implies persistence/TTL (no QoS in names)
- Update Pulsar backend map_topic() to parse new format
- Librarian queues use flow class (persistent, for chunking)
- Config push uses state class (persistent, last-value)
- Remove 15 dead topic imports from schema files
- Update init_trustgraph.py namespace: config → state
Confine Pulsar to pulsar_backend.py:
- Delete legacy PulsarClient class from pubsub.py
- Move add_args to add_pubsub_args() with standalone flag
for CLI tools (defaults to localhost)
- PulsarBackendConsumer.receive() catches _pulsar.Timeout,
raises standard TimeoutError
- Remove Pulsar imports from: async_processor, flow_processor,
log_level, all 11 client files, 4 storage writers, gateway
service, gateway config receiver
- Remove log_level/LoggerLevel from client API
- Rewrite tg-monitor-prompts to use backend abstraction
- Update tg-dump-queues to use add_pubsub_args
Also: pubsub-abstraction.md tech spec covering problem statement,
design goals, as-is requirements, candidate broker assessment,
approach, and implementation order.
2026-04-01 20:16:53 +01:00
|
|
|
|
|
|
|
|
def test_knowledge_request(self):
|
|
|
|
|
from trustgraph.schema.knowledge.knowledge import knowledge_request_queue
|
|
|
|
|
assert knowledge_request_queue == 'request:tg:knowledge'
|