mirror of
https://github.com/trustgraph-ai/trustgraph.git
synced 2026-04-25 00:16:23 +02:00
24f0190ce7
Adds a RabbitMQ backend as an alternative to Pulsar, selectable via PUBSUB_BACKEND=rabbitmq. Both backends implement the same PubSubBackend protocol — no application code changes needed to switch. RabbitMQ topology: - Single topic exchange per topicspace (e.g. 'tg') - Routing key derived from queue class and topic name - Shared consumers: named queue bound to exchange (competing, round-robin) - Exclusive consumers: anonymous auto-delete queue (broadcast, each gets every message). Used by Subscriber and config push consumer. - Thread-local producer connections (pika is not thread-safe) - Push-based consumption via basic_consume with process_data_events for heartbeat processing Consumer model changes: - Consumer class creates one backend consumer per concurrent task (required for pika thread safety, harmless for Pulsar) - Consumer class accepts consumer_type parameter - Subscriber passes consumer_type='exclusive' for broadcast semantics - Config push consumer uses consumer_type='exclusive' so every processor instance receives config updates - handle_one_from_queue receives consumer as parameter for correct per-connection ack/nack LibrarianClient: - New shared client class replacing duplicated librarian request-response code across 6+ services (chunking, decoders, RAG, etc.) - Uses stream-document instead of get-document-content for fetching document content in 1MB chunks (avoids broker message size limits) - Standalone object (self.librarian = LibrarianClient(...)) not a mixin - get-document-content marked deprecated in schema and OpenAPI spec Serialisation: - Extracted dataclass_to_dict/dict_to_dataclass to shared serialization.py (used by both Pulsar and RabbitMQ backends) Librarian queues: - Changed from flow class (persistent) back to request/response class now that stream-document eliminates large single messages - API upload chunk size reduced from 5MB to 3MB to stay under broker limits after base64 encoding Factory and CLI: - get_pubsub() handles 'rabbitmq' backend with RabbitMQ connection params - add_pubsub_args() includes RabbitMQ options (host, port, credentials) - add_pubsub_args(standalone=True) defaults to localhost for CLI tools - init_trustgraph skips Pulsar admin setup for non-Pulsar backends - tg-dump-queues and tg-monitor-prompts use backend abstraction - BaseClient and ConfigClient accept generic pubsub config |
||
|---|---|---|
| .. | ||
| components | ||
| paths | ||
| security | ||
| openapi.yaml | ||
| README.md | ||
TrustGraph OpenAPI Specification
This directory contains the modular OpenAPI 3.1 specification for the TrustGraph REST API Gateway.
Structure
specs/api/
├── openapi.yaml # Main entry point
├── paths/ # Endpoint definitions
│ ├── config.yaml
│ ├── flow.yaml
│ ├── flow-services/ # Flow-hosted services
│ └── import-export/
├── components/
│ ├── schemas/ # Request/response schemas
│ │ ├── config/
│ │ ├── flow/
│ │ ├── ai-services/
│ │ ├── common/
│ │ └── errors/
│ ├── parameters/ # Reusable parameters
│ ├── responses/ # Reusable responses
│ └── examples/ # Example payloads
└── security/ # Security schemes
└── bearerAuth.yaml
Viewing the Spec
Swagger UI
# Install swagger-ui
npm install -g swagger-ui-watcher
# View in browser
swagger-ui-watcher specs/api/openapi.yaml
Redoc
# Install redoc-cli
npm install -g redoc-cli
# Generate static HTML
redoc-cli bundle specs/api/openapi.yaml -o api-docs.html
# View
open api-docs.html
Online Validators
Upload openapi.yaml to:
Validation
# Install openapi-spec-validator
pip install openapi-spec-validator
# Validate
openapi-spec-validator specs/api/openapi.yaml
Development
When adding a new service:
- Create schema files in
components/schemas/{service}/ - Create path file in
paths/orpaths/flow-services/ - Add examples if needed
- Reference from
openapi.yaml - Validate