apunkt/trustgraph
1
0
Fork 0
mirror of https://github.com/trustgraph-ai/trustgraph.git synced 2026-04-25 00:16:23 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
trustgraph/Makefile
cybermaggedon 24f0190ce7
RabbitMQ pub/sub backend with topic exchange architecture (#752) 
Adds a RabbitMQ backend as an alternative to Pulsar, selectable via
PUBSUB_BACKEND=rabbitmq. Both backends implement the same PubSubBackend
protocol — no application code changes needed to switch.

RabbitMQ topology:
- Single topic exchange per topicspace (e.g. 'tg')
- Routing key derived from queue class and topic name
- Shared consumers: named queue bound to exchange (competing, round-robin)
- Exclusive consumers: anonymous auto-delete queue (broadcast, each gets
  every message). Used by Subscriber and config push consumer.
- Thread-local producer connections (pika is not thread-safe)
- Push-based consumption via basic_consume with process_data_events
  for heartbeat processing

Consumer model changes:
- Consumer class creates one backend consumer per concurrent task
  (required for pika thread safety, harmless for Pulsar)
- Consumer class accepts consumer_type parameter
- Subscriber passes consumer_type='exclusive' for broadcast semantics
- Config push consumer uses consumer_type='exclusive' so every
  processor instance receives config updates
- handle_one_from_queue receives consumer as parameter for correct
  per-connection ack/nack

LibrarianClient:
- New shared client class replacing duplicated librarian request-response
  code across 6+ services (chunking, decoders, RAG, etc.)
- Uses stream-document instead of get-document-content for fetching
  document content in 1MB chunks (avoids broker message size limits)
- Standalone object (self.librarian = LibrarianClient(...)) not a mixin
- get-document-content marked deprecated in schema and OpenAPI spec

Serialisation:
- Extracted dataclass_to_dict/dict_to_dataclass to shared
  serialization.py (used by both Pulsar and RabbitMQ backends)

Librarian queues:
- Changed from flow class (persistent) back to request/response class
  now that stream-document eliminates large single messages
- API upload chunk size reduced from 5MB to 3MB to stay under broker
  limits after base64 encoding

Factory and CLI:
- get_pubsub() handles 'rabbitmq' backend with RabbitMQ connection params
- add_pubsub_args() includes RabbitMQ options (host, port, credentials)
- add_pubsub_args(standalone=True) defaults to localhost for CLI tools
- init_trustgraph skips Pulsar admin setup for non-Pulsar backends
- tg-dump-queues and tg-monitor-prompts use backend abstraction
- BaseClient and ConfigClient accept generic pubsub config
2026-04-02 12:47:16 +01:00

216 lines
8.6 KiB
Makefile
Raw Blame History

# VERSION=$(shell git describe | sed 's/^v//')
VERSION=0.0.0
DOCKER=podman
all: containers
# Not used
wheels:
pip3 wheel --no-deps --wheel-dir dist trustgraph/
pip3 wheel --no-deps --wheel-dir dist trustgraph-base/
pip3 wheel --no-deps --wheel-dir dist trustgraph-flow/
pip3 wheel --no-deps --wheel-dir dist trustgraph-vertexai/
pip3 wheel --no-deps --wheel-dir dist trustgraph-bedrock/
pip3 wheel --no-deps --wheel-dir dist trustgraph-embeddings-hf/
pip3 wheel --no-deps --wheel-dir dist trustgraph-cli/
pip3 wheel --no-deps --wheel-dir dist trustgraph-ocr/
pip3 wheel --no-deps --wheel-dir dist trustgraph-unstructured/
pip3 wheel --no-deps --wheel-dir dist trustgraph-mcp/
packages: update-package-versions
rm -rf dist/
cd trustgraph && python -m build --sdist --outdir ../dist/
cd trustgraph-base && python -m build --sdist --outdir ../dist/
cd trustgraph-flow && python -m build --sdist --outdir ../dist/
cd trustgraph-vertexai && python -m build --sdist --outdir ../dist/
cd trustgraph-bedrock && python -m build --sdist --outdir ../dist/
cd trustgraph-embeddings-hf && python -m build --sdist --outdir ../dist/
cd trustgraph-cli && python -m build --sdist --outdir ../dist/
cd trustgraph-ocr && python -m build --sdist --outdir ../dist/
cd trustgraph-unstructured && python -m build --sdist --outdir ../dist/
cd trustgraph-mcp && python -m build --sdist --outdir ../dist/
pypi-upload:
twine upload dist/*-${VERSION}.*
CONTAINER_BASE=docker.io/trustgraph
update-package-versions:
mkdir -p trustgraph-cli/trustgraph
mkdir -p trustgraph/trustgraph
echo __version__ = \"${VERSION}\" > trustgraph-base/trustgraph/base_version.py
echo __version__ = \"${VERSION}\" > trustgraph-flow/trustgraph/flow_version.py
echo __version__ = \"${VERSION}\" > trustgraph-vertexai/trustgraph/vertexai_version.py
echo __version__ = \"${VERSION}\" > trustgraph-bedrock/trustgraph/bedrock_version.py
echo __version__ = \"${VERSION}\" > trustgraph-embeddings-hf/trustgraph/embeddings_hf_version.py
echo __version__ = \"${VERSION}\" > trustgraph-cli/trustgraph/cli_version.py
echo __version__ = \"${VERSION}\" > trustgraph-ocr/trustgraph/ocr_version.py
echo __version__ = \"${VERSION}\" > trustgraph-unstructured/trustgraph/unstructured_version.py
echo __version__ = \"${VERSION}\" > trustgraph/trustgraph/trustgraph_version.py
echo __version__ = \"${VERSION}\" > trustgraph-mcp/trustgraph/mcp_version.py
FORCE:
containers: FORCE
${DOCKER} build -f containers/Containerfile.base \
-t ${CONTAINER_BASE}/trustgraph-base:${VERSION} .
${DOCKER} build -f containers/Containerfile.flow \
-t ${CONTAINER_BASE}/trustgraph-flow:${VERSION} .
${DOCKER} build -f containers/Containerfile.bedrock \
-t ${CONTAINER_BASE}/trustgraph-bedrock:${VERSION} .
${DOCKER} build -f containers/Containerfile.vertexai \
-t ${CONTAINER_BASE}/trustgraph-vertexai:${VERSION} .
${DOCKER} build -f containers/Containerfile.hf \
-t ${CONTAINER_BASE}/trustgraph-hf:${VERSION} .
${DOCKER} build -f containers/Containerfile.ocr \
-t ${CONTAINER_BASE}/trustgraph-ocr:${VERSION} .
${DOCKER} build -f containers/Containerfile.unstructured \
-t ${CONTAINER_BASE}/trustgraph-unstructured:${VERSION} .
${DOCKER} build -f containers/Containerfile.mcp \
-t ${CONTAINER_BASE}/trustgraph-mcp:${VERSION} .
some-containers:
${DOCKER} build -f containers/Containerfile.base \
-t ${CONTAINER_BASE}/trustgraph-base:${VERSION} .
${DOCKER} build -f containers/Containerfile.flow \
-t ${CONTAINER_BASE}/trustgraph-flow:${VERSION} .
${DOCKER} build -f containers/Containerfile.unstructured \
-t ${CONTAINER_BASE}/trustgraph-unstructured:${VERSION} .
# ${DOCKER} build -f containers/Containerfile.vertexai \
# -t ${CONTAINER_BASE}/trustgraph-vertexai:${VERSION} .
# ${DOCKER} build -f containers/Containerfile.mcp \
# -t ${CONTAINER_BASE}/trustgraph-mcp:${VERSION} .
# ${DOCKER} build -f containers/Containerfile.vertexai \
# -t ${CONTAINER_BASE}/trustgraph-vertexai:${VERSION} .
# ${DOCKER} build -f containers/Containerfile.bedrock \
# -t ${CONTAINER_BASE}/trustgraph-bedrock:${VERSION} .
basic-containers: update-package-versions
${DOCKER} build -f containers/Containerfile.base \
-t ${CONTAINER_BASE}/trustgraph-base:${VERSION} .
${DOCKER} build -f containers/Containerfile.flow \
-t ${CONTAINER_BASE}/trustgraph-flow:${VERSION} .
container.ocr:
${DOCKER} build -f containers/Containerfile.ocr \
-t ${CONTAINER_BASE}/trustgraph-ocr:${VERSION} .
push:
${DOCKER} push ${CONTAINER_BASE}/trustgraph-base:${VERSION}
${DOCKER} push ${CONTAINER_BASE}/trustgraph-flow:${VERSION}
${DOCKER} push ${CONTAINER_BASE}/trustgraph-bedrock:${VERSION}
${DOCKER} push ${CONTAINER_BASE}/trustgraph-vertexai:${VERSION}
${DOCKER} push ${CONTAINER_BASE}/trustgraph-hf:${VERSION}
${DOCKER} push ${CONTAINER_BASE}/trustgraph-ocr:${VERSION}
${DOCKER} push ${CONTAINER_BASE}/trustgraph-unstructured:${VERSION}
${DOCKER} push ${CONTAINER_BASE}/trustgraph-mcp:${VERSION}
# Individual container build targets
container-trustgraph-base: update-package-versions
${DOCKER} build -f containers/Containerfile.base -t ${CONTAINER_BASE}/trustgraph-base:${VERSION} .
container-trustgraph-flow: update-package-versions
${DOCKER} build -f containers/Containerfile.flow -t ${CONTAINER_BASE}/trustgraph-flow:${VERSION} .
container-trustgraph-bedrock: update-package-versions
${DOCKER} build -f containers/Containerfile.bedrock -t ${CONTAINER_BASE}/trustgraph-bedrock:${VERSION} .
container-trustgraph-vertexai: update-package-versions
${DOCKER} build -f containers/Containerfile.vertexai -t ${CONTAINER_BASE}/trustgraph-vertexai:${VERSION} .
container-trustgraph-hf: update-package-versions
${DOCKER} build -f containers/Containerfile.hf -t ${CONTAINER_BASE}/trustgraph-hf:${VERSION} .
container-trustgraph-ocr: update-package-versions
${DOCKER} build -f containers/Containerfile.ocr -t ${CONTAINER_BASE}/trustgraph-ocr:${VERSION} .
container-trustgraph-unstructured: update-package-versions
${DOCKER} build -f containers/Containerfile.unstructured -t ${CONTAINER_BASE}/trustgraph-unstructured:${VERSION} .
container-trustgraph-mcp: update-package-versions
${DOCKER} build -f containers/Containerfile.mcp -t ${CONTAINER_BASE}/trustgraph-mcp:${VERSION} .
# Individual container push targets
push-trustgraph-base:
${DOCKER} push ${CONTAINER_BASE}/trustgraph-base:${VERSION}
push-trustgraph-flow:
${DOCKER} push ${CONTAINER_BASE}/trustgraph-flow:${VERSION}
push-trustgraph-bedrock:
${DOCKER} push ${CONTAINER_BASE}/trustgraph-bedrock:${VERSION}
push-trustgraph-vertexai:
${DOCKER} push ${CONTAINER_BASE}/trustgraph-vertexai:${VERSION}
push-trustgraph-hf:
${DOCKER} push ${CONTAINER_BASE}/trustgraph-hf:${VERSION}
push-trustgraph-ocr:
${DOCKER} push ${CONTAINER_BASE}/trustgraph-ocr:${VERSION}
push-trustgraph-unstructured:
${DOCKER} push ${CONTAINER_BASE}/trustgraph-unstructured:${VERSION}
push-trustgraph-mcp:
${DOCKER} push ${CONTAINER_BASE}/trustgraph-mcp:${VERSION}
clean:
rm -rf wheels/
set-version:
echo '"${VERSION}"' > templates/values/version.jsonnet
TEMPLATES=azure bedrock claude cohere mix llamafile mistral ollama openai vertexai \
openai-neo4j storage
DCS=$(foreach template,${TEMPLATES},${template:%=tg-launch-%.yaml})
MODELS=azure bedrock claude cohere llamafile mistral ollama openai vertexai
GRAPHS=cassandra neo4j falkordb memgraph
# tg-launch-%.yaml: templates/%.jsonnet templates/components/version.jsonnet
# jsonnet -Jtemplates \
# -S ${@:tg-launch-%.yaml=templates/%.jsonnet} > $@
# VECTORDB=milvus
VECTORDB=qdrant
JSONNET_FLAGS=-J templates -J .
# Temporarily going back to how templates were built in 0.9 because this
# is going away in 0.11.
update-templates: update-dcs
JSON_TO_YAML=python -c 'import sys, yaml, json; j=json.loads(sys.stdin.read()); print(yaml.safe_dump(j))'
update-dcs: set-version
for graph in ${GRAPHS}; do \
cm=$${graph},pulsar,${VECTORDB},grafana; \
input=templates/opts-to-docker-compose.jsonnet; \
output=tg-storage-$${graph}.yaml; \
echo $${graph} '->' $${output}; \
jsonnet ${JSONNET_FLAGS} \
--ext-str options=$${cm} $${input} | \
${JSON_TO_YAML} > $${output}; \
done
for model in ${MODELS}; do \
for graph in ${GRAPHS}; do \
cm=$${graph},pulsar,${VECTORDB},embeddings-hf,graph-rag,grafana,trustgraph,$${model}; \
input=templates/opts-to-docker-compose.jsonnet; \
output=tg-launch-$${model}-$${graph}.yaml; \
echo $${model} + $${graph} '->' $${output}; \
jsonnet ${JSONNET_FLAGS} \
--ext-str options=$${cm} $${input} | \
${JSON_TO_YAML} > $${output}; \
done; \
done
docker-hub-login:
cat docker-token.txt | \
${DOCKER} login -u trustgraph --password-stdin registry-1.docker.io
Reference in a new issue View git blame Copy permalink