From f25c6949c42d7d6a47cdd1026339dbf5719257b5 Mon Sep 17 00:00:00 2001 From: Cyber MacGeddon Date: Thu, 11 Jul 2024 22:28:34 +0100 Subject: [PATCH] Update docs --- README.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/README.md b/README.md index de1a62c4..406778dd 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,17 @@ by deploying multiple containers. ![architecture](architecture.png) +A set of modules are executed which use Apache Pulsar as a pub/sub system. +This means that Pulsar provides input the modules and accept output. + +Pulsar provides two types of connectivity: +- For processing flows, Pulsar accepts the output of a processing module + and queues it for input to the next module. +- For services such as LLMs and embeddings, Pulsar provides a client/server + model. A Pulsar queue is used as the input to the service. When + processed, the output is delivered to a separate queue so that the caller + can collect the data. + ## Included modules - `chunker-recursive` - Accepts text documents and uses LangChain recurse