- Keeps processing in different flows separate so that data can go to different stores / collections etc.
- Potentially supports different processing flows
- Tidies the processing API with common base-classes for e.g. LLMs, and automatic configuration of 'clients' to use the right queue names in a flow
* Change document-rag and graph-rag processing so that the user can
specify parameters. Changes in Pulsar services, Pulsar message
schemas, gateway and command-line tools. User-visible changes in
new parameters on command-line tools.
* Fix bugs, graph-rag working
* Get subgraph truncation in the right place
* Graph RAG and document RAG working and configurable
* Multi-hop path traversal GraphRAG
* Add safety valve for path_size set too high
* Update schema defs for source -> metadata
* Migrate to use metadata part of schema, also add metadata to triples & vecs
* Add user/collection metadata to query
* Use user/collection in RAG
* Write and query working on triples
