Commit graph

125 commits

Author SHA1 Message Date
Cyber MacGeddon
91d28e9d56 Change input_output to flow_processor 2025-04-16 13:47:23 +01:00
Cyber MacGeddon
8f89336db3 API tweak for input_output 2025-04-16 12:55:47 +01:00
Cyber MacGeddon
510cfb1103 Token chunker 2025-04-15 23:13:21 +01:00
Cyber MacGeddon
cfdeadfb34 Chunker ported 2025-04-15 23:09:32 +01:00
Cyber MacGeddon
c3f1aacbfc Remove cruft 2025-04-15 16:18:03 +01:00
Cyber MacGeddon
60ca5aa3a6 input/output is working 2025-04-15 16:13:18 +01:00
Cyber MacGeddon
98faec7798 PDF done but not working 2025-04-15 00:12:50 +01:00
Cyber MacGeddon
9fdc408a95 Metrics working 2025-04-14 22:39:09 +01:00
Cyber MacGeddon
726baa9932 Tidying 2025-04-14 20:44:22 +01:00
Cyber MacGeddon
74fb52dd3b Separate config part of config-sc 2025-04-14 19:43:36 +01:00
Cyber MacGeddon
d924dad25f Roughly working on config svc 2025-04-14 19:23:43 +01:00
Cyber MacGeddon
1d150cdc21 Working on processor API 2025-04-14 17:51:23 +01:00
Cyber MacGeddon
e66c06f99a New API emerging 2025-04-14 09:05:42 +01:00
Cyber MacGeddon
df6a11bb86 - Changed module names to idents in all processors
- Add --id argument to base
- Tweaked config into handler
2025-04-10 19:14:31 +01:00
cybermaggedon
298d09f388
Prompt and agent manager, dynamically load configuration from config-svc
- prompt-template takes config from the config-svc, dynamically reloads
  as new config appears.
- agent-react takes config from config-svc, dynamically reloads
- Fixed lack of data in config queue, needed to take the Earliest, not the
  Latest values.
- Changed text-completion and knowledge-query tool to both use 'query'
  as the argument.
- Prompt and agent no longer have command line args to supply
  configuration.
2025-04-02 16:37:08 +01:00
cybermaggedon
1d222235d3
Configuration initialisation (#335)
* - Fixed error reporting in config
- Updated tg-init-pulsar to be able to load initial config to config-svc
- Tweaked API naming and added more config calls

* Tools to dump out prompts and agent tools
2025-04-02 13:52:33 +01:00
cybermaggedon
88eae0a9f0
Fix no version/config at startup (#333) 2025-04-01 20:54:59 +01:00
cybermaggedon
fa09dc319e
Feature/config service (#332)
Configuration service provides an API to change configuration. Complete configuration is pushed down a config queue so that users have a complete copy of config object.
2025-04-01 19:47:05 +01:00
Cyber MacGeddon
b1cefbe1f7 Update setup.py files to prep 0.22 branch 2025-03-31 22:14:38 +01:00
cybermaggedon
8efc3b281d
Fix missing openai symbol / allow specifying base URL (#330) 2025-03-25 21:12:34 +00:00
cybermaggedon
b1ab2938b0
Fix/minor issues (#329)
* Fix MistralAI, symbol error caused it to fail to start

* Update templates
2025-03-25 19:14:56 +00:00
cybermaggedon
482592b976
Added Mistral OCR client (#326)
- Added Mistral OCR client
- Template updates for pdf-ocr
- Template updates for pdf-ocr-mistral
2025-03-22 00:27:20 +00:00
Tyler Oliver
fe422b2b95
Add support for Cassandra auth with SSL check (#318)
Following recommended approach in Datastax documenation I've added the necessary TLS/SSL check

https://docs.datastax.com/en/developer/python-driver/3.17/security/index.html
2025-03-20 22:25:23 +00:00
cybermaggedon
cbfe37fec7
Integration with LM Studio LLM hosting (#323) 2025-03-19 19:07:56 +00:00
cybermaggedon
64e42bed6f
Fix/async send typos (#322) 2025-03-19 00:03:58 +00:00
cybermaggedon
1495666e65
Fix typo (#321) 2025-03-18 23:40:31 +00:00
cybermaggedon
dd4a119f49
Fix broken setup.py (#320) 2025-03-18 21:09:18 +00:00
cybermaggedon
6565adb1ec
Enable AKS integration of text-completion-azure-openai (#319)
Reconfigure so that AZURE_TOKEN, AZURE_MODEL and AZURE_ENDPOINT
can be used to set the token/model/endpoint parameters.  This allows it to
be deployed in K8s and use secrets to set these environment variables
2025-03-18 20:27:45 +00:00
cybermaggedon
a922852017
Fix async/sync load issues with knowledge streaming APIs (#315) 2025-03-15 12:38:10 +00:00
JackColquitt
a46de297c1 More Mistral pieces 2025-03-14 18:12:33 -07:00
JackColquitt
5f5cf8fd07 Added basic Mistral API support 2025-03-14 17:47:59 -07:00
cybermaggedon
ef845d6c9b
Feature/rag parameters (#311)
* Change document-rag and graph-rag processing so that the user can
specify parameters.  Changes in Pulsar services, Pulsar message
schemas, gateway and command-line tools.  User-visible changes in
new parameters on command-line tools.

* Fix bugs, graph-rag working

* Get subgraph truncation in the right place

* Graph RAG and document RAG working and configurable

* Multi-hop path traversal GraphRAG

* Add safety valve for path_size set too high
2025-03-13 00:38:18 +00:00
cybermaggedon
f1559c5944
Feature/librarian (#310)
* Add fields to library schema

* Added list function, incomplete

* Librarian list operation
2025-03-11 16:52:59 +00:00
cybermaggedon
5575e885e5
Fix merge issues with api-key, conflicts with librarian changes (#309) 2025-02-15 12:25:26 +00:00
cybermaggedon
617eb7efd5
Feature/pulsar api key support (#308)
* Add pulsar API token check

* Added missing api_key references

---------

Co-authored-by: Tyler O <4535788+toliver38@users.noreply.github.com>
2025-02-15 11:22:48 +00:00
cybermaggedon
f7df2df266
Feature/librarian (#307)
* Bring QDrant up-to-date

* Tables for data from queue outputs

- Pass single Pulsar client to everything in gateway & librarian
- Pulsar listener-name support in gateway
- PDF and text load working in librarian

* Complete Cassandra schema

* Add librarian support to templates
2025-02-12 23:39:24 +00:00
cybermaggedon
f350abb415
Maint/asyncio (#305)
* Move to asyncio services, even though everything is largely sync
2025-02-11 23:24:46 +00:00
cybermaggedon
a0bf2362f6
Librarian (#304) 2025-02-11 16:01:03 +00:00
Tyler Oliver
e99c0ac238 Add support for Qdrant API Auth (#300)
Added the necessary changes to support API Key in Qdrant Client Query+Storage
- Doc Embeddings
- Graph Embeddings
2025-02-08 11:46:22 +00:00
Tyler Oliver
41ccb6c976 Add user and password auth for Cassandra (#301) 2025-02-08 11:42:14 +00:00
Cyber MacGeddon
15c8afa97b Changed setup.py for 0.21 release branch 2025-01-30 21:40:44 +00:00
cybermaggedon
6c3d2e7f97
Feature/fastembed (#286)
* Copied in base files for fastembed

* Added fastembed implementation

* Added template support for fastembed
2025-01-28 18:26:17 +00:00
cybermaggedon
75a72b0d2d
Fix ollama embeddings client to work (#285) 2025-01-27 23:47:15 +00:00
cybermaggedon
552637c1f7
Fix deps (#284)
* Add cryptography module back in

* Lock in known working version of the AI modules, suspecting a problem
in sentence-transformers with Python 3.12?
2025-01-27 21:57:22 +00:00
cybermaggedon
0ebd770cb5
Back out cryptography (#282) 2025-01-27 19:58:38 +00:00
cybermaggedon
0e03bc05a4
Refactor rate limit handling (#280)
* - Refactored retry for rate limits into the base class
- ConsumerProducer is derived from Consumer to simplify code
- Added rate_limit_count metrics for rate limit events

* Add rate limit events to VertexAI and Google AI Studio

* Added Grafana rate limit dashboard

* Add rate limit handling to all LLMs
2025-01-27 17:04:49 +00:00
cybermaggedon
26a586034c
Add cryptography dep (#281) 2025-01-27 17:04:09 +00:00
cybermaggedon
8fc22155dd
Agent tweaks (#272)
- Removed unused LLM client configuration from agent-manager-react
- Change agent-manager-react template to use prompt-rag instead of
  prompt
- Changed TextCompletion tool to use 'question' instead of 'computation'
  for its parameter.
2025-01-20 14:05:51 +00:00
Cyber MacGeddon
9bf916f3c6 Update Python files for 0.20 2025-01-16 00:02:18 +00:00
cybermaggedon
bed7423c26
- Added load/save API for document embeddings (#269)
- tg-load-doc-embeds and tg-save-doc-embeds command line utils
2025-01-16 00:00:59 +00:00