Commit graph

130 commits

Author SHA1 Message Date
cybermaggedon
31b7ade44d
Feature/knowledge load (#372)
* Switch off retry in Cassandra until we can differentiate retryable errors

* Fix config getvalues

* Loading knowledge cores works
2025-05-08 00:41:45 +01:00
cybermaggedon
4461d7b289
Feature/persist config (#370)
* Cassandra tables for config

* Config is backed by Cassandra
2025-05-07 12:58:32 +01:00
cybermaggedon
f7123ac57f
tg-put-kg-core command (#369) 2025-05-07 11:13:21 +01:00
cybermaggedon
807c19fd22
knowledge service (#367)
* Write knowledge core elements to Cassandra

* Store service works, building management service

* kg-manager
2025-05-06 23:44:10 +01:00
cybermaggedon
d0da122bed
Fix/llms (#366)
* Fix LMStudio, cache documents with tg-load-sample-documents

* Fix Mistral
2025-05-06 16:17:16 +01:00
cybermaggedon
54e475fa3a
Sample docs loader (#365) 2025-05-06 13:43:17 +01:00
cybermaggedon
9e4eb634a4
Fix/chunking not enabled (#364)
* Enable chunking by default in producer

* Fix some issues including uploading large docs
2025-05-06 00:28:20 +01:00
cybermaggedon
8146f0f2ff
Librarian doc submission (#362) 2025-05-04 22:56:47 +01:00
cybermaggedon
ff28d26f4d
Feature/flow librarian (#361)
* Update librarian to new API

* Implementing new schema with document + processing objects
2025-05-04 22:26:19 +01:00
cybermaggedon
3b8b9ea866
Feature/flow api 3 (#358)
* Working mux socket

* Change API to incorporate flow

* Add Flow ID to all relevant CLIs, not completely implemented

* Change tg-processor-state to use API gateway

* Updated all CLIs

* New tg-show-flow-state command

* tg-show-flow-state shows classes too
2025-05-03 10:39:53 +01:00
cybermaggedon
a70ae9793a
Flow API - update gateway (#357)
* Altered API to incorporate Flow IDs, refactored for dynamic start/stop of flows
* Gateway: Split endpoint / dispatcher for maintainability
2025-05-02 21:11:50 +01:00
cybermaggedon
450f664b1b
Feature/flow enable api gateway (#356)
* Tweak timeouts, reduce stop time for publishers / subscribers

* More APIs working as flow endpoint
2025-04-29 23:34:41 +01:00
cybermaggedon
027b52cd7c
Fix/get multiple flows working (#355)
* Reduce log output

* Fix problems
2025-04-29 00:06:41 +01:00
cybermaggedon
9508ac6c69
Port metering to new API, not tested. (#354)
- Port metering to new API
- Moved price list to configuration
- Added tg-set-token-costs and tg-show-token-costs utils.
2025-04-28 21:26:38 +01:00
cybermaggedon
5af7909122
Update LLMs to LlmService API (#353) 2025-04-25 19:57:42 +01:00
cybermaggedon
099018e103
Update package versions (#352) 2025-04-25 19:45:02 +01:00
cybermaggedon
342f555948
Feature/interface descriptions (#348) 2025-04-25 15:59:34 +01:00
cybermaggedon
3b021720c5
Feature/flow management cli (#346)
Flow management API + various flow management commands

trustgraph-cli/scripts/tg-delete-flow-class
trustgraph-cli/scripts/tg-get-flow-class
trustgraph-cli/scripts/tg-put-flow-class
trustgraph-cli/scripts/tg-show-flow-classes
trustgraph-cli/scripts/tg-show-flows
trustgraph-cli/scripts/tg-start-flow
trustgraph-cli/scripts/tg-stop-flow
2025-04-24 18:57:33 +01:00
cybermaggedon
a9197d11ee
Feature/configure flows (#345)
- Keeps processing in different flows separate so that data can go to different stores / collections etc.
- Potentially supports different processing flows
- Tidies the processing API with common base-classes for e.g. LLMs, and automatic configuration of 'clients' to use the right queue names in a flow
2025-04-22 20:21:38 +01:00
cybermaggedon
298d09f388
Prompt and agent manager, dynamically load configuration from config-svc
- prompt-template takes config from the config-svc, dynamically reloads
  as new config appears.
- agent-react takes config from config-svc, dynamically reloads
- Fixed lack of data in config queue, needed to take the Earliest, not the
  Latest values.
- Changed text-completion and knowledge-query tool to both use 'query'
  as the argument.
- Prompt and agent no longer have command line args to supply
  configuration.
2025-04-02 16:37:08 +01:00
cybermaggedon
1d222235d3
Configuration initialisation (#335)
* - Fixed error reporting in config
- Updated tg-init-pulsar to be able to load initial config to config-svc
- Tweaked API naming and added more config calls

* Tools to dump out prompts and agent tools
2025-04-02 13:52:33 +01:00
cybermaggedon
88eae0a9f0
Fix no version/config at startup (#333) 2025-04-01 20:54:59 +01:00
cybermaggedon
fa09dc319e
Feature/config service (#332)
Configuration service provides an API to change configuration. Complete configuration is pushed down a config queue so that users have a complete copy of config object.
2025-04-01 19:47:05 +01:00
Cyber MacGeddon
b1cefbe1f7 Update setup.py files to prep 0.22 branch 2025-03-31 22:14:38 +01:00
cybermaggedon
8efc3b281d
Fix missing openai symbol / allow specifying base URL (#330) 2025-03-25 21:12:34 +00:00
cybermaggedon
b1ab2938b0
Fix/minor issues (#329)
* Fix MistralAI, symbol error caused it to fail to start

* Update templates
2025-03-25 19:14:56 +00:00
cybermaggedon
482592b976
Added Mistral OCR client (#326)
- Added Mistral OCR client
- Template updates for pdf-ocr
- Template updates for pdf-ocr-mistral
2025-03-22 00:27:20 +00:00
Tyler Oliver
fe422b2b95
Add support for Cassandra auth with SSL check (#318)
Following recommended approach in Datastax documenation I've added the necessary TLS/SSL check

https://docs.datastax.com/en/developer/python-driver/3.17/security/index.html
2025-03-20 22:25:23 +00:00
cybermaggedon
cbfe37fec7
Integration with LM Studio LLM hosting (#323) 2025-03-19 19:07:56 +00:00
cybermaggedon
64e42bed6f
Fix/async send typos (#322) 2025-03-19 00:03:58 +00:00
cybermaggedon
1495666e65
Fix typo (#321) 2025-03-18 23:40:31 +00:00
cybermaggedon
dd4a119f49
Fix broken setup.py (#320) 2025-03-18 21:09:18 +00:00
cybermaggedon
6565adb1ec
Enable AKS integration of text-completion-azure-openai (#319)
Reconfigure so that AZURE_TOKEN, AZURE_MODEL and AZURE_ENDPOINT
can be used to set the token/model/endpoint parameters.  This allows it to
be deployed in K8s and use secrets to set these environment variables
2025-03-18 20:27:45 +00:00
cybermaggedon
a922852017
Fix async/sync load issues with knowledge streaming APIs (#315) 2025-03-15 12:38:10 +00:00
JackColquitt
a46de297c1 More Mistral pieces 2025-03-14 18:12:33 -07:00
JackColquitt
5f5cf8fd07 Added basic Mistral API support 2025-03-14 17:47:59 -07:00
cybermaggedon
ef845d6c9b
Feature/rag parameters (#311)
* Change document-rag and graph-rag processing so that the user can
specify parameters.  Changes in Pulsar services, Pulsar message
schemas, gateway and command-line tools.  User-visible changes in
new parameters on command-line tools.

* Fix bugs, graph-rag working

* Get subgraph truncation in the right place

* Graph RAG and document RAG working and configurable

* Multi-hop path traversal GraphRAG

* Add safety valve for path_size set too high
2025-03-13 00:38:18 +00:00
cybermaggedon
f1559c5944
Feature/librarian (#310)
* Add fields to library schema

* Added list function, incomplete

* Librarian list operation
2025-03-11 16:52:59 +00:00
cybermaggedon
5575e885e5
Fix merge issues with api-key, conflicts with librarian changes (#309) 2025-02-15 12:25:26 +00:00
cybermaggedon
617eb7efd5
Feature/pulsar api key support (#308)
* Add pulsar API token check

* Added missing api_key references

---------

Co-authored-by: Tyler O <4535788+toliver38@users.noreply.github.com>
2025-02-15 11:22:48 +00:00
cybermaggedon
f7df2df266
Feature/librarian (#307)
* Bring QDrant up-to-date

* Tables for data from queue outputs

- Pass single Pulsar client to everything in gateway & librarian
- Pulsar listener-name support in gateway
- PDF and text load working in librarian

* Complete Cassandra schema

* Add librarian support to templates
2025-02-12 23:39:24 +00:00
cybermaggedon
f350abb415
Maint/asyncio (#305)
* Move to asyncio services, even though everything is largely sync
2025-02-11 23:24:46 +00:00
cybermaggedon
a0bf2362f6
Librarian (#304) 2025-02-11 16:01:03 +00:00
Tyler Oliver
e99c0ac238 Add support for Qdrant API Auth (#300)
Added the necessary changes to support API Key in Qdrant Client Query+Storage
- Doc Embeddings
- Graph Embeddings
2025-02-08 11:46:22 +00:00
Tyler Oliver
41ccb6c976 Add user and password auth for Cassandra (#301) 2025-02-08 11:42:14 +00:00
Cyber MacGeddon
15c8afa97b Changed setup.py for 0.21 release branch 2025-01-30 21:40:44 +00:00
cybermaggedon
6c3d2e7f97
Feature/fastembed (#286)
* Copied in base files for fastembed

* Added fastembed implementation

* Added template support for fastembed
2025-01-28 18:26:17 +00:00
cybermaggedon
75a72b0d2d
Fix ollama embeddings client to work (#285) 2025-01-27 23:47:15 +00:00
cybermaggedon
552637c1f7
Fix deps (#284)
* Add cryptography module back in

* Lock in known working version of the AI modules, suspecting a problem
in sentence-transformers with Python 3.12?
2025-01-27 21:57:22 +00:00
cybermaggedon
0ebd770cb5
Back out cryptography (#282) 2025-01-27 19:58:38 +00:00