Commit graph

21 commits

Author SHA1 Message Date
cybermaggedon
0e03bc05a4
Refactor rate limit handling (#280)
* - Refactored retry for rate limits into the base class
- ConsumerProducer is derived from Consumer to simplify code
- Added rate_limit_count metrics for rate limit events

* Add rate limit events to VertexAI and Google AI Studio

* Added Grafana rate limit dashboard

* Add rate limit handling to all LLMs
2025-01-27 17:04:49 +00:00
cybermaggedon
cd9a208432
Processors use shared queues, means there can be more than process on a queue to share load (#265) 2025-01-11 18:10:04 +00:00
cybermaggedon
6aa212061d
Fix/document embeddings (#247)
* Update schema for doc embeddings

* Rename embeddings-vectorize to graph-embeddings

* Added document-embeddings processor (broken, needs fixing)

* Added scripts

* Fixed DE queue schema

* Add missing DE process

* Fix doc RAG processing, put graph-rag and doc-rag in appropriate component files.
2025-01-04 21:51:28 +00:00
cybermaggedon
a458d57af2
Feature/refactor entity embeddings (#235)
* Make schema changes
* Core entity context flow in place
* extract-def outputs entity contexts
* Refactored qdrant write
* Refactoring of all vector stores in place
2024-12-30 12:53:19 +00:00
cybermaggedon
187b0e6581
Fix/websocket capacity increase (#230)
* Fix invalid variable name invocation
* Fix error responses in websockets
* Increase websocket limits to 50MB max message.  Turn on Pulsar chunking by default.
2024-12-29 18:08:12 +00:00
cybermaggedon
e3d06ab80b
Fix isinstance test on null values (#192)
Co-authored-by: Mark Adams <mark.adams@surevine.com>
2024-12-04 14:42:55 +00:00
cybermaggedon
6d200c79c5
Feature/wikipedia ddg (#185)
API-side support for Wikipedia, DBpedia and internet search functions  This incorporates a refactor of the API code to break it up, separate classes for endpoints to reduce duplication
2024-12-02 17:41:30 +00:00
cybermaggedon
212102c61c
Tweak queue names (#184) 2024-11-29 17:17:20 +00:00
cybermaggedon
887fafcf8c
Fix/core save api (#172)
* Acknowledge messaages from Pulsar, doh!
* Change API to deliver a boolean e if value is an entity
* Change loaders to use new API
* Changes, entity-aware API is complete
2024-11-26 16:46:38 +00:00
cybermaggedon
ae1264f5c4
Add Python support to calling the API (#169) 2024-11-22 15:55:32 +00:00
cybermaggedon
36cdeab588
Feature/agent manager (#146)
* Agent schema
* Agent working through client
* Add agent-manager-react command line
* test-agent test script
* Add tg-invoke-agent CLI
2024-11-10 11:44:01 +00:00
cybermaggedon
ae8661fe2e
Fix/improve command line help (#145)
* Make command line consistent, fix incorrect documentation.

* Improve tg-invoke-prompt help
2024-11-08 18:14:14 +00:00
cybermaggedon
65cda7b276
Implement system in text completion API (#137)
* Add system prompt to LLM invocation

* Added system parameter to LLMs

* Added to Bedrock and VertexAI
2024-11-05 22:46:17 +00:00
cybermaggedon
9ebfe0d94a
Maint/knowledge load collections (#132)
* Add user/collection support to knowledge loaders

* Fix timeouts
2024-10-31 16:28:03 +00:00
cybermaggedon
24d099793d
Feature/doc metadata labels (#130)
* Add schema load util

* Added a sample schema turtle file will be useful for future testing and
tutorials.

* Fixed graph label metadata confusion, was created incorrect subjectOf
edges.
2024-10-29 21:18:02 +00:00
cybermaggedon
b07d539802
- Fix topics extract (#126)
- Add schemas to extraction
2024-10-27 08:22:12 +00:00
cybermaggedon
1e137768ca
Prompt refactor (#125)
* Prompt manager integrated and working with 6 tests
* Updated templates to for prompt-template update
2024-10-26 22:17:43 +01:00
cybermaggedon
7954e863cc
Feature: document metadata (#123)
* Rework metadata structure in processing messages to be a subgraph
* Add subgraph creation for tg-load-pdf and tg-load-text based on command-line passing of doc attributes
* Document metadata is added to knowledge graph with subjectOf linkage to extracted entities
2024-10-23 18:04:04 +01:00
cybermaggedon
b0f4c58200
Feature / collections (#96)
* Update schema defs for source -> metadata
* Migrate to use metadata part of schema, also add metadata to triples & vecs
* Add user/collection metadata to query
* Use user/collection in RAG
* Write and query working on triples
2024-10-02 18:14:29 +01:00
cybermaggedon
f00baab1b8
Maint/fix build env (#84)
* Put README placeholders for packages in place
* Bump version
2024-09-30 19:47:09 +01:00
cybermaggedon
9b91d5eee3
Feature/pkgsplit (#83)
* Starting to spawn base package
* More package hacking
* Bedrock and VertexAI
* Parquet split
* Updated templates
* Utils
2024-09-30 19:36:09 +01:00