Commit graph

271 commits

Author SHA1 Message Date
cybermaggedon
2d8dbf4cdb
Move GAIStudio to vertexai package to simplify deps (#639) 2026-02-20 08:46:29 +00:00
cybermaggedon
769c56bbea
Use ClientError & code to determine 429 error (#638) 2026-02-20 08:00:07 +00:00
cybermaggedon
89b69fdb08
Fix weird Onttology URI issue (#637) 2026-02-16 19:18:29 +00:00
cybermaggedon
d886358be6
Entity & triple batch size limits (#635)
* Entities and triples are emitted in batches with a batch limit to manage
overloading downstream.

* Update tests
2026-02-16 17:38:03 +00:00
cybermaggedon
fe389354f6
Fix d/g attribute error (#634) 2026-02-16 13:34:08 +00:00
cybermaggedon
00c1ca681b
Entity-centric graph (#633)
* Tech spec for new entity-centric graph schema

* Graph implementation
2026-02-16 13:26:43 +00:00
cybermaggedon
f24f1ebd80
Migrate to VertexAI to google-genai SDK from deprecated library (#632)
* Migrate to VertexAI to google-genai SDK from deprecated library

* Fix tests, mock the correct API
2026-02-09 20:43:33 +00:00
cybermaggedon
2781c7d87c
Fix LLM metrics (#631)
* Fix mistral metrics

* Fix to other models
2026-02-09 19:35:42 +00:00
cybermaggedon
4fca97d555
Output the entity term as well as its definition as entity contexts (#629) 2026-02-09 15:18:05 +00:00
cybermaggedon
8574861196
Protect null embeddings - v2.0 (#627)
* Don't emit graph embeddings if there aren't any.

* Don't store graph embeddings in a knowledge store if there's an empty list.

* Translate between Cassandra's 'null' representing an empty list and an
  empty list which is what the surrounding code wants (and stored in the
  first place).

* Avoid emitting empty embedding lists

* Avoid output empty triple lists

* Fix tests
2026-02-09 14:57:36 +00:00
cybermaggedon
6bf08c3ace
Feature/more cli diags (#624)
* CLI tools for tg-invoke-graph-embeddings, tg-invoke-document-embeddings,
and tg-invoke-embeddings.  Just useful for diagnostics.

* Fix tg-load-knowledge
2026-02-04 14:10:30 +00:00
cybermaggedon
cf0daedefa
Changed schema for Value -> Term, majorly breaking change (#622)
* Changed schema for Value -> Term, majorly breaking change

* Following the schema change, Value -> Term into all processing

* Updated Cassandra for g, p, s, o index patterns (7 indexes)

* Reviewed and updated all tests

* Neo4j, Memgraph and FalkorDB remain broken, will look at once settled down
2026-01-27 13:48:08 +00:00
cybermaggedon
e214eb4e02
Feature/prompts jsonl (#619)
* Tech spec

* JSONL implementation complete

* Updated prompt client users

* Fix tests
2026-01-26 17:38:00 +00:00
cybermaggedon
e4f0013841
Open 1.9 branch (#620) 2026-01-26 17:36:25 +00:00
cybermaggedon
11f41b07ab
Get neo4j to use limit (#618)
* Get neo4j to use limit

* Fix tests - they we exact matching on query strings
2026-01-22 15:16:34 +00:00
cybermaggedon
58c00149a7
Normalise URLs so that / suffix is optional (#617) 2026-01-19 13:34:33 +00:00
cybermaggedon
83ea15dae7
Fixed flows/flow key issue in config (#616) 2026-01-16 00:10:44 +00:00
cybermaggedon
62b754d788
Fix flow loading (#611) 2026-01-14 16:23:15 +00:00
cybermaggedon
387afee7b7
Fix load-doc (#610) 2026-01-14 15:46:29 +00:00
cybermaggedon
b08db761d7
Fix config inconsistency (#609)
* Plural/singular confusion in config key

* Flow class vs flow blueprint nomenclature change

* Update docs & CLI to reflect the above
2026-01-14 12:31:40 +00:00
cybermaggedon
807f6cc4e2
Fix non streaming RAG problems (#607)
* Fix non-streaming failure in RAG services

* Fix non-streaming failure in API

* Fix agent non-streaming messaging

* Agent messaging unit & contract tests
2026-01-12 18:45:52 +00:00
cybermaggedon
16a5cf966a
Fix agent streaming tool failure (#602)
* Fix agent streaming linkage

* Update tests
2026-01-06 23:00:50 +00:00
cybermaggedon
f0c95a4c5e
Fix streaming API niggles (#599)
* Fix end-of-stream anomally with some graph-rag and document-rag

* Fix gateway translators dropping responses
2026-01-06 16:41:35 +00:00
cybermaggedon
7a5bf47959
Fix collection existence test logic (#597) 2026-01-05 16:31:26 +00:00
cybermaggedon
ae13190093
Address legacy issues in storage management (#595)
* Removed legacy storage management cruft.  Tidied tech specs.

* Fix deletion of last collection

* Storage processor ignores data on the queue which is for a deleted collection

* Updated tests
2026-01-05 13:45:14 +00:00
cybermaggedon
25563bae3c
Change MinIO integration options in librarian to be more generic - to support a Garage integration (#594)
* Tweak object store parameters to be more generic for other S3-type store integration

* Update librarian to have region & SSL params

* Update MinIO migration tech spec
2025-12-27 18:01:51 +00:00
cybermaggedon
34eb083836
Messaging fabric plugins (#592)
* Plugin architecture for messaging fabric

* Schemas use a technology neutral expression

* Schemas strictness has uncovered some incorrect schema use which is fixed
2025-12-17 21:40:43 +00:00
Cyber MacGeddon
1865b3f3c8 Start 1.8 release branch 2025-12-17 21:32:13 +00:00
cybermaggedon
2b5ba68f00
Add model to meter metrics (#589) 2025-12-10 11:34:48 +00:00
cybermaggedon
727b6bc9d6
Add service ID to log entry instead of module name (#588) 2025-12-10 11:07:43 +00:00
cybermaggedon
aebdf9444b
Fix incorrect Cassandra config invocation (#587) 2025-12-10 10:55:14 +00:00
cybermaggedon
f12fcc2652
Loki logging (#586)
* Consolidate logging into a single module

* Added Loki logging

* Update tech spec

* Add processor label

* Fix recursive log entries, logging Loki"s internals
2025-12-09 23:24:41 +00:00
cybermaggedon
39f6a8b940
Fix/queue configurations (#585)
* Fix config-svc startup dupe CLI args

* Fix missing params on collection service

* Fix collection management handling
2025-12-06 14:54:47 +00:00
cybermaggedon
ba95fa226b
Gateway queue overrides (#584) 2025-12-06 11:01:20 +00:00
cybermaggedon
7d07f802a8
Basic multitenant support (#583)
* Tech spec

* Address multi-tenant queue option problems in CLI

* Modified collection service to use config

* Changed storage management to use the config service definition
2025-12-05 21:45:30 +00:00
cybermaggedon
b957004db9
Feature/improve ontology extract (#576)
* Tech spec to change ontology extraction

* Ontology extract refactoring
2025-12-03 13:36:10 +00:00
Cyber MacGeddon
98aaa4f67e Configure for 1.7 release branch 2025-12-03 09:46:55 +00:00
cybermaggedon
72cb1c98e0
Fix tests (#571) 2025-11-28 16:37:01 +00:00
cybermaggedon
e24de6081f
Fix streaming agent interactions (#570)
* Fix observer, thought streaming

* Fix end of message indicators

* Remove double-delivery of answer
2025-11-28 16:25:57 +00:00
cybermaggedon
1948edaa50
Streaming rag responses (#568)
* Tech spec for streaming RAG

* Support for streaming Graph/Doc RAG
2025-11-26 19:47:39 +00:00
cybermaggedon
b1cc724f7d
Streaming LLM part 2 (#567)
* Updates for agent API with streaming support

* Added tg-dump-queues tool to dump Pulsar queues to a log

* Updated tg-invoke-agent, incremental output

* Queue dumper CLI - might be useful for debug

* Updating for tests
2025-11-26 15:16:17 +00:00
cybermaggedon
310a2deb06
Feature/streaming llm phase 1 (#566)
* Tidy up duplicate tech specs in doc directory

* Streaming LLM text-completion service tech spec.

* text-completion and prompt interfaces

* streaming change applied to all LLMs, so far tested with VertexAI

* Skip Pinecone unit tests, upstream module issue is affecting things, tests are passing again

* Added agent streaming, not working and has broken tests
2025-11-26 09:59:10 +00:00
cybermaggedon
943a9d83b0
Fix Minio incompatible library change (#565) 2025-11-25 22:45:48 +00:00
cybermaggedon
97d8b84d7f
Open 1.6 release branch (#564) 2025-11-24 10:05:29 +00:00
cybermaggedon
7501db01f1
Reconcile master with 1.6 (#563)
- Reconcile all master changes (documentation) to the 1.6 release branch
2025-11-24 10:02:30 +00:00
cybermaggedon
3580e7a7ae
Remove some 'unnecessary' parameters from OpenAI invocation (#561)
* Remove some 'unnecessary' parameters from OpenAI invocation.  The OpenAI
API is getting complicated with the API and SDK changing on OpenAI's end,
but this not getting mapped through to other services which are 'compatible'
with OpenAI.

* Update OpenAI test for this change

* Trying running tests with Python 3.13
2025-11-20 17:56:31 +00:00
cybermaggedon
2baf21c5e1
Collection deletion batch error fixed (#559)
Collection deletion caused a batch error, this reduces batch size on deletion
2025-11-13 16:29:40 +00:00
cybermaggedon
c69f5207a4
OntoRAG: Ontology-Based Knowledge Extraction and Query Technical Specification (#523)
* Onto-rag tech spec

* New processor kg-extract-ontology, use 'ontology' objects from config to guide triple extraction

* Also entity contexts

* Integrate with ontology extractor from workbench

This is first phase, the extraction is tested and working, also GraphRAG with the extracted knowledge works
2025-11-12 20:38:08 +00:00
cybermaggedon
4c3db4dbbe
MCP auth for the simple case (#557)
* MCP auth token header

* Mention limitations

* Fix AgentStep schema error by converting argument values to strings.

* Added tests for MCP auth and agent step parsing
2025-11-11 12:28:53 +00:00
cybermaggedon
d9d4c91363
Dynamic embeddings model (#556)
* Dynamic embeddings model selection

* Added tests

* HF embeddings are skipped, tests don't run with that package currently tests
2025-11-10 20:38:01 +00:00