Commit graph

76 commits

Author SHA1 Message Date
cybermaggedon
89be656990
Release/v1.2 (#457)
* Bump setup.py versions for 1.1

* PoC MCP server (#419)

* Very initial MCP server PoC for TrustGraph

* Put service on port 8000

* Add MCP container and packages to buildout

* Update docs for API/CLI changes in 1.0 (#421)

* Update some API basics for the 0.23/1.0 API change

* Add MCP container push (#425)

* Add command args to the MCP server (#426)

* Host and port parameters

* Added websocket arg

* More docs

* MCP client support (#427)

- MCP client service
- Tool request/response schema
- API gateway support for mcp-tool
- Message translation for tool request & response
- Make mcp-tool using configuration service for information
  about where the MCP services are.

* Feature/react call mcp (#428)

Key Features

  - MCP Tool Integration: Added core MCP tool support with ToolClientSpec and ToolClient classes
  - API Enhancement: New mcp_tool method for flow-specific tool invocation
  - CLI Tooling: New tg-invoke-mcp-tool command for testing MCP integration
  - React Agent Enhancement: Fixed and improved multi-tool invocation capabilities
  - Tool Management: Enhanced CLI for tool configuration and management

Changes

  - Added MCP tool invocation to API with flow-specific integration
  - Implemented ToolClientSpec and ToolClient for tool call handling
  - Updated agent-manager-react to invoke MCP tools with configurable types
  - Enhanced CLI with new commands and improved help text
  - Added comprehensive documentation for new CLI commands
  - Improved tool configuration management

Testing

  - Added tg-invoke-mcp-tool CLI command for isolated MCP integration testing
  - Enhanced agent capability to invoke multiple tools simultaneously

* Test suite executed from CI pipeline (#433)

* Test strategy & test cases

* Unit tests

* Integration tests

* Extending test coverage (#434)

* Contract tests

* Testing embeedings

* Agent unit tests

* Knowledge pipeline tests

* Turn on contract tests

* Increase storage test coverage (#435)

* Fixing storage and adding tests

* PR pipeline only runs quick tests

* Empty configuration is returned as empty list, previously was not in response (#436)

* Update config util to take files as well as command-line text (#437)

* Updated CLI invocation and config model for tools and mcp (#438)

* Updated CLI invocation and config model for tools and mcp

* CLI anomalies

* Tweaked the MCP tool implementation for new model

* Update agent implementation to match the new model

* Fix agent tools, now all tested

* Fixed integration tests

* Fix MCP delete tool params

* Update Python deps to 1.2

* Update to enable knowledge extraction using the agent framework (#439)

* Implement KG extraction agent (kg-extract-agent)

* Using ReAct framework (agent-manager-react)
 
* ReAct manager had an issue when emitting JSON, which conflicts which ReAct manager's own JSON messages, so refactored ReAct manager to use traditional ReAct messages, non-JSON structure.
 
* Minor refactor to take the prompt template client out of prompt-template so it can be more readily used by other modules. kg-extract-agent uses this framework.

* Migrate from setup.py to pyproject.toml (#440)

* Converted setup.py to pyproject.toml

* Modern package infrastructure as recommended by py docs

* Install missing build deps (#441)

* Install missing build deps (#442)

* Implement logging strategy (#444)

* Logging strategy and convert all prints() to logging invocations

* Fix/startup failure (#445)

* Fix loggin startup problems

* Fix logging startup problems (#446)

* Fix logging startup problems (#447)

* Fixed Mistral OCR to use current API (#448)

* Fixed Mistral OCR to use current API

* Added PDF decoder tests

* Fix Mistral OCR ident to be standard pdf-decoder (#450)

* Fix Mistral OCR ident to be standard pdf-decoder

* Correct test

* Schema structure refactor (#451)

* Write schema refactor spec

* Implemented schema refactor spec

* Structure data mvp (#452)

* Structured data tech spec

* Architecture principles

* New schemas

* Updated schemas and specs

* Object extractor

* Add .coveragerc

* New tests

* Cassandra object storage

* Trying to object extraction working, issues exist

* Validate librarian collection (#453)

* Fix token chunker, broken API invocation (#454)

* Fix token chunker, broken API invocation (#455)

* Knowledge load utility CLI (#456)

* Knowledge loader

* More tests
2025-08-18 20:56:09 +01:00
cybermaggedon
3fa004d628
Misc fixes (#413)
- tg-load-turtle, removed cruft output 
- Fixed symbol error in triples_store_service.py
- Removed unnecessary import
2025-06-19 15:17:09 +01:00
cybermaggedon
3c6d875e79
Add token rate measure utility (#412) 2025-06-16 14:18:27 +01:00
cybermaggedon
083702d3d4
Fix turtle loader (#407) 2025-06-02 14:55:22 +01:00
cybermaggedon
8828c00cf6
Fix sample doc loader (#400) 2025-05-28 11:30:53 +01:00
cybermaggedon
ef34d951fe
Renamed default flow from 0000 to default (#395) 2025-05-24 12:27:56 +01:00
Cyber MacGeddon
6dadf30c66 Bump package versions 2025-05-08 22:06:58 +01:00
cybermaggedon
9e9b21068c
ID field is used (#375) 2025-05-08 15:28:27 +01:00
cybermaggedon
31b7ade44d
Feature/knowledge load (#372)
* Switch off retry in Cassandra until we can differentiate retryable errors

* Fix config getvalues

* Loading knowledge cores works
2025-05-08 00:41:45 +01:00
cybermaggedon
fdd9a9a9ae
Fix kg-core downloading (#371) 2025-05-07 14:38:48 +01:00
cybermaggedon
f7123ac57f
tg-put-kg-core command (#369) 2025-05-07 11:13:21 +01:00
cybermaggedon
8080b54328
Knowledge core CLI (#368) 2025-05-07 00:20:59 +01:00
cybermaggedon
d0da122bed
Fix/llms (#366)
* Fix LMStudio, cache documents with tg-load-sample-documents

* Fix Mistral
2025-05-06 16:17:16 +01:00
cybermaggedon
54e475fa3a
Sample docs loader (#365) 2025-05-06 13:43:17 +01:00
cybermaggedon
9e4eb634a4
Fix/chunking not enabled (#364)
* Enable chunking by default in producer

* Fix some issues including uploading large docs
2025-05-06 00:28:20 +01:00
cybermaggedon
844547ab5f
Feature/library cli (#363)
* Major Python client API rework, break down API & colossal class

* Complete rest of library API

* Library CLI support
2025-05-05 11:09:18 +01:00
cybermaggedon
6bf485788a
Fix broken CLI (#360) 2025-05-03 11:07:17 +01:00
cybermaggedon
b3c9dda57f
Fix broken build (#359) 2025-05-03 10:45:42 +01:00
cybermaggedon
3b8b9ea866
Feature/flow api 3 (#358)
* Working mux socket

* Change API to incorporate flow

* Add Flow ID to all relevant CLIs, not completely implemented

* Change tg-processor-state to use API gateway

* Updated all CLIs

* New tg-show-flow-state command

* tg-show-flow-state shows classes too
2025-05-03 10:39:53 +01:00
cybermaggedon
a70ae9793a
Flow API - update gateway (#357)
* Altered API to incorporate Flow IDs, refactored for dynamic start/stop of flows
* Gateway: Split endpoint / dispatcher for maintainability
2025-05-02 21:11:50 +01:00
cybermaggedon
9508ac6c69
Port metering to new API, not tested. (#354)
- Port metering to new API
- Moved price list to configuration
- Added tg-set-token-costs and tg-show-token-costs utils.
2025-04-28 21:26:38 +01:00
cybermaggedon
099018e103
Update package versions (#352) 2025-04-25 19:45:02 +01:00
cybermaggedon
342f555948
Feature/interface descriptions (#348) 2025-04-25 15:59:34 +01:00
cybermaggedon
76e85d895b
Add missing files (#347) 2025-04-24 19:02:51 +01:00
cybermaggedon
3b021720c5
Feature/flow management cli (#346)
Flow management API + various flow management commands

trustgraph-cli/scripts/tg-delete-flow-class
trustgraph-cli/scripts/tg-get-flow-class
trustgraph-cli/scripts/tg-put-flow-class
trustgraph-cli/scripts/tg-show-flow-classes
trustgraph-cli/scripts/tg-show-flows
trustgraph-cli/scripts/tg-start-flow
trustgraph-cli/scripts/tg-stop-flow
2025-04-24 18:57:33 +01:00
cybermaggedon
a759a1771e
Added CLI tg-set-prompt command, can be used to update prompts dynamically. (#338)
Added CLI tg-set-prompt command, can be used to update prompts dynamically. (#338)
2025-04-04 15:43:40 +01:00
cybermaggedon
298d09f388
Prompt and agent manager, dynamically load configuration from config-svc
- prompt-template takes config from the config-svc, dynamically reloads
  as new config appears.
- agent-react takes config from config-svc, dynamically reloads
- Fixed lack of data in config queue, needed to take the Earliest, not the
  Latest values.
- Changed text-completion and knowledge-query tool to both use 'query'
  as the argument.
- Prompt and agent no longer have command line args to supply
  configuration.
2025-04-02 16:37:08 +01:00
cybermaggedon
1d222235d3
Configuration initialisation (#335)
* - Fixed error reporting in config
- Updated tg-init-pulsar to be able to load initial config to config-svc
- Tweaked API naming and added more config calls

* Tools to dump out prompts and agent tools
2025-04-02 13:52:33 +01:00
cybermaggedon
a2c64cad4a
Added config reload handler mechanism, calls a Python method on config (#334) 2025-04-02 00:23:30 +01:00
cybermaggedon
fa09dc319e
Feature/config service (#332)
Configuration service provides an API to change configuration. Complete configuration is pushed down a config queue so that users have a complete copy of config object.
2025-04-01 19:47:05 +01:00
Cyber MacGeddon
b1cefbe1f7 Update setup.py files to prep 0.22 branch 2025-03-31 22:14:38 +01:00
cybermaggedon
ef845d6c9b
Feature/rag parameters (#311)
* Change document-rag and graph-rag processing so that the user can
specify parameters.  Changes in Pulsar services, Pulsar message
schemas, gateway and command-line tools.  User-visible changes in
new parameters on command-line tools.

* Fix bugs, graph-rag working

* Get subgraph truncation in the right place

* Graph RAG and document RAG working and configurable

* Multi-hop path traversal GraphRAG

* Add safety valve for path_size set too high
2025-03-13 00:38:18 +00:00
cybermaggedon
617eb7efd5
Feature/pulsar api key support (#308)
* Add pulsar API token check

* Added missing api_key references

---------

Co-authored-by: Tyler O <4535788+toliver38@users.noreply.github.com>
2025-02-15 11:22:48 +00:00
cybermaggedon
1000a9de8f
Add missing aiohttp dep to trustgraph-cli, needed for knowledge cores (#297)
* Add missing aiohttp dep to trustgraph-cli, needed for knowledge cores
2025-02-04 12:24:46 +00:00
Cyber MacGeddon
15c8afa97b Changed setup.py for 0.21 release branch 2025-01-30 21:40:44 +00:00
Cyber MacGeddon
9bf916f3c6 Update Python files for 0.20 2025-01-16 00:02:18 +00:00
cybermaggedon
bed7423c26
- Added load/save API for document embeddings (#269)
- tg-load-doc-embeds and tg-save-doc-embeds command line utils
2025-01-16 00:00:59 +00:00
cybermaggedon
1280af3eff
Fix/kg core save (#267)
* Add a 'localhost' Pulsar endpoint for docker env

* - Fix broken socket endpoint streamers
- Add unused listener endpoints for publisher/subscriber
- Fix graph embedding serialisation

* Fix GE load

* Remove Gossip settling delay, this is single-node Cassandra.
2025-01-13 14:42:33 +00:00
cybermaggedon
44f8ce8834
Migrate cli utils to REST API (#239)
* Port a number of commands to use API gateway instead of Pulsar

* Ported tg-invoke-agent to websockets API

* Rename the 2 RAG commands: tg-query-... to tg-invoke-...
2025-01-02 19:49:22 +00:00
cybermaggedon
9942f63773
tg-dump-msgpack previously output Python 'representation', now does JSON. (#233) 2024-12-30 10:51:35 +00:00
Cyber MacGeddon
cff90cada1 Prepare for 0.19 2024-12-30 10:44:33 +00:00
Cyber MacGeddon
d6cdce8391 Open 0.18 branch 2024-12-10 22:13:10 +00:00
cybermaggedon
61031270e4
Fix loop logic flaws in loader (#203) 2024-12-09 12:44:30 +00:00
cybermaggedon
803f110891
Timeout alias error (#202) 2024-12-09 00:31:03 +00:00
cybermaggedon
cf8c76b5c6
Fix/save core hang (#201)
* Working around an exception class change in Python 3.11
2024-12-09 00:01:01 +00:00
cybermaggedon
fd3db3c925
Feature/tidy kg load save (#198)
* Clean exit on ctrl-C
* More functionality in dump
* Dump some metadata
2024-12-06 15:16:09 +00:00
cybermaggedon
2818ec9f23
Fix header (#194) 2024-12-06 08:50:49 +00:00
cybermaggedon
6d200c79c5
Feature/wikipedia ddg (#185)
API-side support for Wikipedia, DBpedia and internet search functions  This incorporates a refactor of the API code to break it up, separate classes for endpoints to reduce duplication
2024-12-02 17:41:30 +00:00
Cyber MacGeddon
c844d805e5 Setup for release 0.17 branch 2024-11-29 17:03:31 +00:00
cybermaggedon
cf564ed147
PDF loader symbol error fixed (#183) 2024-11-28 21:26:36 +00:00