Release/v1.2 (#457)

* Bump setup.py versions for 1.1

* PoC MCP server (#419)

* Very initial MCP server PoC for TrustGraph

* Put service on port 8000

* Add MCP container and packages to buildout

* Update docs for API/CLI changes in 1.0 (#421)

* Update some API basics for the 0.23/1.0 API change

* Add MCP container push (#425)

* Add command args to the MCP server (#426)

* Host and port parameters

* Added websocket arg

* More docs

* MCP client support (#427)

- MCP client service
- Tool request/response schema
- API gateway support for mcp-tool
- Message translation for tool request & response
- Make mcp-tool using configuration service for information
  about where the MCP services are.

* Feature/react call mcp (#428)

Key Features

  - MCP Tool Integration: Added core MCP tool support with ToolClientSpec and ToolClient classes
  - API Enhancement: New mcp_tool method for flow-specific tool invocation
  - CLI Tooling: New tg-invoke-mcp-tool command for testing MCP integration
  - React Agent Enhancement: Fixed and improved multi-tool invocation capabilities
  - Tool Management: Enhanced CLI for tool configuration and management

Changes

  - Added MCP tool invocation to API with flow-specific integration
  - Implemented ToolClientSpec and ToolClient for tool call handling
  - Updated agent-manager-react to invoke MCP tools with configurable types
  - Enhanced CLI with new commands and improved help text
  - Added comprehensive documentation for new CLI commands
  - Improved tool configuration management

Testing

  - Added tg-invoke-mcp-tool CLI command for isolated MCP integration testing
  - Enhanced agent capability to invoke multiple tools simultaneously

* Test suite executed from CI pipeline (#433)

* Test strategy & test cases

* Unit tests

* Integration tests

* Extending test coverage (#434)

* Contract tests

* Testing embeedings

* Agent unit tests

* Knowledge pipeline tests

* Turn on contract tests

* Increase storage test coverage (#435)

* Fixing storage and adding tests

* PR pipeline only runs quick tests

* Empty configuration is returned as empty list, previously was not in response (#436)

* Update config util to take files as well as command-line text (#437)

* Updated CLI invocation and config model for tools and mcp (#438)

* Updated CLI invocation and config model for tools and mcp

* CLI anomalies

* Tweaked the MCP tool implementation for new model

* Update agent implementation to match the new model

* Fix agent tools, now all tested

* Fixed integration tests

* Fix MCP delete tool params

* Update Python deps to 1.2

* Update to enable knowledge extraction using the agent framework (#439)

* Implement KG extraction agent (kg-extract-agent)

* Using ReAct framework (agent-manager-react)
 
* ReAct manager had an issue when emitting JSON, which conflicts which ReAct manager's own JSON messages, so refactored ReAct manager to use traditional ReAct messages, non-JSON structure.
 
* Minor refactor to take the prompt template client out of prompt-template so it can be more readily used by other modules. kg-extract-agent uses this framework.

* Migrate from setup.py to pyproject.toml (#440)

* Converted setup.py to pyproject.toml

* Modern package infrastructure as recommended by py docs

* Install missing build deps (#441)

* Install missing build deps (#442)

* Implement logging strategy (#444)

* Logging strategy and convert all prints() to logging invocations

* Fix/startup failure (#445)

* Fix loggin startup problems

* Fix logging startup problems (#446)

* Fix logging startup problems (#447)

* Fixed Mistral OCR to use current API (#448)

* Fixed Mistral OCR to use current API

* Added PDF decoder tests

* Fix Mistral OCR ident to be standard pdf-decoder (#450)

* Fix Mistral OCR ident to be standard pdf-decoder

* Correct test

* Schema structure refactor (#451)

* Write schema refactor spec

* Implemented schema refactor spec

* Structure data mvp (#452)

* Structured data tech spec

* Architecture principles

* New schemas

* Updated schemas and specs

* Object extractor

* Add .coveragerc

* New tests

* Cassandra object storage

* Trying to object extraction working, issues exist

* Validate librarian collection (#453)

* Fix token chunker, broken API invocation (#454)

* Fix token chunker, broken API invocation (#455)

* Knowledge load utility CLI (#456)

* Knowledge loader

* More tests

.coveragerc

@@ -0,0 +1,35 @@
+[run]
+source = 
+    trustgraph-base/trustgraph
+    trustgraph-flow/trustgraph
+    trustgraph-bedrock/trustgraph
+    trustgraph-vertexai/trustgraph
+    trustgraph-embeddings-hf/trustgraph
+omit = 
+    */tests/*
+    */test_*
+    */conftest.py
+    */__pycache__/*
+    */venv/*
+    */env/*
+    */site-packages/*
+
+# Disable coverage warnings for contract tests
+disable_warnings = no-data-collected
+
+[report]
+exclude_lines =
+    pragma: no cover
+    def __repr__
+    raise AssertionError
+    raise NotImplementedError
+    if __name__ == .__main__.:
+    class .*\(Protocol\):
+    @(abc\.)?abstractmethod
+
+[html]
+directory = htmlcov
+skip_covered = False
+
+[xml]
+output = coverage.xml

.github/workflows/pull-request.yaml

@@ -9,12 +9,51 @@ permissions:
 
 jobs:
 
-  container-push:
+  test:
 
-    name: Do nothing
+    name: Run tests
     runs-on: ubuntu-latest
 
+    container:
+      image: python:3.12
+
     steps:
       - name: Checkout
         uses: actions/checkout@v3
 
+      - name: Setup packages
+        run: make update-package-versions VERSION=1.2.999
+
+      - name: Setup environment
+        run: python3 -m venv env
+
+      - name: Invoke environment
+        run: . env/bin/activate
+
+      - name: Install trustgraph-base
+        run: (cd trustgraph-base; pip install .)
+
+      - name: Install trustgraph-cli
+        run: (cd trustgraph-cli; pip install .)
+
+      - name: Install trustgraph-flow
+        run: (cd trustgraph-flow; pip install .)
+
+      - name: Install trustgraph-vertexai
+        run: (cd trustgraph-vertexai; pip install .)
+
+      - name: Install trustgraph-bedrock
+        run: (cd trustgraph-bedrock; pip install .)
+
+      - name: Install some stuff
+        run: pip install pytest pytest-cov pytest-asyncio pytest-mock testcontainers
+
+      - name: Unit tests
+        run: pytest tests/unit
+
+      - name: Integration tests (cut the out the long-running tests)
+        run: pytest tests/integration -m 'not slow'
+
+      - name: Contract tests
+        run: pytest tests/contract
+

.github/workflows/release.yaml

@@ -31,6 +31,9 @@ jobs:
         id: version
         run: echo VERSION=$(git describe --exact-match --tags | sed 's/^v//') >> $GITHUB_OUTPUT
 
+      - name: Install dependencies
+        run: pip install build wheel
+
       - name: Build packages
         run: make packages VERSION=${{ steps.version.outputs.VERSION }}
 

.gitignore

@@ -6,11 +6,12 @@ env/
 *.parquet
 templates/values/version.jsonnet
 trustgraph-base/trustgraph/base_version.py
+trustgraph-cli/trustgraph/cli_version.py
 trustgraph-bedrock/trustgraph/bedrock_version.py
 trustgraph-embeddings-hf/trustgraph/embeddings_hf_version.py
 trustgraph-flow/trustgraph/flow_version.py
 trustgraph-ocr/trustgraph/ocr_version.py
 trustgraph-parquet/trustgraph/parquet_version.py
 trustgraph-vertexai/trustgraph/vertexai_version.py
-trustgraph-cli/trustgraph/
+trustgraph-mcp/trustgraph/mcp_version.py
 vertexai/

Makefile

@@ -17,17 +17,19 @@ wheels:
 	pip3 wheel --no-deps --wheel-dir dist trustgraph-embeddings-hf/
 	pip3 wheel --no-deps --wheel-dir dist trustgraph-cli/
 	pip3 wheel --no-deps --wheel-dir dist trustgraph-ocr/
+	pip3 wheel --no-deps --wheel-dir dist trustgraph-mcp/
 
 packages: update-package-versions
 	rm -rf dist/
-	cd trustgraph && python3 setup.py sdist --dist-dir ../dist/
+	cd trustgraph && python -m build --sdist --outdir ../dist/
-	cd trustgraph-base && python3 setup.py sdist --dist-dir ../dist/
+	cd trustgraph-base && python -m build --sdist --outdir ../dist/
-	cd trustgraph-flow && python3 setup.py sdist --dist-dir ../dist/
+	cd trustgraph-flow && python -m build --sdist --outdir ../dist/
-	cd trustgraph-vertexai && python3 setup.py sdist --dist-dir ../dist/
+	cd trustgraph-vertexai && python -m build --sdist --outdir ../dist/
-	cd trustgraph-bedrock && python3 setup.py sdist --dist-dir ../dist/
+	cd trustgraph-bedrock && python -m build --sdist --outdir ../dist/
-	cd trustgraph-embeddings-hf && python3 setup.py sdist --dist-dir ../dist/
+	cd trustgraph-embeddings-hf && python -m build --sdist --outdir ../dist/
-	cd trustgraph-cli && python3 setup.py sdist --dist-dir ../dist/
+	cd trustgraph-cli && python -m build --sdist --outdir ../dist/
-	cd trustgraph-ocr && python3 setup.py sdist --dist-dir ../dist/
+	cd trustgraph-ocr && python -m build --sdist --outdir ../dist/
+	cd trustgraph-mcp && python -m build --sdist --outdir ../dist/
 
 pypi-upload:
 	twine upload dist/*-${VERSION}.*
@@ -45,6 +47,7 @@ update-package-versions:
 	echo __version__ = \"${VERSION}\" > trustgraph-cli/trustgraph/cli_version.py
 	echo __version__ = \"${VERSION}\" > trustgraph-ocr/trustgraph/ocr_version.py
 	echo __version__ = \"${VERSION}\" > trustgraph/trustgraph/trustgraph_version.py
+	echo __version__ = \"${VERSION}\" > trustgraph-mcp/trustgraph/mcp_version.py
 
 container: update-package-versions
 	${DOCKER} build -f containers/Containerfile.base \
@@ -59,12 +62,16 @@ container: update-package-versions
 	    -t ${CONTAINER_BASE}/trustgraph-hf:${VERSION} .
 	${DOCKER} build -f containers/Containerfile.ocr \
 	    -t ${CONTAINER_BASE}/trustgraph-ocr:${VERSION} .
+	${DOCKER} build -f containers/Containerfile.mcp \
+	    -t ${CONTAINER_BASE}/trustgraph-mcp:${VERSION} .
 
 some-containers:
 	${DOCKER} build -f containers/Containerfile.base \
 	    -t ${CONTAINER_BASE}/trustgraph-base:${VERSION} .
 	${DOCKER} build -f containers/Containerfile.flow \
 	    -t ${CONTAINER_BASE}/trustgraph-flow:${VERSION} .
+#	${DOCKER} build -f containers/Containerfile.mcp \
+#	    -t ${CONTAINER_BASE}/trustgraph-mcp:${VERSION} .
 #	${DOCKER} build -f containers/Containerfile.vertexai \
 #	    -t ${CONTAINER_BASE}/trustgraph-vertexai:${VERSION} .
 #	${DOCKER} build -f containers/Containerfile.bedrock \
@@ -87,6 +94,7 @@ push:
 	${DOCKER} push ${CONTAINER_BASE}/trustgraph-vertexai:${VERSION}
 	${DOCKER} push ${CONTAINER_BASE}/trustgraph-hf:${VERSION}
 	${DOCKER} push ${CONTAINER_BASE}/trustgraph-ocr:${VERSION}
+	${DOCKER} push ${CONTAINER_BASE}/trustgraph-mcp:${VERSION}
 
 clean:
 	rm -rf wheels/
@@ -116,7 +124,7 @@ JSONNET_FLAGS=-J templates -J .
 
 update-templates: update-dcs
 
-JSON_TO_YAML=python3 -c 'import sys, yaml, json; j=json.loads(sys.stdin.read()); print(yaml.safe_dump(j))'
+JSON_TO_YAML=python -c 'import sys, yaml, json; j=json.loads(sys.stdin.read()); print(yaml.safe_dump(j))'
 
 update-dcs: set-version
 	for graph in ${GRAPHS}; do \

TESTS.md

@@ -0,0 +1,590 @@
+# TrustGraph Test Suite
+
+This document provides instructions for running and maintaining the TrustGraph test suite.
+
+## Overview
+
+The TrustGraph test suite follows the testing strategy outlined in [TEST_STRATEGY.md](TEST_STRATEGY.md) and implements the test cases defined in [TEST_CASES.md](TEST_CASES.md). The tests are organized into unit tests, integration tests, and performance tests.
+
+## Test Structure
+
+```
+tests/
+├── unit/
+│   ├── test_text_completion/
+│   │   ├── test_vertexai_processor.py
+│   │   ├── conftest.py
+│   │   └── __init__.py
+│   ├── test_embeddings/
+│   ├── test_storage/
+│   └── test_query/
+├── integration/
+│   ├── test_flows/
+│   └── test_databases/
+├── fixtures/
+│   ├── messages.py
+│   ├── configs.py
+│   └── mocks.py
+├── requirements.txt
+├── pytest.ini
+└── conftest.py
+```
+
+## Prerequisites
+
+### Install TrustGraph Packages
+
+The tests require TrustGraph packages to be installed. You can use the provided scripts:
+
+#### Option 1: Automated Setup (Recommended)
+```bash
+# From the project root directory - runs all setup steps
+./run_tests.sh
+```
+
+#### Option 2: Step-by-step Setup
+```bash
+# Check what imports are working
+./check_imports.py
+
+# Install TrustGraph packages
+./install_packages.sh
+
+# Verify imports work
+./check_imports.py
+
+# Install test dependencies
+cd tests/
+pip install -r requirements.txt
+cd ..
+```
+
+#### Option 3: Manual Installation
+```bash
+# Install base package first (required by others)
+cd trustgraph-base
+pip install -e .
+cd ..
+
+# Install vertexai package (depends on base)
+cd trustgraph-vertexai  
+pip install -e .
+cd ..
+
+# Install flow package (for additional components)
+cd trustgraph-flow
+pip install -e .
+cd ..
+```
+
+### Install Test Dependencies
+
+```bash
+cd tests/
+pip install -r requirements.txt
+```
+
+### Required Dependencies
+
+- `pytest>=7.0.0` - Testing framework
+- `pytest-asyncio>=0.21.0` - Async testing support
+- `pytest-mock>=3.10.0` - Mocking utilities
+- `pytest-cov>=4.0.0` - Coverage reporting
+- `google-cloud-aiplatform>=1.25.0` - Google Cloud dependencies
+- `google-auth>=2.17.0` - Authentication
+- `google-api-core>=2.11.0` - API core
+- `pulsar-client>=3.0.0` - Pulsar messaging
+- `prometheus-client>=0.16.0` - Metrics
+
+## Running Tests
+
+### Basic Test Execution
+
+```bash
+# Run all tests
+pytest
+
+# Run tests with verbose output
+pytest -v
+
+# Run specific test file
+pytest tests/unit/test_text_completion/test_vertexai_processor.py
+
+# Run specific test class
+pytest tests/unit/test_text_completion/test_vertexai_processor.py::TestVertexAIProcessorInitialization
+
+# Run specific test method
+pytest tests/unit/test_text_completion/test_vertexai_processor.py::TestVertexAIProcessorInitialization::test_processor_initialization_with_valid_credentials
+```
+
+### Test Categories
+
+```bash
+# Run only unit tests
+pytest -m unit
+
+# Run only integration tests
+pytest -m integration
+
+# Run only VertexAI tests
+pytest -m vertexai
+
+# Exclude slow tests
+pytest -m "not slow"
+```
+
+### Coverage Reports
+
+```bash
+# Run tests with coverage
+pytest --cov=trustgraph
+
+# Generate HTML coverage report
+pytest --cov=trustgraph --cov-report=html
+
+# Generate terminal coverage report
+pytest --cov=trustgraph --cov-report=term-missing
+
+# Fail if coverage is below 80%
+pytest --cov=trustgraph --cov-fail-under=80
+```
+
+## VertexAI Text Completion Tests
+
+### Test Implementation
+
+The VertexAI text completion service tests are located in:
+- **Main test file**: `tests/unit/test_text_completion/test_vertexai_processor.py`
+- **Fixtures**: `tests/unit/test_text_completion/conftest.py`
+
+### Test Coverage
+
+The VertexAI tests include **139 test cases** covering:
+
+#### 1. Processor Initialization Tests (6 tests)
+- Service account credential loading
+- Model configuration (Gemini models)
+- Custom parameters (temperature, max_output, region)
+- Generation config and safety settings
+
+```bash
+# Run initialization tests
+pytest tests/unit/test_text_completion/test_vertexai_processor.py::TestVertexAIProcessorInitialization -v
+```
+
+#### 2. Message Processing Tests (5 tests)
+- Simple text completion
+- System instructions handling
+- Long context processing
+- Empty prompt handling
+
+```bash
+# Run message processing tests
+pytest tests/unit/test_text_completion/test_vertexai_processor.py::TestVertexAIMessageProcessing -v
+```
+
+#### 3. Safety Filtering Tests (2 tests)
+- Safety settings configuration
+- Blocked content handling
+
+```bash
+# Run safety filtering tests
+pytest tests/unit/test_text_completion/test_vertexai_processor.py::TestVertexAISafetyFiltering -v
+```
+
+#### 4. Error Handling Tests (7 tests)
+- Rate limiting (`ResourceExhausted` → `TooManyRequests`)
+- Authentication errors
+- Generic exceptions
+- Model not found errors
+- Quota exceeded errors
+- Token limit errors
+
+```bash
+# Run error handling tests
+pytest tests/unit/test_text_completion/test_vertexai_processor.py::TestVertexAIErrorHandling -v
+```
+
+#### 5. Metrics Collection Tests (4 tests)
+- Token usage tracking
+- Request duration measurement
+- Error rate collection
+- Cost calculation basis
+
+```bash
+# Run metrics collection tests
+pytest tests/unit/test_text_completion/test_vertexai_processor.py::TestVertexAIMetricsCollection -v
+```
+
+### Running All VertexAI Tests
+
+#### Option 1: Simple Tests (Recommended for getting started)
+```bash
+# Run simple tests that don't require full TrustGraph infrastructure
+./run_simple_tests.sh
+
+# Or run manually:
+pytest tests/unit/test_text_completion/test_vertexai_simple.py -v
+pytest tests/unit/test_text_completion/test_vertexai_core.py -v
+```
+
+#### Option 2: Full Infrastructure Tests
+```bash
+# Run all VertexAI tests (requires full TrustGraph setup)
+pytest tests/unit/test_text_completion/test_vertexai_processor.py -v
+
+# Run with coverage
+pytest tests/unit/test_text_completion/test_vertexai_processor.py --cov=trustgraph.model.text_completion.vertexai
+
+# Run with detailed output
+pytest tests/unit/test_text_completion/test_vertexai_processor.py -v -s
+```
+
+#### Option 3: All VertexAI Tests
+```bash
+# Run all VertexAI-related tests
+pytest tests/unit/test_text_completion/ -k "vertexai" -v
+```
+
+## Test Configuration
+
+### Pytest Configuration
+
+The test suite uses the following configuration in `pytest.ini`:
+
+```ini
+[tool:pytest]
+testpaths = tests
+python_files = test_*.py
+python_classes = Test*
+python_functions = test_*
+addopts = 
+    -v
+    --tb=short
+    --strict-markers
+    --disable-warnings
+    --cov=trustgraph
+    --cov-report=html
+    --cov-report=term-missing
+    --cov-fail-under=80
+asyncio_mode = auto
+markers =
+    slow: marks tests as slow (deselect with '-m "not slow"')
+    integration: marks tests as integration tests
+    unit: marks tests as unit tests
+    vertexai: marks tests as vertex ai specific tests
+```
+
+### Test Markers
+
+Use pytest markers to categorize and filter tests:
+
+```python
+@pytest.mark.unit
+@pytest.mark.vertexai
+async def test_vertexai_functionality():
+    pass
+
+@pytest.mark.integration
+@pytest.mark.slow
+async def test_end_to_end_flow():
+    pass
+```
+
+## Test Development Guidelines
+
+### Following TEST_STRATEGY.md
+
+1. **Mock External Dependencies**: Always mock external services (APIs, databases, Pulsar)
+2. **Test Business Logic**: Focus on testing your code, not external infrastructure
+3. **Use Dependency Injection**: Make services testable by injecting dependencies
+4. **Async Testing**: Use proper async test patterns for async services
+5. **Comprehensive Coverage**: Test success paths, error paths, and edge cases
+
+### Test Structure Example
+
+```python
+class TestServiceName(IsolatedAsyncioTestCase):
+    """Test service functionality"""
+
+    def setUp(self):
+        """Set up test fixtures"""
+        self.config = {...}
+
+    @patch('external.dependency')
+    async def test_success_case(self, mock_dependency):
+        """Test successful operation"""
+        # Arrange
+        mock_dependency.return_value = expected_result
+        
+        # Act
+        result = await service.method()
+        
+        # Assert
+        assert result == expected_result
+        mock_dependency.assert_called_once()
+```
+
+### Fixture Usage
+
+Use fixtures from `conftest.py` to reduce code duplication:
+
+```python
+async def test_with_fixtures(self, mock_vertexai_model, sample_text_completion_request):
+    """Test using shared fixtures"""
+    # Fixtures are automatically injected
+    result = await processor.process(sample_text_completion_request)
+    assert result.text == "Test response"
+```
+
+## Debugging Tests
+
+### Running Tests with Debug Information
+
+```bash
+# Run with debug output
+pytest -v -s tests/unit/test_text_completion/test_vertexai_processor.py
+
+# Run with pdb on failures
+pytest --pdb tests/unit/test_text_completion/test_vertexai_processor.py
+
+# Run with detailed tracebacks
+pytest --tb=long tests/unit/test_text_completion/test_vertexai_processor.py
+```
+
+### Common Issues and Solutions
+
+#### 1. Import Errors
+
+**Symptom**: `ModuleNotFoundError: No module named 'trustgraph'` or similar import errors
+
+**Solution**:
+```bash
+# First, check what's working
+./check_imports.py
+
+# Install the required packages
+./install_packages.sh
+
+# Verify installation worked
+./check_imports.py
+
+# If still having issues, check Python path
+echo $PYTHONPATH
+export PYTHONPATH=/home/mark/work/trustgraph.ai/trustgraph:$PYTHONPATH
+
+# Try running tests from project root
+cd /home/mark/work/trustgraph.ai/trustgraph
+pytest tests/unit/test_text_completion/test_vertexai_processor.py -v
+```
+
+**Common causes**:
+- TrustGraph packages not installed (`pip install -e .` in each package directory)
+- Wrong working directory (should be in project root)
+- Python path not set correctly
+- Missing dependencies (install with `pip install -r tests/requirements.txt`)
+
+#### 2. TaskGroup/Infrastructure Errors
+
+**Symptom**: `RuntimeError: Essential taskgroup missing` or similar infrastructure errors
+
+**Solution**:
+```bash
+# Try the simple tests first - they don't require full TrustGraph infrastructure
+./run_simple_tests.sh
+
+# Or run specific simple test files
+pytest tests/unit/test_text_completion/test_vertexai_simple.py -v
+pytest tests/unit/test_text_completion/test_vertexai_core.py -v
+```
+
+**Why this happens**:
+- The full TrustGraph processors require async task groups and Pulsar infrastructure
+- The simple tests focus on testing the core logic without infrastructure dependencies
+- Use simple tests to verify the VertexAI logic works correctly
+
+#### 3. Async Test Issues
+```python
+# Use IsolatedAsyncioTestCase for async tests
+class TestAsyncService(IsolatedAsyncioTestCase):
+    async def test_async_method(self):
+        result = await service.async_method()
+        assert result is not None
+```
+
+#### 3. Mock Issues
+```python
+# Use proper async mocks for async methods
+mock_client = AsyncMock()
+mock_client.async_method.return_value = expected_result
+
+# Use MagicMock for sync methods
+mock_client = MagicMock()
+mock_client.sync_method.return_value = expected_result
+```
+
+## Continuous Integration
+
+### Running Tests in CI
+
+```bash
+# Install dependencies
+pip install -r tests/requirements.txt
+
+# Run tests with coverage
+pytest --cov=trustgraph --cov-report=xml --cov-fail-under=80
+
+# Run tests in parallel (if using pytest-xdist)
+pytest -n auto
+```
+
+### Test Reports
+
+The test suite generates several types of reports:
+
+1. **Coverage Reports**: HTML and XML coverage reports
+2. **Test Results**: JUnit XML format for CI integration
+3. **Performance Reports**: For performance and load tests
+
+```bash
+# Generate all reports
+pytest --cov=trustgraph --cov-report=html --cov-report=xml --junitxml=test-results.xml
+```
+
+## Adding New Tests
+
+### 1. Create Test File
+
+```python
+# tests/unit/test_new_service/test_new_processor.py
+import pytest
+from unittest.mock import AsyncMock, MagicMock, patch
+from unittest import IsolatedAsyncioTestCase
+
+from trustgraph.new_service.processor import Processor
+
+class TestNewProcessor(IsolatedAsyncioTestCase):
+    """Test new processor functionality"""
+    
+    def setUp(self):
+        self.config = {...}
+    
+    @patch('trustgraph.new_service.processor.external_dependency')
+    async def test_processor_method(self, mock_dependency):
+        """Test processor method"""
+        # Arrange
+        mock_dependency.return_value = expected_result
+        processor = Processor(**self.config)
+        
+        # Act
+        result = await processor.method()
+        
+        # Assert
+        assert result == expected_result
+```
+
+### 2. Create Fixtures
+
+```python
+# tests/unit/test_new_service/conftest.py
+import pytest
+from unittest.mock import MagicMock
+
+@pytest.fixture
+def mock_new_service_client():
+    """Mock client for new service"""
+    return MagicMock()
+
+@pytest.fixture
+def sample_request():
+    """Sample request object"""
+    return RequestObject(id="test", data="test data")
+```
+
+### 3. Update pytest.ini
+
+```ini
+markers =
+    new_service: marks tests as new service specific tests
+```
+
+## Performance Testing
+
+### Load Testing
+
+```bash
+# Run performance tests
+pytest -m performance tests/performance/
+
+# Run with custom parameters
+pytest -m performance --count=100 --concurrent=10
+```
+
+### Memory Testing
+
+```bash
+# Run with memory profiling
+pytest --profile tests/unit/test_text_completion/test_vertexai_processor.py
+```
+
+## Best Practices
+
+### 1. Test Naming
+- Use descriptive test names that explain what is being tested
+- Follow the pattern: `test_<method>_<scenario>_<expected_result>`
+
+### 2. Test Organization
+- Group related tests in classes
+- Use meaningful class names that describe the component being tested
+- Keep tests focused on a single aspect of functionality
+
+### 3. Mock Strategy
+- Mock external dependencies, not internal business logic
+- Use the most specific mock type (AsyncMock for async, MagicMock for sync)
+- Verify mock calls to ensure proper interaction
+
+### 4. Assertions
+- Use specific assertions that clearly indicate what went wrong
+- Test both positive and negative cases
+- Include edge cases and boundary conditions
+
+### 5. Test Data
+- Use fixtures for reusable test data
+- Keep test data simple and focused
+- Avoid hardcoded values when possible
+
+## Troubleshooting
+
+### Common Test Failures
+
+1. **Import Errors**: Check PYTHONPATH and module structure
+2. **Async Issues**: Ensure proper async/await usage and AsyncMock
+3. **Mock Failures**: Verify mock setup and expected call patterns
+4. **Coverage Issues**: Check for untested code paths
+
+### Getting Help
+
+- Check the [TEST_STRATEGY.md](TEST_STRATEGY.md) for testing patterns
+- Review [TEST_CASES.md](TEST_CASES.md) for comprehensive test scenarios
+- Examine existing tests for examples and patterns
+- Use pytest's built-in help: `pytest --help`
+
+## Future Enhancements
+
+### Planned Test Additions
+
+1. **Integration Tests**: End-to-end flow testing
+2. **Performance Tests**: Load and stress testing
+3. **Security Tests**: Input validation and authentication
+4. **Contract Tests**: API contract verification
+
+### Test Infrastructure Improvements
+
+1. **Parallel Test Execution**: Using pytest-xdist
+2. **Test Data Management**: Better fixture organization
+3. **Reporting**: Enhanced test reporting and metrics
+4. **CI Integration**: Automated test execution and reporting
+
+---
+
+This testing guide provides comprehensive instructions for running and maintaining the TrustGraph test suite. Follow the patterns and guidelines to ensure consistent, reliable, and maintainable tests across all services.

TEST_CASES.md

@@ -0,0 +1,992 @@
+# Test Cases for TrustGraph Microservices
+
+This document provides comprehensive test cases for all TrustGraph microservices, organized by service category and following the testing strategy outlined in TEST_STRATEGY.md.
+
+## Table of Contents
+
+1. [Text Completion Services](#text-completion-services)
+2. [Embeddings Services](#embeddings-services)
+3. [Storage Services](#storage-services)
+4. [Query Services](#query-services)
+5. [Flow Processing](#flow-processing)
+6. [Configuration Management](#configuration-management)
+7. [Data Extraction Services](#data-extraction-services)
+8. [Retrieval Services](#retrieval-services)
+9. [Integration Test Cases](#integration-test-cases)
+10. [Error Handling Test Cases](#error-handling-test-cases)
+
+---
+
+## Text Completion Services
+
+### OpenAI Text Completion (`trustgraph.model.text_completion.openai`)
+
+#### Unit Tests
+- **test_openai_processor_initialization**
+  - Test processor initialization with valid API key
+  - Test processor initialization with invalid API key
+  - Test processor initialization with default parameters
+  - Test processor initialization with custom parameters (temperature, max_tokens)
+
+- **test_openai_message_processing**
+  - Test successful text completion with simple prompt
+  - Test text completion with complex multi-turn conversation
+  - Test text completion with system message
+  - Test text completion with custom temperature settings
+  - Test text completion with max_tokens limit
+  - Test text completion with streaming enabled/disabled
+
+- **test_openai_error_handling**
+  - Test rate limit error handling and retry logic
+  - Test API key authentication error
+  - Test network timeout error handling
+  - Test malformed response handling
+  - Test token limit exceeded error
+  - Test model not found error
+
+- **test_openai_metrics_collection**
+  - Test token usage metrics collection
+  - Test request duration metrics
+  - Test error rate metrics
+  - Test cost calculation metrics
+
+### Claude Text Completion (`trustgraph.model.text_completion.claude`)
+
+#### Unit Tests
+- **test_claude_processor_initialization**
+  - Test processor initialization with valid API key
+  - Test processor initialization with different model versions
+  - Test processor initialization with custom parameters
+
+- **test_claude_message_processing**
+  - Test successful text completion with simple prompt
+  - Test text completion with long context
+  - Test text completion with structured output
+  - Test text completion with function calling
+
+- **test_claude_error_handling**
+  - Test rate limit error handling
+  - Test content filtering error handling
+  - Test API quota exceeded error
+  - Test invalid model parameter error
+
+### Ollama Text Completion (`trustgraph.model.text_completion.ollama`)
+
+#### Unit Tests
+- **test_ollama_processor_initialization**
+  - Test processor initialization with local Ollama instance
+  - Test processor initialization with remote Ollama instance
+  - Test processor initialization with custom model
+
+- **test_ollama_message_processing**
+  - Test successful text completion with local model
+  - Test text completion with model loading
+  - Test text completion with custom generation parameters
+  - Test text completion with context window management
+
+- **test_ollama_error_handling**
+  - Test connection refused error handling
+  - Test model not available error
+  - Test out of memory error handling
+  - Test invalid model parameter error
+
+### Azure OpenAI Text Completion (`trustgraph.model.text_completion.azure`)
+
+#### Unit Tests
+- **test_azure_processor_initialization**
+  - Test processor initialization with Azure credentials
+  - Test processor initialization with deployment name
+  - Test processor initialization with API version
+
+- **test_azure_message_processing**
+  - Test successful text completion with Azure endpoint
+  - Test text completion with content filtering
+  - Test text completion with regional deployment
+
+- **test_azure_error_handling**
+  - Test Azure authentication error handling
+  - Test deployment not found error
+  - Test content filtering rejection error
+  - Test quota exceeded error
+
+### Google Vertex AI Text Completion (`trustgraph.model.text_completion.vertexai`)
+
+#### Unit Tests
+- **test_vertexai_processor_initialization**
+  - Test processor initialization with GCP credentials
+  - Test processor initialization with project ID and location
+  - Test processor initialization with model selection (gemini-pro, gemini-ultra)
+  - Test processor initialization with custom generation config
+
+- **test_vertexai_message_processing**
+  - Test successful text completion with Gemini models
+  - Test text completion with system instructions
+  - Test text completion with safety settings
+  - Test text completion with function calling
+  - Test text completion with multi-turn conversation
+  - Test text completion with streaming responses
+
+- **test_vertexai_safety_filtering**
+  - Test safety filter configuration
+  - Test blocked content handling
+  - Test safety threshold adjustments
+  - Test safety filter bypass scenarios
+
+- **test_vertexai_error_handling**
+  - Test authentication error handling (service account, ADC)
+  - Test quota exceeded error handling
+  - Test model not found error handling
+  - Test region availability error handling
+  - Test safety filter rejection error handling
+  - Test token limit exceeded error handling
+
+- **test_vertexai_metrics_collection**
+  - Test token usage metrics collection
+  - Test request duration metrics
+  - Test safety filter metrics
+  - Test cost calculation metrics per model type
+
+---
+
+## Embeddings Services
+
+### Document Embeddings (`trustgraph.embeddings.document_embeddings`)
+
+#### Unit Tests
+- **test_document_embeddings_initialization**
+  - Test embeddings processor initialization with default model
+  - Test embeddings processor initialization with custom model
+  - Test embeddings processor initialization with batch size configuration
+
+- **test_document_embeddings_processing**
+  - Test single document embedding generation
+  - Test batch document embedding generation
+  - Test empty document handling
+  - Test very long document handling
+  - Test document with special characters
+  - Test document with multiple languages
+
+- **test_document_embeddings_vector_operations**
+  - Test vector dimension consistency
+  - Test vector normalization
+  - Test similarity calculation
+  - Test vector serialization/deserialization
+
+### Graph Embeddings (`trustgraph.embeddings.graph_embeddings`)
+
+#### Unit Tests
+- **test_graph_embeddings_initialization**
+  - Test graph embeddings processor initialization
+  - Test initialization with custom embedding dimensions
+  - Test initialization with different aggregation methods
+
+- **test_graph_embeddings_processing**
+  - Test entity embedding generation
+  - Test relationship embedding generation
+  - Test subgraph embedding generation
+  - Test dynamic graph embedding updates
+
+- **test_graph_embeddings_aggregation**
+  - Test mean aggregation of entity embeddings
+  - Test weighted aggregation of relationship embeddings
+  - Test hierarchical embedding aggregation
+
+### Ollama Embeddings (`trustgraph.embeddings.ollama`)
+
+#### Unit Tests
+- **test_ollama_embeddings_initialization**
+  - Test Ollama embeddings processor initialization
+  - Test initialization with custom embedding model
+  - Test initialization with connection parameters
+
+- **test_ollama_embeddings_processing**
+  - Test successful embedding generation
+  - Test batch embedding processing
+  - Test embedding caching
+  - Test embedding model switching
+
+- **test_ollama_embeddings_error_handling**
+  - Test connection error handling
+  - Test model loading error handling
+  - Test out of memory error handling
+
+---
+
+## Storage Services
+
+### Document Embeddings Storage
+
+#### Qdrant Storage (`trustgraph.storage.doc_embeddings.qdrant`)
+
+##### Unit Tests
+- **test_qdrant_storage_initialization**
+  - Test Qdrant client initialization with local instance
+  - Test Qdrant client initialization with remote instance
+  - Test Qdrant client initialization with authentication
+  - Test collection creation and configuration
+
+- **test_qdrant_storage_operations**
+  - Test single vector insertion
+  - Test batch vector insertion
+  - Test vector update operations
+  - Test vector deletion operations
+  - Test vector search operations
+  - Test filtered search operations
+
+- **test_qdrant_storage_error_handling**
+  - Test connection error handling
+  - Test collection not found error
+  - Test vector dimension mismatch error
+  - Test storage quota exceeded error
+
+#### Milvus Storage (`trustgraph.storage.doc_embeddings.milvus`)
+
+##### Unit Tests
+- **test_milvus_storage_initialization**
+  - Test Milvus client initialization
+  - Test collection schema creation
+  - Test index creation and configuration
+
+- **test_milvus_storage_operations**
+  - Test entity insertion with metadata
+  - Test bulk insertion operations
+  - Test vector search with filters
+  - Test hybrid search operations
+
+- **test_milvus_storage_error_handling**
+  - Test connection timeout error
+  - Test collection creation error
+  - Test index building error
+  - Test search timeout error
+
+### Graph Embeddings Storage
+
+#### Qdrant Storage (`trustgraph.storage.graph_embeddings.qdrant`)
+
+##### Unit Tests
+- **test_qdrant_graph_storage_initialization**
+  - Test Qdrant client initialization for graph embeddings
+  - Test collection creation with graph-specific schema
+  - Test index configuration for entity and relationship embeddings
+
+- **test_qdrant_graph_storage_operations**
+  - Test entity embedding insertion with metadata
+  - Test relationship embedding insertion
+  - Test subgraph embedding storage
+  - Test batch insertion of graph embeddings
+  - Test embedding updates and versioning
+
+- **test_qdrant_graph_storage_queries**
+  - Test entity similarity search
+  - Test relationship similarity search
+  - Test subgraph similarity search
+  - Test filtered search by graph properties
+  - Test multi-vector search operations
+
+- **test_qdrant_graph_storage_error_handling**
+  - Test connection error handling
+  - Test collection not found error
+  - Test vector dimension mismatch for graph embeddings
+  - Test storage quota exceeded error
+
+#### Milvus Storage (`trustgraph.storage.graph_embeddings.milvus`)
+
+##### Unit Tests
+- **test_milvus_graph_storage_initialization**
+  - Test Milvus client initialization for graph embeddings
+  - Test collection schema creation for graph data
+  - Test index creation for entity and relationship vectors
+
+- **test_milvus_graph_storage_operations**
+  - Test entity embedding insertion with graph metadata
+  - Test relationship embedding insertion
+  - Test graph structure preservation
+  - Test bulk graph embedding operations
+
+- **test_milvus_graph_storage_error_handling**
+  - Test connection timeout error
+  - Test graph schema validation error
+  - Test index building error for graph embeddings
+  - Test search timeout error
+
+### Graph Storage
+
+#### Cassandra Storage (`trustgraph.storage.triples.cassandra`)
+
+##### Unit Tests
+- **test_cassandra_storage_initialization**
+  - Test Cassandra client initialization
+  - Test keyspace creation and configuration
+  - Test table schema creation
+
+- **test_cassandra_storage_operations**
+  - Test triple insertion (subject, predicate, object)
+  - Test batch triple insertion
+  - Test triple querying by subject
+  - Test triple querying by predicate
+  - Test triple deletion operations
+
+- **test_cassandra_storage_consistency**
+  - Test consistency level configuration
+  - Test replication factor handling
+  - Test partition key distribution
+
+#### Neo4j Storage (`trustgraph.storage.triples.neo4j`)
+
+##### Unit Tests
+- **test_neo4j_storage_initialization**
+  - Test Neo4j driver initialization
+  - Test database connection with authentication
+  - Test constraint and index creation
+
+- **test_neo4j_storage_operations**
+  - Test node creation and properties
+  - Test relationship creation
+  - Test graph traversal operations
+  - Test transaction management
+
+- **test_neo4j_storage_error_handling**
+  - Test connection pool exhaustion
+  - Test transaction rollback scenarios
+  - Test constraint violation handling
+
+---
+
+## Query Services
+
+### Document Embeddings Query
+
+#### Qdrant Query (`trustgraph.query.doc_embeddings.qdrant`)
+
+##### Unit Tests
+- **test_qdrant_query_initialization**
+  - Test query service initialization with collection
+  - Test query service initialization with custom parameters
+
+- **test_qdrant_query_operations**
+  - Test similarity search with single vector
+  - Test similarity search with multiple vectors
+  - Test filtered similarity search
+  - Test ranked result retrieval
+  - Test pagination support
+
+- **test_qdrant_query_performance**
+  - Test query timeout handling
+  - Test large result set handling
+  - Test concurrent query handling
+
+#### Milvus Query (`trustgraph.query.doc_embeddings.milvus`)
+
+##### Unit Tests
+- **test_milvus_query_initialization**
+  - Test query service initialization
+  - Test index selection for queries
+
+- **test_milvus_query_operations**
+  - Test vector similarity search
+  - Test hybrid search with scalar filters
+  - Test range search operations
+  - Test top-k result retrieval
+
+### Graph Embeddings Query
+
+#### Qdrant Query (`trustgraph.query.graph_embeddings.qdrant`)
+
+##### Unit Tests
+- **test_qdrant_graph_query_initialization**
+  - Test graph query service initialization with collection
+  - Test graph query service initialization with custom parameters
+  - Test entity and relationship collection configuration
+
+- **test_qdrant_graph_query_operations**
+  - Test entity similarity search with single vector
+  - Test relationship similarity search
+  - Test subgraph pattern matching
+  - Test multi-hop graph traversal queries
+  - Test filtered graph similarity search
+  - Test ranked graph result retrieval
+  - Test graph query pagination
+
+- **test_qdrant_graph_query_optimization**
+  - Test graph query performance optimization
+  - Test graph query result caching
+  - Test concurrent graph query handling
+  - Test graph query timeout handling
+
+- **test_qdrant_graph_query_error_handling**
+  - Test graph collection not found error
+  - Test graph query timeout error
+  - Test invalid graph query parameter error
+  - Test graph result limit exceeded error
+
+#### Milvus Query (`trustgraph.query.graph_embeddings.milvus`)
+
+##### Unit Tests
+- **test_milvus_graph_query_initialization**
+  - Test graph query service initialization
+  - Test graph index selection for queries
+  - Test graph collection configuration
+
+- **test_milvus_graph_query_operations**
+  - Test entity vector similarity search
+  - Test relationship vector similarity search
+  - Test graph hybrid search with scalar filters
+  - Test graph range search operations
+  - Test top-k graph result retrieval
+  - Test graph query result aggregation
+
+- **test_milvus_graph_query_performance**
+  - Test graph query performance with large datasets
+  - Test graph query optimization strategies
+  - Test graph query result caching
+
+- **test_milvus_graph_query_error_handling**
+  - Test graph connection timeout error
+  - Test graph collection not found error
+  - Test graph query syntax error
+  - Test graph search timeout error
+
+### Graph Query
+
+#### Cassandra Query (`trustgraph.query.triples.cassandra`)
+
+##### Unit Tests
+- **test_cassandra_query_initialization**
+  - Test query service initialization
+  - Test prepared statement creation
+
+- **test_cassandra_query_operations**
+  - Test subject-based triple retrieval
+  - Test predicate-based triple retrieval
+  - Test object-based triple retrieval
+  - Test pattern-based triple matching
+  - Test subgraph extraction
+
+- **test_cassandra_query_optimization**
+  - Test query result caching
+  - Test pagination for large result sets
+  - Test query performance with indexes
+
+#### Neo4j Query (`trustgraph.query.triples.neo4j`)
+
+##### Unit Tests
+- **test_neo4j_query_initialization**
+  - Test query service initialization
+  - Test Cypher query preparation
+
+- **test_neo4j_query_operations**
+  - Test node retrieval by properties
+  - Test relationship traversal queries
+  - Test shortest path queries
+  - Test subgraph pattern matching
+  - Test graph analytics queries
+
+---
+
+## Flow Processing
+
+### Base Flow Processor (`trustgraph.processing`)
+
+#### Unit Tests
+- **test_flow_processor_initialization**
+  - Test processor initialization with specifications
+  - Test consumer specification registration
+  - Test producer specification registration
+  - Test request-response specification registration
+
+- **test_flow_processor_message_handling**
+  - Test message consumption from Pulsar
+  - Test message processing pipeline
+  - Test message production to Pulsar
+  - Test message acknowledgment handling
+
+- **test_flow_processor_error_handling**
+  - Test message processing error handling
+  - Test dead letter queue handling
+  - Test retry mechanism
+  - Test circuit breaker pattern
+
+- **test_flow_processor_metrics**
+  - Test processing time metrics
+  - Test message throughput metrics
+  - Test error rate metrics
+  - Test queue depth metrics
+
+### Async Processor Base
+
+#### Unit Tests
+- **test_async_processor_initialization**
+  - Test async processor initialization
+  - Test concurrency configuration
+  - Test resource management
+
+- **test_async_processor_concurrency**
+  - Test concurrent message processing
+  - Test backpressure handling
+  - Test resource pool management
+  - Test graceful shutdown
+
+---
+
+## Configuration Management
+
+### Configuration Service
+
+#### Unit Tests
+- **test_configuration_service_initialization**
+  - Test configuration service startup
+  - Test Cassandra backend initialization
+  - Test configuration schema creation
+
+- **test_configuration_service_operations**
+  - Test configuration retrieval by service
+  - Test configuration update operations
+  - Test configuration validation
+  - Test configuration versioning
+
+- **test_configuration_service_caching**
+  - Test configuration caching mechanism
+  - Test cache invalidation
+  - Test cache consistency
+
+- **test_configuration_service_error_handling**
+  - Test configuration not found error
+  - Test configuration validation error
+  - Test backend connection error
+
+### Flow Configuration
+
+#### Unit Tests
+- **test_flow_configuration_parsing**
+  - Test flow definition parsing from JSON
+  - Test flow validation rules
+  - Test flow dependency resolution
+
+- **test_flow_configuration_deployment**
+  - Test flow deployment to services
+  - Test flow lifecycle management
+  - Test flow rollback operations
+
+---
+
+## Data Extraction Services
+
+### Knowledge Graph Extraction
+
+#### Topic Extraction (`trustgraph.extract.kg.topics`)
+
+##### Unit Tests
+- **test_topic_extraction_initialization**
+  - Test topic extractor initialization
+  - Test LLM client configuration
+  - Test extraction prompt configuration
+
+- **test_topic_extraction_processing**
+  - Test topic extraction from text
+  - Test topic deduplication
+  - Test topic relevance scoring
+  - Test topic hierarchy extraction
+
+- **test_topic_extraction_error_handling**
+  - Test malformed text handling
+  - Test empty text handling
+  - Test extraction timeout handling
+
+#### Relationship Extraction (`trustgraph.extract.kg.relationships`)
+
+##### Unit Tests
+- **test_relationship_extraction_initialization**
+  - Test relationship extractor initialization
+  - Test relationship type configuration
+
+- **test_relationship_extraction_processing**
+  - Test relationship extraction from text
+  - Test relationship validation
+  - Test relationship confidence scoring
+  - Test relationship normalization
+
+#### Definition Extraction (`trustgraph.extract.kg.definitions`)
+
+##### Unit Tests
+- **test_definition_extraction_initialization**
+  - Test definition extractor initialization
+  - Test definition pattern configuration
+
+- **test_definition_extraction_processing**
+  - Test definition extraction from text
+  - Test definition quality assessment
+  - Test definition standardization
+
+### Object Extraction
+
+#### Row Extraction (`trustgraph.extract.object.row`)
+
+##### Unit Tests
+- **test_row_extraction_initialization**
+  - Test row extractor initialization
+  - Test schema configuration
+
+- **test_row_extraction_processing**
+  - Test structured data extraction
+  - Test row validation
+  - Test row normalization
+
+---
+
+## Retrieval Services
+
+### GraphRAG Retrieval (`trustgraph.retrieval.graph_rag`)
+
+#### Unit Tests
+- **test_graph_rag_initialization**
+  - Test GraphRAG retrieval initialization
+  - Test graph and vector store configuration
+  - Test retrieval parameters configuration
+
+- **test_graph_rag_processing**
+  - Test query processing and understanding
+  - Test vector similarity search
+  - Test graph traversal for context
+  - Test context ranking and selection
+  - Test response generation
+
+- **test_graph_rag_optimization**
+  - Test query optimization
+  - Test context size management
+  - Test retrieval caching
+  - Test performance monitoring
+
+### Document RAG Retrieval (`trustgraph.retrieval.document_rag`)
+
+#### Unit Tests
+- **test_document_rag_initialization**
+  - Test Document RAG retrieval initialization
+  - Test document store configuration
+
+- **test_document_rag_processing**
+  - Test document similarity search
+  - Test document chunk retrieval
+  - Test document ranking
+  - Test context assembly
+
+---
+
+## Integration Test Cases
+
+### End-to-End Flow Tests
+
+#### Document Processing Flow
+- **test_document_ingestion_flow**
+  - Test PDF document ingestion
+  - Test text document ingestion
+  - Test document chunking
+  - Test embedding generation
+  - Test storage operations
+
+- **test_knowledge_graph_construction_flow**
+  - Test entity extraction
+  - Test relationship extraction
+  - Test graph construction
+  - Test graph storage
+
+#### Query Processing Flow
+- **test_graphrag_query_flow**
+  - Test query input processing
+  - Test vector similarity search
+  - Test graph traversal
+  - Test context assembly
+  - Test response generation
+
+- **test_agent_flow**
+  - Test agent query processing
+  - Test ReAct reasoning cycle
+  - Test tool usage
+  - Test response formatting
+
+### Service Integration Tests
+
+#### Storage Integration
+- **test_vector_storage_integration**
+  - Test Qdrant integration with embeddings
+  - Test Milvus integration with embeddings
+  - Test storage consistency across services
+
+- **test_graph_storage_integration**
+  - Test Cassandra integration with triples
+  - Test Neo4j integration with graphs
+  - Test cross-storage consistency
+
+#### Model Integration
+- **test_llm_integration**
+  - Test OpenAI integration
+  - Test Claude integration
+  - Test Ollama integration
+  - Test model switching
+
+---
+
+## Error Handling Test Cases
+
+### Network Error Handling
+- **test_connection_timeout_handling**
+  - Test database connection timeouts
+  - Test API connection timeouts
+  - Test Pulsar connection timeouts
+
+- **test_network_interruption_handling**
+  - Test network disconnection scenarios
+  - Test network reconnection scenarios
+  - Test partial network failures
+
+### Resource Error Handling
+- **test_memory_exhaustion_handling**
+  - Test out of memory scenarios
+  - Test memory leak detection
+  - Test memory cleanup
+
+- **test_disk_space_handling**
+  - Test disk full scenarios
+  - Test storage cleanup
+  - Test storage monitoring
+
+### Service Error Handling
+- **test_service_unavailable_handling**
+  - Test external service unavailability
+  - Test service degradation
+  - Test service recovery
+
+- **test_data_corruption_handling**
+  - Test corrupted message handling
+  - Test invalid data detection
+  - Test data recovery procedures
+
+### Rate Limiting Error Handling
+- **test_api_rate_limit_handling**
+  - Test OpenAI rate limit scenarios
+  - Test Claude rate limit scenarios
+  - Test backoff strategies
+
+- **test_resource_quota_handling**
+  - Test storage quota exceeded
+  - Test compute quota exceeded
+  - Test API quota exceeded
+
+---
+
+## Performance Test Cases
+
+### Load Testing
+- **test_concurrent_processing**
+  - Test concurrent message processing
+  - Test concurrent database operations
+  - Test concurrent API calls
+
+- **test_throughput_limits**
+  - Test message processing throughput
+  - Test storage operation throughput
+  - Test query processing throughput
+
+### Stress Testing
+- **test_high_volume_processing**
+  - Test processing large document sets
+  - Test handling large knowledge graphs
+  - Test processing high query volumes
+
+- **test_resource_exhaustion**
+  - Test behavior under memory pressure
+  - Test behavior under CPU pressure
+  - Test behavior under network pressure
+
+### Scalability Testing
+- **test_horizontal_scaling**
+  - Test service scaling behavior
+  - Test load distribution
+  - Test scaling bottlenecks
+
+- **test_vertical_scaling**
+  - Test resource utilization scaling
+  - Test performance scaling
+  - Test cost scaling
+
+---
+
+## Security Test Cases
+
+### Authentication and Authorization
+- **test_api_key_validation**
+  - Test valid API key scenarios
+  - Test invalid API key scenarios
+  - Test expired API key scenarios
+
+- **test_service_authentication**
+  - Test service-to-service authentication
+  - Test authentication token validation
+  - Test authentication failure handling
+
+### Data Protection
+- **test_data_encryption**
+  - Test data encryption at rest
+  - Test data encryption in transit
+  - Test encryption key management
+
+- **test_data_sanitization**
+  - Test input data sanitization
+  - Test output data sanitization
+  - Test sensitive data masking
+
+### Input Validation
+- **test_input_validation**
+  - Test malformed input handling
+  - Test injection attack prevention
+  - Test input size limits
+
+- **test_output_validation**
+  - Test output format validation
+  - Test output content validation
+  - Test output size limits
+
+---
+
+## Monitoring and Observability Test Cases
+
+### Metrics Collection
+- **test_prometheus_metrics**
+  - Test metrics collection and export
+  - Test custom metrics registration
+  - Test metrics aggregation
+
+- **test_performance_metrics**
+  - Test latency metrics collection
+  - Test throughput metrics collection
+  - Test error rate metrics collection
+
+### Logging
+- **test_structured_logging**
+  - Test log format consistency
+  - Test log level configuration
+  - Test log aggregation
+
+- **test_error_logging**
+  - Test error log capture
+  - Test error log correlation
+  - Test error log analysis
+
+### Tracing
+- **test_distributed_tracing**
+  - Test trace propagation
+  - Test trace correlation
+  - Test trace analysis
+
+- **test_request_tracing**
+  - Test request lifecycle tracing
+  - Test cross-service tracing
+  - Test trace performance impact
+
+---
+
+## Configuration Test Cases
+
+### Environment Configuration
+- **test_environment_variables**
+  - Test environment variable loading
+  - Test environment variable validation
+  - Test environment variable defaults
+
+- **test_configuration_files**
+  - Test configuration file loading
+  - Test configuration file validation
+  - Test configuration file precedence
+
+### Dynamic Configuration
+- **test_configuration_updates**
+  - Test runtime configuration updates
+  - Test configuration change propagation
+  - Test configuration rollback
+
+- **test_configuration_validation**
+  - Test configuration schema validation
+  - Test configuration dependency validation
+  - Test configuration constraint validation
+
+---
+
+## Test Data and Fixtures
+
+### Test Data Generation
+- **test_synthetic_data_generation**
+  - Test synthetic document generation
+  - Test synthetic graph data generation
+  - Test synthetic query generation
+
+- **test_data_anonymization**
+  - Test personal data anonymization
+  - Test sensitive data masking
+  - Test data privacy compliance
+
+### Test Fixtures
+- **test_fixture_management**
+  - Test fixture setup and teardown
+  - Test fixture data consistency
+  - Test fixture isolation
+
+- **test_mock_data_quality**
+  - Test mock data realism
+  - Test mock data coverage
+  - Test mock data maintenance
+
+---
+
+## Test Execution and Reporting
+
+### Test Execution
+- **test_parallel_execution**
+  - Test parallel test execution
+  - Test test isolation
+  - Test resource contention
+
+- **test_test_selection**
+  - Test tag-based test selection
+  - Test conditional test execution
+  - Test test prioritization
+
+### Test Reporting
+- **test_coverage_reporting**
+  - Test code coverage measurement
+  - Test branch coverage analysis
+  - Test coverage trend analysis
+
+- **test_performance_reporting**
+  - Test performance regression detection
+  - Test performance trend analysis
+  - Test performance benchmarking
+
+---
+
+## Maintenance and Continuous Integration
+
+### Test Maintenance
+- **test_test_reliability**
+  - Test flaky test detection
+  - Test test stability analysis
+  - Test test maintainability
+
+- **test_test_documentation**
+  - Test test documentation quality
+  - Test test case traceability
+  - Test test requirement coverage
+
+### Continuous Integration
+- **test_ci_pipeline_integration**
+  - Test CI pipeline configuration
+  - Test test execution in CI
+  - Test test result reporting
+
+- **test_automated_testing**
+  - Test automated test execution
+  - Test automated test reporting
+  - Test automated test maintenance
+
+---
+
+This comprehensive test case document provides detailed testing scenarios for all TrustGraph microservices, ensuring thorough coverage of functionality, error handling, performance, security, and operational aspects. Each test case should be implemented following the patterns and best practices outlined in the TEST_STRATEGY.md document.
+

TEST_SETUP.md

@@ -0,0 +1,96 @@
+# Quick Test Setup Guide
+
+## TL;DR - Just Run This
+
+```bash
+# From the trustgraph project root directory
+./run_tests.sh
+```
+
+This script will:
+1. Check current imports
+2. Install all required TrustGraph packages
+3. Install test dependencies
+4. Run the VertexAI tests
+
+## If You Get Import Errors
+
+The most common issue is that TrustGraph packages aren't installed. Here's how to fix it:
+
+### Step 1: Check What's Missing
+```bash
+./check_imports.py
+```
+
+### Step 2: Install TrustGraph Packages
+```bash
+./install_packages.sh
+```
+
+### Step 3: Verify Installation
+```bash
+./check_imports.py
+```
+
+### Step 4: Run Tests
+```bash
+pytest tests/unit/test_text_completion/test_vertexai_processor.py -v
+```
+
+## What the Scripts Do
+
+### `check_imports.py`
+- Tests all the imports needed for the tests
+- Shows exactly what's missing
+- Helps diagnose import issues
+
+### `install_packages.sh`
+- Installs trustgraph-base (required by others)
+- Installs trustgraph-cli
+- Installs trustgraph-vertexai
+- Installs trustgraph-flow
+- Uses `pip install -e .` for editable installs
+
+### `run_tests.sh`
+- Runs all the above steps in order
+- Installs test dependencies
+- Runs the VertexAI tests
+- Shows clear output at each step
+
+## Manual Installation (If Scripts Don't Work)
+
+```bash
+# Install packages in order (base first!)
+cd trustgraph-base && pip install -e . && cd ..
+cd trustgraph-cli && pip install -e . && cd ..
+cd trustgraph-vertexai && pip install -e . && cd ..
+cd trustgraph-flow && pip install -e . && cd ..
+
+# Install test dependencies
+cd tests && pip install -r requirements.txt && cd ..
+
+# Run tests
+pytest tests/unit/test_text_completion/test_vertexai_processor.py -v
+```
+
+## Common Issues
+
+1. **"No module named 'trustgraph'"** → Run `./install_packages.sh`
+2. **"No module named 'trustgraph.base'"** → Install trustgraph-base first
+3. **"No module named 'trustgraph.model.text_completion.vertexai'"** → Install trustgraph-vertexai
+4. **Scripts not executable** → Run `chmod +x *.sh`
+5. **Wrong directory** → Make sure you're in the project root (where README.md is)
+
+## Test Results
+
+When working correctly, you should see:
+- ✅ All imports successful
+- 139 test cases running
+- Tests passing (or failing for logical reasons, not import errors)
+
+## Getting Help
+
+If you're still having issues:
+1. Share the output of `./check_imports.py`
+2. Share the exact error message
+3. Confirm you're in the right directory: `/home/mark/work/trustgraph.ai/trustgraph`

TEST_STRATEGY.md

@@ -0,0 +1,243 @@
+# Unit Testing Strategy for TrustGraph Microservices
+
+## Overview
+
+This document outlines the unit testing strategy for the TrustGraph microservices architecture. The approach focuses on testing business logic while mocking external infrastructure to ensure fast, reliable, and maintainable tests.
+
+## 1. Test Framework: pytest + pytest-asyncio
+
+- **pytest**: Standard Python testing framework with excellent fixture support
+- **pytest-asyncio**: Essential for testing async processors
+- **pytest-mock**: Built-in mocking capabilities
+
+## 2. Core Testing Patterns
+
+### Service Layer Testing
+
+```python
+@pytest.mark.asyncio
+async def test_text_completion_service():
+    # Test the core business logic, not external APIs
+    processor = TextCompletionProcessor(model="test-model")
+    
+    # Mock external dependencies
+    with patch('processor.llm_client') as mock_client:
+        mock_client.generate.return_value = "test response"
+        
+        result = await processor.process_message(test_message)
+        assert result.content == "test response"
+```
+
+### Message Processing Testing
+
+```python
+@pytest.fixture
+def mock_pulsar_consumer():
+    return AsyncMock(spec=pulsar.Consumer)
+
+@pytest.fixture  
+def mock_pulsar_producer():
+    return AsyncMock(spec=pulsar.Producer)
+
+async def test_message_flow(mock_consumer, mock_producer):
+    # Test message handling without actual Pulsar
+    processor = FlowProcessor(consumer=mock_consumer, producer=mock_producer)
+    # Test message processing logic
+```
+
+## 3. Mock Strategy
+
+### Mock External Services (Not Infrastructure)
+
+- ✅ **Mock**: LLM APIs, Vector DBs, Graph DBs
+- ❌ **Don't Mock**: Core business logic, data transformations
+- ✅ **Mock**: Pulsar clients (infrastructure)
+- ❌ **Don't Mock**: Message validation, processing logic
+
+### Dependency Injection Pattern
+
+```python
+class TextCompletionProcessor:
+    def __init__(self, llm_client=None, **kwargs):
+        self.llm_client = llm_client or create_default_client()
+        
+# In tests
+processor = TextCompletionProcessor(llm_client=mock_client)
+```
+
+## 4. Test Categories
+
+### Unit Tests (70%)
+- Individual service business logic
+- Message processing functions
+- Data transformation logic
+- Configuration parsing
+- Error handling
+
+### Integration Tests (20%)
+- Service-to-service communication patterns
+- Database operations with test containers
+- End-to-end message flows
+
+### Contract Tests (10%)
+- Pulsar message schemas
+- API response formats
+- Service interface contracts
+
+## 5. Test Structure
+
+```
+tests/
+├── unit/
+│   ├── test_text_completion/
+│   ├── test_embeddings/
+│   ├── test_storage/
+│   └── test_utils/
+├── integration/
+│   ├── test_flows/
+│   └── test_databases/
+├── fixtures/
+│   ├── messages.py
+│   ├── configs.py
+│   └── mocks.py
+└── conftest.py
+```
+
+## 6. Key Testing Tools
+
+- **testcontainers**: For database integration tests
+- **responses**: Mock HTTP APIs
+- **freezegun**: Time-based testing
+- **factory-boy**: Test data generation
+
+## 7. Service-Specific Testing Approaches
+
+### Text Completion Services
+- Mock LLM provider APIs (OpenAI, Claude, Ollama)
+- Test prompt construction and response parsing
+- Verify rate limiting and error handling
+- Test token counting and metrics collection
+
+### Embeddings Services
+- Mock embedding providers (FastEmbed, Ollama)
+- Test vector dimension consistency
+- Verify batch processing logic
+- Test embedding storage operations
+
+### Storage Services
+- Use testcontainers for database integration tests
+- Mock database clients for unit tests
+- Test query construction and result parsing
+- Verify data persistence and retrieval logic
+
+### Query Services
+- Mock vector similarity search operations
+- Test graph traversal logic
+- Verify result ranking and filtering
+- Test query optimization
+
+## 8. Best Practices
+
+### Test Isolation
+- Each test should be independent
+- Use fixtures for common setup
+- Clean up resources after tests
+- Avoid test order dependencies
+
+### Async Testing
+- Use `@pytest.mark.asyncio` for async tests
+- Mock async dependencies properly
+- Test concurrent operations
+- Handle timeout scenarios
+
+### Error Handling
+- Test both success and failure scenarios
+- Verify proper exception handling
+- Test retry mechanisms
+- Validate error response formats
+
+### Configuration Testing
+- Test different configuration scenarios
+- Verify parameter validation
+- Test environment variable handling
+- Test configuration defaults
+
+## 9. Example Test Implementation
+
+```python
+# tests/unit/test_text_completion/test_openai_processor.py
+import pytest
+from unittest.mock import AsyncMock, patch
+from trustgraph.model.text_completion.openai import Processor
+
+@pytest.fixture
+def mock_openai_client():
+    return AsyncMock()
+
+@pytest.fixture
+def processor(mock_openai_client):
+    return Processor(client=mock_openai_client, model="gpt-4")
+
+@pytest.mark.asyncio
+async def test_process_message_success(processor, mock_openai_client):
+    # Arrange
+    mock_openai_client.chat.completions.create.return_value = AsyncMock(
+        choices=[AsyncMock(message=AsyncMock(content="Test response"))]
+    )
+    
+    message = {
+        "id": "test-id",
+        "prompt": "Test prompt",
+        "temperature": 0.7
+    }
+    
+    # Act
+    result = await processor.process_message(message)
+    
+    # Assert
+    assert result.content == "Test response"
+    mock_openai_client.chat.completions.create.assert_called_once()
+
+@pytest.mark.asyncio
+async def test_process_message_rate_limit(processor, mock_openai_client):
+    # Arrange
+    mock_openai_client.chat.completions.create.side_effect = RateLimitError("Rate limited")
+    
+    message = {"id": "test-id", "prompt": "Test prompt"}
+    
+    # Act & Assert
+    with pytest.raises(RateLimitError):
+        await processor.process_message(message)
+```
+
+## 10. Running Tests
+
+```bash
+# Run all tests
+pytest
+
+# Run unit tests only
+pytest tests/unit/
+
+# Run with coverage
+pytest --cov=trustgraph --cov-report=html
+
+# Run async tests
+pytest -v tests/unit/test_text_completion/
+
+# Run specific test file
+pytest tests/unit/test_text_completion/test_openai_processor.py
+```
+
+## 11. Continuous Integration
+
+- Run tests on every commit
+- Enforce minimum code coverage (80%+)
+- Run tests against multiple Python versions
+- Include integration tests in CI pipeline
+- Generate test reports and coverage metrics
+
+## Conclusion
+
+This testing strategy ensures that TrustGraph microservices are thoroughly tested without relying on external infrastructure. By focusing on business logic and mocking external dependencies, we achieve fast, reliable tests that provide confidence in code quality while maintaining development velocity.
+

check_imports.py

@@ -0,0 +1,74 @@
+#!/usr/bin/env python3
+"""
+Check if TrustGraph imports work correctly for testing
+"""
+
+import sys
+import traceback
+
+def check_import(module_name, description):
+    """Try to import a module and report the result"""
+    try:
+        __import__(module_name)
+        print(f"✅ {description}: {module_name}")
+        return True
+    except ImportError as e:
+        print(f"❌ {description}: {module_name}")
+        print(f"   Error: {e}")
+        return False
+    except Exception as e:
+        print(f"❌ {description}: {module_name}")
+        print(f"   Unexpected error: {e}")
+        return False
+
+def main():
+    print("Checking TrustGraph imports for testing...")
+    print("=" * 50)
+    
+    imports_to_check = [
+        ("trustgraph", "Base trustgraph package"),
+        ("trustgraph.base", "Base classes"),
+        ("trustgraph.base.llm_service", "LLM service base class"),
+        ("trustgraph.schema", "Schema definitions"),
+        ("trustgraph.exceptions", "Exception classes"),
+        ("trustgraph.model", "Model package"),
+        ("trustgraph.model.text_completion", "Text completion package"),
+        ("trustgraph.model.text_completion.vertexai", "VertexAI package"),
+    ]
+    
+    success_count = 0
+    total_count = len(imports_to_check)
+    
+    for module_name, description in imports_to_check:
+        if check_import(module_name, description):
+            success_count += 1
+        print()
+    
+    print("=" * 50)
+    print(f"Import Check Results: {success_count}/{total_count} successful")
+    
+    if success_count == total_count:
+        print("✅ All imports successful! Tests should work.")
+    else:
+        print("❌ Some imports failed. Please install missing packages.")
+        print("\nTo fix, run:")
+        print("  ./install_packages.sh")
+        print("or install packages manually:")
+        print("  cd trustgraph-base && pip install -e . && cd ..")
+        print("  cd trustgraph-vertexai && pip install -e . && cd ..")
+        print("  cd trustgraph-flow && pip install -e . && cd ..")
+    
+    # Test the specific import used in the test
+    print("\n" + "=" * 50)
+    print("Testing specific import from test file...")
+    try:
+        from trustgraph.model.text_completion.vertexai.llm import Processor
+        from trustgraph.schema import TextCompletionRequest, TextCompletionResponse, Error
+        from trustgraph.base import LlmResult
+        print("✅ Test imports successful!")
+    except Exception as e:
+        print(f"❌ Test imports failed: {e}")
+        traceback.print_exc()
+
+if __name__ == "__main__":
+    main()

containers/Containerfile.base

@@ -11,7 +11,7 @@ ENV PIP_BREAK_SYSTEM_PACKAGES=1
 RUN dnf install -y python3.12 && \
   alternatives --install /usr/bin/python python /usr/bin/python3.12 1 && \
   python -m ensurepip --upgrade && \
-  pip3 install --no-cache-dir wheel aiohttp && \
+  pip3 install --no-cache-dir build wheel aiohttp && \
   pip3 install --no-cache-dir pulsar-client==3.7.0 && \
   dnf clean all
 

containers/Containerfile.bedrock

@@ -11,7 +11,7 @@ ENV PIP_BREAK_SYSTEM_PACKAGES=1
 RUN dnf install -y python3.12 && \
   alternatives --install /usr/bin/python python /usr/bin/python3.12 1 && \
   python -m ensurepip --upgrade && \
-  pip3 install --no-cache-dir wheel aiohttp && \
+  pip3 install --no-cache-dir build wheel aiohttp && \
   pip3 install --no-cache-dir pulsar-client==3.7.0 && \
   dnf clean all
 

containers/Containerfile.flow

@@ -11,7 +11,7 @@ ENV PIP_BREAK_SYSTEM_PACKAGES=1
 RUN dnf install -y python3.12 && \
   alternatives --install /usr/bin/python python /usr/bin/python3.12 1 && \
   python -m ensurepip --upgrade && \
-  pip3 install --no-cache-dir wheel aiohttp rdflib && \
+  pip3 install --no-cache-dir build wheel aiohttp rdflib && \
   pip3 install --no-cache-dir pulsar-client==3.7.0 && \
   dnf clean all
 

containers/Containerfile.hf

@@ -11,7 +11,7 @@ ENV PIP_BREAK_SYSTEM_PACKAGES=1
 RUN dnf install -y python3.12 && \
   alternatives --install /usr/bin/python python /usr/bin/python3.12 1 && \
   python -m ensurepip --upgrade && \
-  pip3 install --no-cache-dir wheel aiohttp && \
+  pip3 install --no-cache-dir build wheel aiohttp && \
   pip3 install --no-cache-dir pulsar-client==3.7.0 && \
   dnf clean all
 

containers/Containerfile.mcp

@@ -0,0 +1,48 @@
+
+# ----------------------------------------------------------------------------
+# Build an AI container.  This does the torch install which is huge, and I
+# like to avoid re-doing this.
+# ----------------------------------------------------------------------------
+
+FROM docker.io/fedora:42 AS base
+
+ENV PIP_BREAK_SYSTEM_PACKAGES=1
+
+RUN dnf install -y python3.12 && \
+  alternatives --install /usr/bin/python python /usr/bin/python3.12 1 && \
+  python -m ensurepip --upgrade && \
+  pip3 install --no-cache-dir mcp websockets && \
+  dnf clean all
+
+# ----------------------------------------------------------------------------
+# Build a container which contains the built Python packages.  The build
+# creates a bunch of left-over cruft, a separate phase means this is only
+# needed to support package build
+# ----------------------------------------------------------------------------
+
+FROM base AS build
+
+COPY trustgraph-mcp/ /root/build/trustgraph-mcp/
+
+WORKDIR /root/build/
+
+RUN pip3 install --no-cache-dir build wheel
+
+RUN pip3 wheel -w /root/wheels/ --no-deps ./trustgraph-mcp/
+
+RUN ls /root/wheels
+
+# ----------------------------------------------------------------------------
+# Finally, the target container.  Start with base and add the package.
+# ----------------------------------------------------------------------------
+
+FROM base
+
+COPY --from=build /root/wheels /root/wheels
+
+RUN \
+    pip3 install --no-cache-dir /root/wheels/trustgraph_mcp-* && \
+    rm -rf /root/wheels
+
+WORKDIR /
+

containers/Containerfile.ocr

@@ -12,7 +12,7 @@ RUN dnf install -y python3.12 && \
   dnf install -y tesseract poppler-utils && \
   alternatives --install /usr/bin/python python /usr/bin/python3.12 1 && \
   python -m ensurepip --upgrade && \
-  pip3 install --no-cache-dir wheel aiohttp && \
+  pip3 install --no-cache-dir build wheel aiohttp && \
   pip3 install --no-cache-dir pulsar-client==3.7.0 && \
   dnf clean all
 

containers/Containerfile.vertexai

@@ -11,7 +11,7 @@ ENV PIP_BREAK_SYSTEM_PACKAGES=1
 RUN dnf install -y python3.12 && \
   alternatives --install /usr/bin/python python /usr/bin/python3.12 1 && \
   python -m ensurepip --upgrade && \
-  pip3 install --no-cache-dir wheel aiohttp && \
+  pip3 install --no-cache-dir build wheel aiohttp && \
   pip3 install --no-cache-dir pulsar-client==3.7.0 && \
   pip3 install --no-cache-dir google-cloud-aiplatform && \
   dnf clean all

docs/apis/api-flow.md

@@ -210,6 +210,51 @@ Request schema:
 Response schema:
 `trustgraph.schema.FlowResponse`
 
+## Flow Service Methods
+
+Flow instances provide access to various TrustGraph services through flow-specific endpoints:
+
+### MCP Tool Service - Invoke MCP Tools
+
+The `mcp_tool` method allows invoking MCP (Model Control Protocol) tools within a flow context.
+
+Request:
+```json
+{
+    "name": "file-reader",
+    "parameters": {
+        "path": "/path/to/file.txt"
+    }
+}
+```
+
+Response:
+```json
+{
+    "object": {"content": "file contents here", "size": 1024}
+}
+```
+
+Or for text responses:
+```json
+{
+    "text": "plain text response"
+}
+```
+
+### Other Service Methods
+
+Flow instances also provide access to:
+- `text_completion` - LLM text completion
+- `agent` - Agent question answering
+- `graph_rag` - Graph-based RAG queries
+- `document_rag` - Document-based RAG queries
+- `embeddings` - Text embeddings
+- `prompt` - Prompt template processing
+- `triples_query` - Knowledge graph queries
+- `load_document` - Document loading
+- `load_text` - Text loading
+
 ## Python SDK
 
 The Python SDK provides convenient access to the Flow API:
@@ -233,6 +278,10 @@ flows = await client.list_flows()
 
 # Stop a flow instance
 await client.stop_flow("flow-123")
+
+# Use flow instance services
+flow = client.id("flow-123")
+result = await flow.mcp_tool("file-reader", {"path": "/path/to/file.txt"})
 ```
 
 ## Features

docs/apis/api-librarian.md

@@ -12,6 +12,17 @@ The request contains the following fields:
 - `operation`: The operation to perform (see operations below)
 - `document_id`: Document identifier (for document operations)
 - `document_metadata`: Document metadata object (for add/update operations)
+  - `id`: Document identifier (required)
+  - `time`: Unix timestamp in seconds as a float (required for add operations)
+  - `kind`: MIME type of document (required, e.g., "text/plain", "application/pdf")
+  - `title`: Document title (optional)
+  - `comments`: Document comments (optional)
+  - `user`: Document owner (required)
+  - `tags`: Array of tags (optional)
+  - `metadata`: Array of RDF triples (optional) - each triple has:
+    - `s`: Subject with `v` (value) and `e` (is_uri boolean)
+    - `p`: Predicate with `v` (value) and `e` (is_uri boolean)
+    - `o`: Object with `v` (value) and `e` (is_uri boolean)
 - `content`: Document content as base64-encoded bytes (for add operations)
 - `processing_id`: Processing job identifier (for processing operations)
 - `processing_metadata`: Processing metadata object (for add-processing)
@@ -38,7 +49,7 @@ Request:
     "operation": "add-document",
     "document_metadata": {
         "id": "doc-123",
-        "time": 1640995200000,
+        "time": 1640995200.0,
         "kind": "application/pdf",
         "title": "Research Paper",
         "comments": "Important research findings",
@@ -46,9 +57,18 @@ Request:
         "tags": ["research", "ai", "machine-learning"],
         "metadata": [
             {
-                "subject": "doc-123",
+                "s": {
-                "predicate": "dc:creator",
+                    "v": "http://example.com/doc-123",
-                "object": "Dr. Smith"
+                    "e": true
+                },
+                "p": {
+                    "v": "http://purl.org/dc/elements/1.1/creator",
+                    "e": true
+                },
+                "o": {
+                    "v": "Dr. Smith",
+                    "e": false
+                }
             }
         ]
     },
@@ -77,7 +97,7 @@ Response:
 {
     "document_metadata": {
         "id": "doc-123",
-        "time": 1640995200000,
+        "time": 1640995200.0,
         "kind": "application/pdf",
         "title": "Research Paper",
         "comments": "Important research findings",
@@ -85,9 +105,18 @@ Response:
         "tags": ["research", "ai", "machine-learning"],
         "metadata": [
             {
-                "subject": "doc-123",
+                "s": {
-                "predicate": "dc:creator",
+                    "v": "http://example.com/doc-123",
-                "object": "Dr. Smith"
+                    "e": true
+                },
+                "p": {
+                    "v": "http://purl.org/dc/elements/1.1/creator",
+                    "e": true
+                },
+                "o": {
+                    "v": "Dr. Smith",
+                    "e": false
+                }
             }
         ]
     }
@@ -129,7 +158,7 @@ Response:
     "document_metadatas": [
         {
             "id": "doc-123",
-            "time": 1640995200000,
+            "time": 1640995200.0,
             "kind": "application/pdf",
             "title": "Research Paper",
             "comments": "Important research findings",
@@ -138,7 +167,7 @@ Response:
         },
         {
             "id": "doc-124",
-            "time": 1640995300000,
+            "time": 1640995300.0,
             "kind": "text/plain",
             "title": "Meeting Notes",
             "comments": "Team meeting discussion",
@@ -157,10 +186,12 @@ Request:
     "operation": "update-document",
     "document_metadata": {
         "id": "doc-123",
+        "time": 1640995500.0,
         "title": "Updated Research Paper",
         "comments": "Updated findings and conclusions",
         "user": "alice",
-        "tags": ["research", "ai", "machine-learning", "updated"]
+        "tags": ["research", "ai", "machine-learning", "updated"],
+        "metadata": []
     }
 }
 ```
@@ -197,7 +228,7 @@ Request:
     "processing_metadata": {
         "id": "proc-456",
         "document_id": "doc-123",
-        "time": 1640995400000,
+        "time": 1640995400.0,
         "flow": "pdf-extraction",
         "user": "alice",
         "collection": "research",
@@ -229,7 +260,7 @@ Response:
         {
             "id": "proc-456",
             "document_id": "doc-123",
-            "time": 1640995400000,
+            "time": 1640995400.0,
             "flow": "pdf-extraction",
             "user": "alice",
             "collection": "research",

docs/apis/api-mcp-tool.md

@@ -0,0 +1,137 @@
+# TrustGraph MCP Tool API
+
+This is a higher-level interface to the MCP (Model Control Protocol) tool service. The input
+specifies an MCP tool by name and parameters to pass to the tool.
+
+## Request/response
+
+### Request
+
+The request contains the following fields:
+- `name`: The MCP tool name
+- `parameters`: A set of key/values describing the tool parameters
+
+### Response
+
+The response contains either of these fields:
+- `text`: A plain text response
+- `object`: A structured object response
+
+## REST service
+
+The REST service accepts `name` and `parameters` fields, with parameters
+encoded as a JSON object.
+
+e.g.
+
+In this example, the MCP tool takes parameters and returns a
+structured response in the `object` field.
+
+Request:
+```
+{
+    "name": "file-reader",
+    "parameters": {
+        "path": "/path/to/file.txt"
+    }
+}
+```
+
+Response:
+
+```
+{
+    "object": {"content": "file contents here", "size": 1024}
+}
+```
+
+## Websocket
+
+Requests have `name` and `parameters` fields.
+
+e.g.
+
+Request:
+
+```
+{
+    "id": "akshfkiehfkseffh-142",
+    "service": "mcp-tool",
+    "flow": "default",
+    "request": {
+        "name": "file-reader",
+        "parameters": {
+            "path": "/path/to/file.txt"
+        }
+    }
+}
+```
+
+Responses:
+
+```
+{
+    "id": "akshfkiehfkseffh-142",
+    "response": {
+        "object": {"content": "file contents here", "size": 1024}
+    },
+    "complete": true
+}
+```
+
+e.g.
+
+An example which returns plain text
+
+Request:
+
+```
+{
+    "id": "akshfkiehfkseffh-141",
+    "service": "mcp-tool",
+    "request": {
+        "name": "calculator",
+        "parameters": {
+            "expression": "2 + 2"
+        }
+    }
+}
+```
+
+Response:
+
+```
+{
+    "id": "akshfkiehfkseffh-141",
+    "response": {
+        "text": "4"
+    },
+    "complete": true
+}
+```
+
+
+## Pulsar
+
+The Pulsar schema for the MCP Tool API is defined in Python code here:
+
+https://github.com/trustgraph-ai/trustgraph/blob/master/trustgraph-base/trustgraph/schema/mcp_tool.py
+
+Default request queue:
+`non-persistent://tg/request/mcp-tool`
+
+Default response queue:
+`non-persistent://tg/response/mcp-tool`
+
+Request schema:
+`trustgraph.schema.McpToolRequest`
+
+Response schema:
+`trustgraph.schema.McpToolResponse`
+
+## Pulsar Python client
+
+The client class is
+`trustgraph.clients.McpToolClient`
+
+https://github.com/trustgraph-ai/trustgraph/blob/master/trustgraph-base/trustgraph/clients/mcp_tool_client.py

docs/cli/tg-delete-mcp-tool.md

@@ -0,0 +1,374 @@
+# tg-delete-mcp-tool
+
+## Synopsis
+
+```
+tg-delete-mcp-tool [OPTIONS] --name NAME
+```
+
+## Description
+
+The `tg-delete-mcp-tool` command deletes MCP (Model Control Protocol) tools from the TrustGraph system. It removes MCP tool configurations by name from the 'mcp' configuration group. Once deleted, MCP tools are no longer available for agent use.
+
+This command is useful for:
+- Removing obsolete or deprecated MCP tools
+- Cleaning up MCP tool configurations
+- Managing MCP tool registry maintenance
+- Updating MCP tool deployments by removing old versions
+
+The command removes MCP tool configurations from the 'mcp' configuration group in the TrustGraph API.
+
+## Options
+
+- `-u, --api-url URL`
+  - TrustGraph API URL for configuration management
+  - Default: `http://localhost:8088/` (or `TRUSTGRAPH_URL` environment variable)
+  - Should point to a running TrustGraph API instance
+
+- `--name NAME`
+  - **Required.** MCP tool name to delete
+  - Must match an existing MCP tool name in the registry
+  - MCP tool will be completely removed from the system
+
+- `-h, --help`
+  - Show help message and exit
+
+## Examples
+
+### Basic MCP Tool Deletion
+
+Delete a weather MCP tool:
+```bash
+tg-delete-mcp-tool --name weather
+```
+
+### Calculator MCP Tool Deletion
+
+Delete a calculator MCP tool:
+```bash
+tg-delete-mcp-tool --name calculator
+```
+
+### Custom API URL
+
+Delete an MCP tool from a specific TrustGraph instance:
+```bash
+tg-delete-mcp-tool --api-url http://trustgraph.example.com:8088/ --name custom-mcp
+```
+
+### Batch MCP Tool Deletion
+
+Delete multiple MCP tools in a script:
+```bash
+#!/bin/bash
+# Delete obsolete MCP tools
+tg-delete-mcp-tool --name old-search
+tg-delete-mcp-tool --name deprecated-calc
+tg-delete-mcp-tool --name unused-mcp
+```
+
+### Conditional Deletion
+
+Delete an MCP tool only if it exists:
+```bash
+#!/bin/bash
+# Check if MCP tool exists before deletion
+if tg-show-mcp-tools | grep -q "test-mcp"; then
+    tg-delete-mcp-tool --name test-mcp
+    echo "MCP tool deleted"
+else
+    echo "MCP tool not found"
+fi
+```
+
+## Deletion Process
+
+The deletion process involves:
+
+1. **Existence Check**: Verify the MCP tool exists in the configuration
+2. **Configuration Removal**: Delete the MCP tool configuration from the 'mcp' group
+
+The command performs validation before deletion to ensure the tool exists.
+
+## Error Handling
+
+The command handles various error conditions:
+
+- **Tool not found**: If the specified MCP tool name doesn't exist
+- **API connection errors**: If the TrustGraph API is unavailable
+- **Configuration errors**: If the MCP tool configuration cannot be removed
+
+Common error scenarios:
+```bash
+# MCP tool not found
+tg-delete-mcp-tool --name nonexistent-mcp
+# Output: MCP tool 'nonexistent-mcp' not found.
+
+# Missing required field
+tg-delete-mcp-tool
+# Output: Exception: Must specify --name for MCP tool to delete
+
+# API connection error
+tg-delete-mcp-tool --api-url http://invalid-host:8088/ --name tool1
+# Output: Exception: [Connection error details]
+```
+
+## Verification
+
+The command provides feedback on the deletion process:
+
+- **Success**: `MCP tool 'tool-name' deleted successfully.`
+- **Not found**: `MCP tool 'tool-name' not found.`
+- **Error**: `Error deleting MCP tool 'tool-name': [error details]`
+
+## Advanced Usage
+
+### Safe Deletion with Verification
+
+Verify MCP tool exists before deletion:
+```bash
+#!/bin/bash
+MCP_NAME="weather"
+
+# Check if MCP tool exists
+if tg-show-mcp-tools | grep -q "^$MCP_NAME"; then
+    echo "Deleting MCP tool: $MCP_NAME"
+    tg-delete-mcp-tool --name "$MCP_NAME"
+    
+    # Verify deletion
+    if ! tg-show-mcp-tools | grep -q "^$MCP_NAME"; then
+        echo "MCP tool successfully deleted"
+    else
+        echo "MCP tool deletion failed"
+    fi
+else
+    echo "MCP tool $MCP_NAME not found"
+fi
+```
+
+### Backup Before Deletion
+
+Backup MCP tool configuration before deletion:
+```bash
+#!/bin/bash
+MCP_NAME="important-mcp"
+
+# Export MCP tool configuration
+echo "Backing up MCP tool configuration..."
+tg-show-mcp-tools | grep -A 10 "^$MCP_NAME" > "${MCP_NAME}_backup.txt"
+
+# Delete MCP tool
+echo "Deleting MCP tool..."
+tg-delete-mcp-tool --name "$MCP_NAME"
+
+echo "MCP tool deleted, backup saved to ${MCP_NAME}_backup.txt"
+```
+
+### Cleanup Script
+
+Clean up multiple MCP tools based on patterns:
+```bash
+#!/bin/bash
+# Delete all test MCP tools
+echo "Cleaning up test MCP tools..."
+
+# Get list of test MCP tools
+TEST_MCPS=$(tg-show-mcp-tools | grep "^test-" | cut -d: -f1)
+
+for mcp in $TEST_MCPS; do
+    echo "Deleting $mcp..."
+    tg-delete-mcp-tool --name "$mcp"
+done
+
+echo "Cleanup complete"
+```
+
+### Environment-Specific Deletion
+
+Delete MCP tools from specific environments:
+```bash
+#!/bin/bash
+# Delete development MCP tools from production
+export TRUSTGRAPH_URL="http://prod.trustgraph.com:8088/"
+
+DEV_MCPS=("dev-mcp" "debug-mcp" "test-helper")
+
+for mcp in "${DEV_MCPS[@]}"; do
+    echo "Removing development MCP tool: $mcp"
+    tg-delete-mcp-tool --name "$mcp"
+done
+```
+
+### MCP Service Shutdown
+
+Remove MCP tools when services are decommissioned:
+```bash
+#!/bin/bash
+# Remove MCP tools for decommissioned service
+SERVICE_NAME="old-service"
+
+# Find MCP tools for this service
+MCP_TOOLS=$(tg-show-mcp-tools | grep "$SERVICE_NAME" | cut -d: -f1)
+
+for tool in $MCP_TOOLS; do
+    echo "Removing MCP tool for decommissioned service: $tool"
+    tg-delete-mcp-tool --name "$tool"
+done
+```
+
+## Integration with Other Commands
+
+### With MCP Tool Management
+
+List and delete MCP tools:
+```bash
+# List all MCP tools
+tg-show-mcp-tools
+
+# Delete specific MCP tool
+tg-delete-mcp-tool --name unwanted-mcp
+
+# Verify deletion
+tg-show-mcp-tools | grep unwanted-mcp
+```
+
+### With Configuration Management
+
+Manage MCP tool configurations:
+```bash
+# View current configuration
+tg-show-config
+
+# Delete MCP tool
+tg-delete-mcp-tool --name old-mcp
+
+# View updated configuration
+tg-show-config
+```
+
+### With MCP Tool Invocation
+
+Ensure MCP tools can't be invoked after deletion:
+```bash
+# Delete MCP tool
+tg-delete-mcp-tool --name deprecated-mcp
+
+# Verify tool is no longer available
+tg-invoke-mcp-tool --name deprecated-mcp
+# Should fail with tool not found error
+```
+
+## Best Practices
+
+1. **Verification**: Always verify MCP tool exists before deletion
+2. **Backup**: Backup important MCP tool configurations before deletion
+3. **Dependencies**: Check for MCP tool dependencies before deletion
+4. **Service Coordination**: Coordinate with MCP service owners before deletion
+5. **Testing**: Test system functionality after MCP tool deletion
+6. **Documentation**: Document reasons for MCP tool deletion
+7. **Gradual Removal**: Remove MCP tools gradually in production environments
+8. **Monitoring**: Monitor for errors after MCP tool deletion
+
+## Troubleshooting
+
+### MCP Tool Not Found
+
+If MCP tool deletion reports "not found":
+1. Verify the MCP tool name is correct
+2. Check MCP tool exists with `tg-show-mcp-tools`
+3. Ensure you're connected to the correct TrustGraph instance
+4. Check for case sensitivity in MCP tool name
+
+### Deletion Errors
+
+If deletion fails:
+1. Check TrustGraph API connectivity
+2. Verify API permissions
+3. Check for configuration corruption
+4. Retry the deletion operation
+5. Check MCP service status
+
+### Permission Errors
+
+If deletion fails due to permissions:
+1. Verify API access credentials
+2. Check TrustGraph API permissions
+3. Ensure proper authentication
+4. Contact system administrator if needed
+
+## Recovery
+
+### Restore Deleted MCP Tool
+
+If an MCP tool was accidentally deleted:
+1. Use backup configuration if available
+2. Re-register the MCP tool with `tg-set-mcp-tool`
+3. Restore from version control if MCP tool definitions are tracked
+4. Contact system administrator for recovery options
+
+### Verify System State
+
+After deletion, verify system state:
+```bash
+# Check MCP tool registry
+tg-show-mcp-tools
+
+# Verify no orphaned configurations
+tg-show-config | grep "mcp\."
+
+# Test MCP tool functionality
+tg-invoke-mcp-tool --name remaining-tool
+```
+
+## MCP Tool Lifecycle
+
+### Development to Production
+
+Manage MCP tool lifecycle:
+```bash
+#!/bin/bash
+# Promote MCP tool from dev to production
+
+# Remove development version
+tg-delete-mcp-tool --name dev-tool
+
+# Add production version
+tg-set-mcp-tool --name prod-tool --tool-url "http://prod.mcp.com/api"
+```
+
+### Version Management
+
+Manage MCP tool versions:
+```bash
+#!/bin/bash
+# Update MCP tool to new version
+
+# Remove old version
+tg-delete-mcp-tool --name tool-v1
+
+# Add new version
+tg-set-mcp-tool --name tool-v2 --tool-url "http://new.mcp.com/api"
+```
+
+## Security Considerations
+
+When deleting MCP tools:
+
+1. **Access Control**: Ensure proper authorization for deletion
+2. **Audit Trail**: Log MCP tool deletions for security auditing
+3. **Impact Assessment**: Assess security impact of tool removal
+4. **Credential Cleanup**: Remove associated credentials if applicable
+5. **Network Security**: Update firewall rules if MCP endpoints are no longer needed
+
+## Related Commands
+
+- [`tg-show-mcp-tools`](tg-show-mcp-tools.md) - Display registered MCP tools
+- [`tg-set-mcp-tool`](tg-set-mcp-tool.md) - Configure and register MCP tools
+- [`tg-invoke-mcp-tool`](tg-invoke-mcp-tool.md) - Execute MCP tools
+- [`tg-delete-tool`](tg-delete-tool.md) - Delete regular agent tools
+
+## See Also
+
+- MCP Protocol Documentation
+- TrustGraph MCP Integration Guide
+- MCP Tool Management Manual

docs/cli/tg-delete-tool.md

@@ -0,0 +1,317 @@
+# tg-delete-tool
+
+## Synopsis
+
+```
+tg-delete-tool [OPTIONS] --id ID
+```
+
+## Description
+
+The `tg-delete-tool` command deletes tools from the TrustGraph system. It removes tool configurations by ID from the agent configuration and updates the tool index accordingly. Once deleted, tools are no longer available for agent use.
+
+This command is useful for:
+- Removing obsolete or deprecated tools
+- Cleaning up tool configurations
+- Managing tool registry maintenance
+- Updating tool deployments by removing old versions
+
+The command removes both the tool from the tool index and deletes the complete tool configuration from the TrustGraph API.
+
+## Options
+
+- `-u, --api-url URL`
+  - TrustGraph API URL for configuration management
+  - Default: `http://localhost:8088/` (or `TRUSTGRAPH_URL` environment variable)
+  - Should point to a running TrustGraph API instance
+
+- `--id ID`
+  - **Required.** Tool ID to delete
+  - Must match an existing tool ID in the registry
+  - Tool will be completely removed from the system
+
+- `-h, --help`
+  - Show help message and exit
+
+## Examples
+
+### Basic Tool Deletion
+
+Delete a weather tool:
+```bash
+tg-delete-tool --id weather
+```
+
+### Calculator Tool Deletion
+
+Delete a calculator tool:
+```bash
+tg-delete-tool --id calculator
+```
+
+### Custom API URL
+
+Delete a tool from a specific TrustGraph instance:
+```bash
+tg-delete-tool --api-url http://trustgraph.example.com:8088/ --id custom-tool
+```
+
+### Batch Tool Deletion
+
+Delete multiple tools in a script:
+```bash
+#!/bin/bash
+# Delete obsolete tools
+tg-delete-tool --id old-search
+tg-delete-tool --id deprecated-calc
+tg-delete-tool --id unused-tool
+```
+
+### Conditional Deletion
+
+Delete a tool only if it exists:
+```bash
+#!/bin/bash
+# Check if tool exists before deletion
+if tg-show-tools | grep -q "test-tool"; then
+    tg-delete-tool --id test-tool
+    echo "Tool deleted"
+else
+    echo "Tool not found"
+fi
+```
+
+## Deletion Process
+
+The deletion process involves two steps:
+
+1. **Index Update**: Remove the tool ID from the tool index
+2. **Configuration Removal**: Delete the tool configuration data
+
+Both operations must succeed for the deletion to be complete.
+
+## Error Handling
+
+The command handles various error conditions:
+
+- **Tool not found**: If the specified tool ID doesn't exist
+- **Missing configuration**: If tool is in index but configuration is missing
+- **API connection errors**: If the TrustGraph API is unavailable
+- **Partial deletion**: If index update or configuration removal fails
+
+Common error scenarios:
+```bash
+# Tool not found
+tg-delete-tool --id nonexistent-tool
+# Output: Tool 'nonexistent-tool' not found in tool index.
+
+# Missing required field
+tg-delete-tool
+# Output: Exception: Must specify --id for tool to delete
+
+# API connection error
+tg-delete-tool --api-url http://invalid-host:8088/ --id tool1
+# Output: Exception: [Connection error details]
+```
+
+## Verification
+
+The command provides feedback on the deletion process:
+
+- **Success**: `Tool 'tool-id' deleted successfully.`
+- **Not found**: `Tool 'tool-id' not found in tool index.`
+- **Configuration missing**: `Tool configuration for 'tool-id' not found.`
+- **Error**: `Error deleting tool 'tool-id': [error details]`
+
+## Advanced Usage
+
+### Safe Deletion with Verification
+
+Verify tool exists before deletion:
+```bash
+#!/bin/bash
+TOOL_ID="weather"
+
+# Check if tool exists
+if tg-show-tools | grep -q "^$TOOL_ID:"; then
+    echo "Deleting tool: $TOOL_ID"
+    tg-delete-tool --id "$TOOL_ID"
+    
+    # Verify deletion
+    if ! tg-show-tools | grep -q "^$TOOL_ID:"; then
+        echo "Tool successfully deleted"
+    else
+        echo "Tool deletion failed"
+    fi
+else
+    echo "Tool $TOOL_ID not found"
+fi
+```
+
+### Backup Before Deletion
+
+Backup tool configuration before deletion:
+```bash
+#!/bin/bash
+TOOL_ID="important-tool"
+
+# Export tool configuration
+echo "Backing up tool configuration..."
+tg-show-tools | grep -A 20 "^$TOOL_ID:" > "${TOOL_ID}_backup.txt"
+
+# Delete tool
+echo "Deleting tool..."
+tg-delete-tool --id "$TOOL_ID"
+
+echo "Tool deleted, backup saved to ${TOOL_ID}_backup.txt"
+```
+
+### Cleanup Script
+
+Clean up multiple tools based on patterns:
+```bash
+#!/bin/bash
+# Delete all test tools
+echo "Cleaning up test tools..."
+
+# Get list of test tools
+TEST_TOOLS=$(tg-show-tools | grep "^test-" | cut -d: -f1)
+
+for tool in $TEST_TOOLS; do
+    echo "Deleting $tool..."
+    tg-delete-tool --id "$tool"
+done
+
+echo "Cleanup complete"
+```
+
+### Environment-Specific Deletion
+
+Delete tools from specific environments:
+```bash
+#!/bin/bash
+# Delete development tools from production
+export TRUSTGRAPH_URL="http://prod.trustgraph.com:8088/"
+
+DEV_TOOLS=("dev-tool" "debug-tool" "test-helper")
+
+for tool in "${DEV_TOOLS[@]}"; do
+    echo "Removing development tool: $tool"
+    tg-delete-tool --id "$tool"
+done
+```
+
+## Integration with Other Commands
+
+### With Tool Management
+
+List and delete tools:
+```bash
+# List all tools
+tg-show-tools
+
+# Delete specific tool
+tg-delete-tool --id unwanted-tool
+
+# Verify deletion
+tg-show-tools | grep unwanted-tool
+```
+
+### With Configuration Management
+
+Manage tool configurations:
+```bash
+# View current configuration
+tg-show-config
+
+# Delete tool
+tg-delete-tool --id old-tool
+
+# View updated configuration
+tg-show-config
+```
+
+### With Agent Workflows
+
+Ensure agents don't use deleted tools:
+```bash
+# Delete tool
+tg-delete-tool --id deprecated-tool
+
+# Check agent configuration
+tg-show-config | grep deprecated-tool
+```
+
+## Best Practices
+
+1. **Verification**: Always verify tool exists before deletion
+2. **Backup**: Backup important tool configurations before deletion
+3. **Dependencies**: Check for tool dependencies before deletion
+4. **Testing**: Test system functionality after tool deletion
+5. **Documentation**: Document reasons for tool deletion
+6. **Gradual Removal**: Remove tools gradually in production environments
+7. **Monitoring**: Monitor for errors after tool deletion
+
+## Troubleshooting
+
+### Tool Not Found
+
+If tool deletion reports "not found":
+1. Verify the tool ID is correct
+2. Check tool exists with `tg-show-tools`
+3. Ensure you're connected to the correct TrustGraph instance
+4. Check for case sensitivity in tool ID
+
+### Partial Deletion
+
+If deletion partially fails:
+1. Check TrustGraph API connectivity
+2. Verify API permissions
+3. Check for configuration corruption
+4. Retry the deletion operation
+5. Manual cleanup may be required
+
+### Permission Errors
+
+If deletion fails due to permissions:
+1. Verify API access credentials
+2. Check TrustGraph API permissions
+3. Ensure proper authentication
+4. Contact system administrator if needed
+
+## Recovery
+
+### Restore Deleted Tool
+
+If a tool was accidentally deleted:
+1. Use backup configuration if available
+2. Re-register the tool with `tg-set-tool`
+3. Restore from version control if tool definitions are tracked
+4. Contact system administrator for recovery options
+
+### Verify System State
+
+After deletion, verify system state:
+```bash
+# Check tool index consistency
+tg-show-tools
+
+# Verify no orphaned configurations
+tg-show-config | grep "tool\."
+
+# Test agent functionality
+tg-invoke-agent --prompt "Test prompt"
+```
+
+## Related Commands
+
+- [`tg-show-tools`](tg-show-tools.md) - Display registered tools
+- [`tg-set-tool`](tg-set-tool.md) - Configure and register tools
+- [`tg-delete-mcp-tool`](tg-delete-mcp-tool.md) - Delete MCP tools
+- [`tg-show-config`](tg-show-config.md) - View system configuration
+
+## See Also
+
+- TrustGraph Tool Management Guide
+- Agent Configuration Documentation
+- System Administration Manual

docs/cli/tg-invoke-mcp-tool.md

@@ -0,0 +1,448 @@
+# tg-invoke-mcp-tool
+
+Invokes MCP (Model Control Protocol) tools through the TrustGraph API with parameter support.
+
+## Synopsis
+
+```bash
+tg-invoke-mcp-tool [options] -n tool-name [-P parameters]
+```
+
+## Description
+
+The `tg-invoke-mcp-tool` command invokes MCP (Model Control Protocol) tools through the TrustGraph API. MCP tools are external services that provide standardized interfaces for AI model interactions within the TrustGraph ecosystem.
+
+MCP tools offer extensible functionality with consistent APIs, stateful interactions, and built-in security mechanisms. They can be used for various purposes including file operations, calculations, web requests, database queries, and custom integrations.
+
+## Options
+
+### Required Arguments
+
+- `-n, --name TOOL_NAME`: MCP tool name to invoke
+
+### Optional Arguments
+
+- `-u, --url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
+- `-f, --flow-id ID`: Flow instance ID to use (default: `default`)
+- `-P, --parameters JSON`: Tool parameters as JSON-encoded dictionary
+
+## Examples
+
+### Basic Tool Invocation
+```bash
+tg-invoke-mcp-tool -n weather
+```
+
+### Tool with Parameters
+```bash
+tg-invoke-mcp-tool -n calculator -P '{"expression": "2 + 2"}'
+```
+
+### File Operations
+```bash
+tg-invoke-mcp-tool -n file-reader -P '{"path": "/path/to/file.txt"}'
+```
+
+### Web Request Tool
+```bash
+tg-invoke-mcp-tool -n http-client -P '{"url": "https://api.example.com/data", "method": "GET"}'
+```
+
+### Database Query
+```bash
+tg-invoke-mcp-tool -n database -P '{"query": "SELECT * FROM users LIMIT 10", "database": "main"}'
+```
+
+### Custom Flow and API URL
+```bash
+tg-invoke-mcp-tool -u http://custom-api:8088/ -f my-flow -n weather -P '{"location": "London"}'
+```
+
+## Parameter Format
+
+### Simple Parameters
+```bash
+tg-invoke-mcp-tool -n calculator -P '{"operation": "add", "a": 10, "b": 5}'
+```
+
+### Complex Parameters
+```bash
+tg-invoke-mcp-tool -n data-processor -P '{
+  "input_data": [1, 2, 3, 4, 5],
+  "operations": ["sum", "average", "max"],
+  "output_format": "json"
+}'
+```
+
+### File Input Parameters
+```bash
+tg-invoke-mcp-tool -n text-analyzer -P "{\"text\": \"$(cat document.txt)\", \"analysis_type\": \"sentiment\"}"
+```
+
+### Multiple Parameters
+```bash
+tg-invoke-mcp-tool -n report-generator -P '{
+  "template": "monthly-report",
+  "data_source": "sales_database",
+  "period": "2024-01",
+  "format": "pdf",
+  "recipients": ["admin@example.com"]
+}'
+```
+
+## Common MCP Tools
+
+### File Operations
+```bash
+# Read file content
+tg-invoke-mcp-tool -n file-reader -P '{"path": "/path/to/file.txt"}'
+
+# Write file content
+tg-invoke-mcp-tool -n file-writer -P '{"path": "/path/to/output.txt", "content": "Hello World"}'
+
+# List directory contents
+tg-invoke-mcp-tool -n directory-lister -P '{"path": "/home/user", "recursive": false}'
+```
+
+### Data Processing
+```bash
+# JSON processing
+tg-invoke-mcp-tool -n json-processor -P '{"data": "{\"key\": \"value\"}", "operation": "validate"}'
+
+# CSV analysis
+tg-invoke-mcp-tool -n csv-analyzer -P '{"file": "data.csv", "columns": ["name", "age"], "operation": "statistics"}'
+
+# Text transformation
+tg-invoke-mcp-tool -n text-transformer -P '{"text": "Hello World", "operation": "uppercase"}'
+```
+
+### Web and API
+```bash
+# HTTP requests
+tg-invoke-mcp-tool -n http-client -P '{"url": "https://api.github.com/users/octocat", "method": "GET"}'
+
+# Web scraping
+tg-invoke-mcp-tool -n web-scraper -P '{"url": "https://example.com", "selector": "h1"}'
+
+# API testing
+tg-invoke-mcp-tool -n api-tester -P '{"endpoint": "/api/v1/users", "method": "POST", "payload": {"name": "John"}}'
+```
+
+### Database Operations
+```bash
+# Query execution
+tg-invoke-mcp-tool -n database -P '{"query": "SELECT COUNT(*) FROM users", "database": "production"}'
+
+# Schema inspection
+tg-invoke-mcp-tool -n db-inspector -P '{"database": "main", "operation": "list_tables"}'
+
+# Data migration
+tg-invoke-mcp-tool -n db-migrator -P '{"source": "old_db", "target": "new_db", "table": "users"}'
+```
+
+## Output Formats
+
+### String Response
+```bash
+tg-invoke-mcp-tool -n calculator -P '{"expression": "10 + 5"}'
+# Output: "15"
+```
+
+### JSON Response
+```bash
+tg-invoke-mcp-tool -n weather -P '{"location": "New York"}'
+# Output:
+# {
+#     "location": "New York",
+#     "temperature": 22,
+#     "conditions": "sunny",
+#     "humidity": 45
+# }
+```
+
+### Complex Object Response
+```bash
+tg-invoke-mcp-tool -n data-analyzer -P '{"dataset": "sales.csv"}'
+# Output:
+# {
+#     "summary": {
+#         "total_records": 1000,
+#         "columns": ["date", "product", "amount"],
+#         "date_range": "2024-01-01 to 2024-12-31"
+#     },
+#     "statistics": {
+#         "total_sales": 50000,
+#         "average_transaction": 50.0,
+#         "top_product": "Widget A"
+#     }
+# }
+```
+
+## Error Handling
+
+### Tool Not Found
+```bash
+Exception: MCP tool 'nonexistent-tool' not found
+```
+**Solution**: Check available tools with `tg-show-mcp-tools`.
+
+### Invalid Parameters
+```bash
+Exception: Invalid JSON in parameters: Expecting property name enclosed in double quotes
+```
+**Solution**: Verify JSON parameter format and escape special characters.
+
+### Missing Required Parameters
+```bash
+Exception: Required parameter 'input_data' not provided
+```
+**Solution**: Check tool documentation for required parameters.
+
+### Flow Not Found
+```bash
+Exception: Flow instance 'invalid-flow' not found
+```
+**Solution**: Verify flow ID exists with `tg-show-flows`.
+
+### Tool Execution Error
+```bash
+Exception: Tool execution failed: Connection timeout
+```
+**Solution**: Check network connectivity and tool service availability.
+
+## Advanced Usage
+
+### Batch Processing
+```bash
+# Process multiple files
+for file in *.txt; do
+    echo "Processing $file..."
+    tg-invoke-mcp-tool -n text-analyzer -P "{\"file\": \"$file\", \"analysis\": \"sentiment\"}"
+done
+```
+
+### Error Handling in Scripts
+```bash
+#!/bin/bash
+# robust-tool-invoke.sh
+tool_name="$1"
+parameters="$2"
+
+if ! result=$(tg-invoke-mcp-tool -n "$tool_name" -P "$parameters" 2>&1); then
+    echo "Error invoking tool: $result" >&2
+    exit 1
+fi
+
+echo "Success: $result"
+```
+
+### Pipeline Processing
+```bash
+# Chain multiple tools
+data=$(tg-invoke-mcp-tool -n data-loader -P '{"source": "database"}')
+processed=$(tg-invoke-mcp-tool -n data-processor -P "{\"data\": \"$data\", \"operation\": \"clean\"}")
+tg-invoke-mcp-tool -n report-generator -P "{\"data\": \"$processed\", \"format\": \"pdf\"}"
+```
+
+### Configuration-Driven Invocation
+```bash
+# Use configuration file
+config_file="tool-config.json"
+tool_name=$(jq -r '.tool' "$config_file")
+parameters=$(jq -c '.parameters' "$config_file")
+
+tg-invoke-mcp-tool -n "$tool_name" -P "$parameters"
+```
+
+### Interactive Tool Usage
+```bash
+#!/bin/bash
+# interactive-mcp-tool.sh
+echo "Available tools:"
+tg-show-mcp-tools
+
+read -p "Enter tool name: " tool_name
+read -p "Enter parameters (JSON): " parameters
+
+echo "Invoking tool..."
+tg-invoke-mcp-tool -n "$tool_name" -P "$parameters"
+```
+
+### Parallel Tool Execution
+```bash
+# Execute multiple tools in parallel
+tools=("weather" "calculator" "file-reader")
+params=('{"location": "NYC"}' '{"expression": "2+2"}' '{"path": "file.txt"}')
+
+for i in "${!tools[@]}"; do
+    (
+        echo "Executing ${tools[$i]}..."
+        tg-invoke-mcp-tool -n "${tools[$i]}" -P "${params[$i]}" > "result-${tools[$i]}.json"
+    ) &
+done
+wait
+```
+
+## Tool Management
+
+### List Available Tools
+```bash
+# Show all registered MCP tools
+tg-show-mcp-tools
+```
+
+### Register New Tools
+```bash
+# Register a new MCP tool
+tg-set-mcp-tool weather-service "http://weather-api:8080/mcp" "Weather data provider"
+```
+
+### Remove Tools
+```bash
+# Remove an MCP tool
+tg-delete-mcp-tool weather-service
+```
+
+## Use Cases
+
+### Data Processing Workflows
+```bash
+# Extract, transform, and load data
+raw_data=$(tg-invoke-mcp-tool -n data-extractor -P '{"source": "external_api"}')
+clean_data=$(tg-invoke-mcp-tool -n data-cleaner -P "{\"data\": \"$raw_data\"}")
+tg-invoke-mcp-tool -n data-loader -P "{\"data\": \"$clean_data\", \"target\": \"warehouse\"}"
+```
+
+### Automation Scripts
+```bash
+# Automated system monitoring
+status=$(tg-invoke-mcp-tool -n system-monitor -P '{"checks": ["cpu", "memory", "disk"]}')
+if echo "$status" | grep -q "warning"; then
+    tg-invoke-mcp-tool -n alert-system -P "{\"message\": \"System warning detected\", \"severity\": \"medium\"}"
+fi
+```
+
+### Integration Testing
+```bash
+# Test API endpoints
+endpoints=("/api/users" "/api/orders" "/api/products")
+for endpoint in "${endpoints[@]}"; do
+    result=$(tg-invoke-mcp-tool -n api-tester -P "{\"endpoint\": \"$endpoint\", \"method\": \"GET\"}")
+    echo "Testing $endpoint: $result"
+done
+```
+
+### Content Generation
+```bash
+# Generate documentation
+code_analysis=$(tg-invoke-mcp-tool -n code-analyzer -P '{"directory": "./src", "language": "python"}')
+tg-invoke-mcp-tool -n doc-generator -P "{\"analysis\": \"$code_analysis\", \"format\": \"markdown\"}"
+```
+
+## Performance Optimization
+
+### Caching Tool Results
+```bash
+# Cache expensive tool operations
+cache_dir="mcp-cache"
+mkdir -p "$cache_dir"
+
+invoke_with_cache() {
+    local tool="$1"
+    local params="$2"
+    local cache_key=$(echo "$tool-$params" | md5sum | cut -d' ' -f1)
+    local cache_file="$cache_dir/$cache_key.json"
+    
+    if [ -f "$cache_file" ]; then
+        echo "Cache hit for $tool"
+        cat "$cache_file"
+    else
+        echo "Cache miss, invoking $tool..."
+        tg-invoke-mcp-tool -n "$tool" -P "$params" | tee "$cache_file"
+    fi
+}
+```
+
+### Asynchronous Processing
+```bash
+# Non-blocking tool execution
+async_invoke() {
+    local tool="$1"
+    local params="$2"
+    local output_file="$3"
+    
+    tg-invoke-mcp-tool -n "$tool" -P "$params" > "$output_file" 2>&1 &
+    echo $!  # Return process ID
+}
+
+# Execute multiple tools asynchronously
+pid1=$(async_invoke "data-processor" '{"file": "data1.csv"}' "result1.json")
+pid2=$(async_invoke "data-processor" '{"file": "data2.csv"}' "result2.json")
+
+# Wait for completion
+wait $pid1 $pid2
+```
+
+## Environment Variables
+
+- `TRUSTGRAPH_URL`: Default API URL
+
+## Related Commands
+
+- [`tg-show-mcp-tools`](tg-show-mcp-tools.md) - List available MCP tools
+- [`tg-set-mcp-tool`](tg-set-mcp-tool.md) - Register MCP tools
+- [`tg-delete-mcp-tool`](tg-delete-mcp-tool.md) - Remove MCP tools
+- [`tg-show-flows`](tg-show-flows.md) - List available flow instances
+- [`tg-invoke-prompt`](tg-invoke-prompt.md) - Invoke prompt templates
+
+## API Integration
+
+This command uses the TrustGraph API flow interface to execute MCP tools within the context of specified flows. MCP tools are external services that implement the Model Control Protocol for standardized AI tool interactions.
+
+## Best Practices
+
+1. **Parameter Validation**: Always validate JSON parameters before execution
+2. **Error Handling**: Implement robust error handling for production use
+3. **Tool Discovery**: Use `tg-show-mcp-tools` to discover available tools
+4. **Resource Management**: Consider performance implications of long-running tools
+5. **Security**: Avoid passing sensitive data in parameters; use secure tool configurations
+6. **Documentation**: Document custom tool parameters and expected responses
+7. **Testing**: Test tool integrations thoroughly before production deployment
+
+## Troubleshooting
+
+### Tool Not Available
+```bash
+# Check tool registration
+tg-show-mcp-tools | grep "tool-name"
+
+# Verify tool service is running
+curl -f http://tool-service:8080/health
+```
+
+### Parameter Issues
+```bash
+# Validate JSON format
+echo '{"key": "value"}' | jq .
+
+# Test with minimal parameters
+tg-invoke-mcp-tool -n tool-name -P '{}'
+```
+
+### Flow Problems
+```bash
+# Check flow status
+tg-show-flows | grep "flow-id"
+
+# Verify flow supports MCP tools
+tg-get-flow-class -n "flow-class" | jq '.interfaces.mcp_tool'
+```
+
+### Connection Issues
+```bash
+# Test API connectivity
+curl -f http://localhost:8088/health
+
+# Check environment variables
+echo $TRUSTGRAPH_URL
+```

docs/cli/tg-set-mcp-tool.md

@@ -0,0 +1,267 @@
+# tg-set-mcp-tool
+
+## Synopsis
+
+```
+tg-set-mcp-tool [OPTIONS] --name NAME --tool-url URL
+```
+
+## Description
+
+The `tg-set-mcp-tool` command configures and registers MCP (Model Control Protocol) tools in the TrustGraph system. It allows defining MCP tool configurations with name and URL. Tools are stored in the 'mcp' configuration group for discovery and execution.
+
+This command is useful for:
+- Registering MCP tool endpoints for agent use
+- Configuring external MCP server connections
+- Managing MCP tool registry for agent workflows
+- Integrating third-party MCP tools into TrustGraph
+
+The command stores MCP tool configurations in the 'mcp' configuration group, separate from regular agent tools.
+
+## Options
+
+- `-u, --api-url URL`
+  - TrustGraph API URL for configuration storage
+  - Default: `http://localhost:8088/` (or `TRUSTGRAPH_URL` environment variable)
+  - Should point to a running TrustGraph API instance
+
+- `--name NAME`
+  - **Required.** MCP tool name identifier
+  - Used to reference the MCP tool in configurations
+  - Must be unique within the MCP tool registry
+
+- `--tool-url URL`
+  - **Required.** MCP tool URL endpoint
+  - Should point to the MCP server endpoint providing the tool functionality
+  - Must be a valid URL accessible by the TrustGraph system
+
+- `-h, --help`
+  - Show help message and exit
+
+## Examples
+
+### Basic MCP Tool Registration
+
+Register a weather service MCP tool:
+```bash
+tg-set-mcp-tool --name weather --tool-url "http://localhost:3000/weather"
+```
+
+### Calculator MCP Tool
+
+Register a calculator MCP tool:
+```bash
+tg-set-mcp-tool --name calculator --tool-url "http://mcp-tools.example.com/calc"
+```
+
+### Remote MCP Service
+
+Register a remote MCP service:
+```bash
+tg-set-mcp-tool --name document-processor \
+                --tool-url "https://api.example.com/mcp/documents"
+```
+
+### Custom API URL
+
+Register MCP tool with custom TrustGraph API:
+```bash
+tg-set-mcp-tool -u http://trustgraph.example.com:8088/ \
+                --name custom-mcp --tool-url "http://custom.mcp.com/api"
+```
+
+### Local Development Setup
+
+Register MCP tools for local development:
+```bash
+tg-set-mcp-tool --name dev-tool --tool-url "http://localhost:8080/mcp"
+```
+
+## MCP Tool Configuration
+
+MCP tools are configured with minimal metadata:
+
+- **name**: Unique identifier for the tool
+- **url**: Endpoint URL for the MCP server
+
+The configuration is stored as JSON in the 'mcp' configuration group:
+```json
+{
+  "name": "weather",
+  "url": "http://localhost:3000/weather"
+}
+```
+
+## Advanced Usage
+
+### Updating Existing MCP Tools
+
+Update an existing MCP tool configuration:
+```bash
+# Update MCP tool URL
+tg-set-mcp-tool --name weather --tool-url "http://new-weather-server:3000/api"
+```
+
+### Batch MCP Tool Registration
+
+Register multiple MCP tools in a script:
+```bash
+#!/bin/bash
+# Register a suite of MCP tools
+tg-set-mcp-tool --name search --tool-url "http://search-mcp:3000/api"
+tg-set-mcp-tool --name translate --tool-url "http://translate-mcp:3000/api"
+tg-set-mcp-tool --name summarize --tool-url "http://summarize-mcp:3000/api"
+```
+
+### Environment-Specific Configuration
+
+Configure MCP tools for different environments:
+```bash
+# Development environment
+export TRUSTGRAPH_URL="http://dev.trustgraph.com:8088/"
+tg-set-mcp-tool --name dev-mcp --tool-url "http://dev.mcp.com/api"
+
+# Production environment
+export TRUSTGRAPH_URL="http://prod.trustgraph.com:8088/"
+tg-set-mcp-tool --name prod-mcp --tool-url "http://prod.mcp.com/api"
+```
+
+### MCP Tool Validation
+
+Verify MCP tool registration:
+```bash
+# Register MCP tool and verify
+tg-set-mcp-tool --name test-mcp --tool-url "http://test.mcp.com/api"
+
+# Check if MCP tool was registered
+tg-show-mcp-tools | grep test-mcp
+```
+
+## Error Handling
+
+The command handles various error conditions:
+
+- **Missing required arguments**: Both name and tool-url must be provided
+- **Invalid URLs**: Tool URLs must be valid and accessible
+- **API connection errors**: If the TrustGraph API is unavailable
+- **Configuration errors**: If MCP tool data cannot be stored
+
+Common error scenarios:
+```bash
+# Missing required field
+tg-set-mcp-tool --name tool1
+# Output: Exception: Must specify --tool-url for MCP tool
+
+# Missing name
+tg-set-mcp-tool --tool-url "http://example.com/mcp"
+# Output: Exception: Must specify --name for MCP tool
+
+# Invalid API URL
+tg-set-mcp-tool -u "invalid-url" --name tool1 --tool-url "http://mcp.com"
+# Output: Exception: [API connection error]
+```
+
+## Integration with Other Commands
+
+### With MCP Tool Management
+
+View registered MCP tools:
+```bash
+# Register MCP tool
+tg-set-mcp-tool --name new-mcp --tool-url "http://new.mcp.com/api"
+
+# View all MCP tools
+tg-show-mcp-tools
+```
+
+### With Agent Workflows
+
+Use MCP tools in agent workflows:
+```bash
+# Register MCP tool
+tg-set-mcp-tool --name weather --tool-url "http://weather.mcp.com/api"
+
+# Invoke MCP tool directly
+tg-invoke-mcp-tool --name weather --input "location=London"
+```
+
+### With Configuration Management
+
+MCP tools integrate with configuration management:
+```bash
+# Register MCP tool
+tg-set-mcp-tool --name config-mcp --tool-url "http://config.mcp.com/api"
+
+# View configuration including MCP tools
+tg-show-config
+```
+
+## Best Practices
+
+1. **Clear Naming**: Use descriptive, unique MCP tool names
+2. **Reliable URLs**: Ensure MCP endpoints are stable and accessible
+3. **Health Checks**: Verify MCP endpoints are operational before registration
+4. **Documentation**: Document MCP tool capabilities and usage
+5. **Error Handling**: Implement proper error handling for MCP endpoints
+6. **Security**: Use secure URLs (HTTPS) when possible
+7. **Monitoring**: Monitor MCP tool availability and performance
+
+## Troubleshooting
+
+### MCP Tool Not Appearing
+
+If a registered MCP tool doesn't appear in listings:
+1. Verify the MCP tool was registered successfully
+2. Check MCP tool registry with `tg-show-mcp-tools`
+3. Ensure the API URL is correct
+4. Verify TrustGraph API is running
+
+### MCP Tool Registration Errors
+
+If MCP tool registration fails:
+1. Check all required arguments are provided
+2. Verify the tool URL is accessible
+3. Ensure the MCP endpoint is operational
+4. Check API connectivity
+5. Review error messages for specific issues
+
+### MCP Tool Connectivity Issues
+
+If MCP tools aren't working as expected:
+1. Verify MCP endpoint is accessible from TrustGraph
+2. Check MCP server logs for errors
+3. Ensure MCP protocol compatibility
+4. Review network connectivity and firewall rules
+5. Test MCP endpoint directly
+
+## MCP Protocol
+
+The Model Control Protocol (MCP) is a standardized interface for AI model tools:
+
+- **Standardized API**: Consistent interface across different tools
+- **Extensible**: Support for complex tool interactions
+- **Stateful**: Can maintain state across multiple interactions
+- **Secure**: Built-in security and authentication mechanisms
+
+## Security Considerations
+
+When registering MCP tools:
+
+1. **URL Validation**: Ensure URLs are legitimate and secure
+2. **Network Security**: Use HTTPS when possible
+3. **Access Control**: Implement proper authentication for MCP endpoints
+4. **Input Validation**: Validate all inputs to MCP tools
+5. **Error Handling**: Don't expose sensitive information in error messages
+
+## Related Commands
+
+- [`tg-show-mcp-tools`](tg-show-mcp-tools.md) - Display registered MCP tools
+- [`tg-delete-mcp-tool`](tg-delete-mcp-tool.md) - Remove MCP tool configurations
+- [`tg-invoke-mcp-tool`](tg-invoke-mcp-tool.md) - Execute MCP tools
+- [`tg-set-tool`](tg-set-tool.md) - Configure regular agent tools
+
+## See Also
+
+- MCP Protocol Documentation
+- TrustGraph MCP Integration Guide
+- Agent Tool Configuration Guide

docs/cli/tg-set-tool.md

@@ -0,0 +1,321 @@
+# tg-set-tool
+
+## Synopsis
+
+```
+tg-set-tool [OPTIONS] --id ID --name NAME --type TYPE --description DESCRIPTION [--argument ARG...]
+```
+
+## Description
+
+The `tg-set-tool` command configures and registers tools in the TrustGraph system. It allows defining tool metadata including ID, name, description, type, and argument specifications. Tools are stored in the agent configuration and indexed for discovery and execution.
+
+This command is useful for:
+- Registering new tools for agent use
+- Updating existing tool configurations
+- Defining tool arguments and parameter types
+- Managing the tool registry for agent workflows
+
+The command updates both the tool index and stores the complete tool configuration in the TrustGraph API.
+
+## Options
+
+- `-u, --api-url URL`
+  - TrustGraph API URL for configuration storage
+  - Default: `http://localhost:8088/` (or `TRUSTGRAPH_URL` environment variable)
+  - Should point to a running TrustGraph API instance
+
+- `--id ID`
+  - **Required.** Unique identifier for the tool
+  - Used to reference the tool in configurations and agent workflows
+  - Must be unique within the tool registry
+
+- `--name NAME`
+  - **Required.** Human-readable name for the tool
+  - Displayed in tool listings and user interfaces
+  - Should be descriptive and clear
+
+- `--type TYPE`
+  - **Required.** Tool type defining its functionality
+  - Valid types:
+    - `knowledge-query` - Query knowledge bases
+    - `text-completion` - Text completion/generation
+    - `mcp-tool` - Model Control Protocol tool
+
+- `--description DESCRIPTION`
+  - **Required.** Detailed description of what the tool does
+  - Used by agents to understand tool capabilities
+  - Should clearly explain the tool's purpose and function
+
+- `--argument ARG`
+  - Tool argument specification in format: `name:type:description`
+  - Can be specified multiple times for multiple arguments
+  - Valid argument types:
+    - `string` - String/text parameter
+    - `number` - Numeric parameter
+
+- `-h, --help`
+  - Show help message and exit
+
+## Examples
+
+### Basic Tool Registration
+
+Register a simple weather lookup tool:
+```bash
+tg-set-tool --id weather --name "Weather Lookup" \
+            --type knowledge-query \
+            --description "Get current weather information" \
+            --argument location:string:"Location to query" \
+            --argument units:string:"Temperature units (C/F)"
+```
+
+### Calculator Tool
+
+Register a calculator tool with MCP type:
+```bash
+tg-set-tool --id calculator --name "Calculator" --type mcp-tool \
+            --description "Perform mathematical calculations" \
+            --argument expression:string:"Mathematical expression to evaluate"
+```
+
+### Text Completion Tool
+
+Register a text completion tool:
+```bash
+tg-set-tool --id text-generator --name "Text Generator" \
+            --type text-completion \
+            --description "Generate text based on prompts" \
+            --argument prompt:string:"Text prompt for generation" \
+            --argument max_tokens:number:"Maximum tokens to generate"
+```
+
+### Custom API URL
+
+Register a tool with custom API endpoint:
+```bash
+tg-set-tool -u http://trustgraph.example.com:8088/ \
+            --id custom-tool --name "Custom Tool" \
+            --type knowledge-query \
+            --description "Custom tool functionality"
+```
+
+### Tool Without Arguments
+
+Register a simple tool with no arguments:
+```bash
+tg-set-tool --id status-check --name "Status Check" \
+            --type knowledge-query \
+            --description "Check system status"
+```
+
+## Tool Types
+
+### knowledge-query
+Tools that query knowledge bases, databases, or information systems:
+- Used for information retrieval
+- Typically return structured data or search results
+- Examples: web search, document lookup, database queries
+
+### text-completion
+Tools that generate or complete text:
+- Used for text generation tasks
+- Process prompts and return generated content
+- Examples: language models, text generators, summarizers
+
+### mcp-tool
+Model Control Protocol tools:
+- Standardized tool interface for AI models
+- Support complex interactions and state management
+- Examples: external API integrations, complex workflows
+
+## Argument Types
+
+### string
+Text or string parameters:
+- Accept any text input
+- Used for queries, prompts, identifiers
+- Should include clear description of expected format
+
+### number
+Numeric parameters:
+- Accept integer or floating-point values
+- Used for limits, thresholds, quantities
+- Should specify valid ranges when applicable
+
+## Configuration Storage
+
+The tool configuration is stored in two parts:
+
+1. **Tool Index** (`agent.tool-index`)
+   - List of all registered tool IDs
+   - Updated to include new tools
+   - Used for tool discovery
+
+2. **Tool Configuration** (`agent.tool.{id}`)
+   - Complete tool definition as JSON
+   - Includes metadata and argument specifications
+   - Used for tool execution and validation
+
+## Advanced Usage
+
+### Updating Existing Tools
+
+Update an existing tool configuration:
+```bash
+# Update tool description
+tg-set-tool --id weather --name "Weather Lookup" \
+            --type knowledge-query \
+            --description "Updated weather information service" \
+            --argument location:string:"Location to query"
+```
+
+### Batch Tool Registration
+
+Register multiple tools in a script:
+```bash
+#!/bin/bash
+# Register a suite of tools
+tg-set-tool --id search --name "Web Search" --type knowledge-query \
+            --description "Search the web" \
+            --argument query:string:"Search query"
+
+tg-set-tool --id summarize --name "Text Summarizer" --type text-completion \
+            --description "Summarize text content" \
+            --argument text:string:"Text to summarize"
+
+tg-set-tool --id translate --name "Translator" --type mcp-tool \
+            --description "Translate text between languages" \
+            --argument text:string:"Text to translate" \
+            --argument target_lang:string:"Target language"
+```
+
+### Tool Validation
+
+Verify tool registration:
+```bash
+# Register tool and verify
+tg-set-tool --id test-tool --name "Test Tool" \
+            --type knowledge-query \
+            --description "Test tool for validation"
+
+# Check if tool was registered
+tg-show-tools | grep test-tool
+```
+
+## Error Handling
+
+The command handles various error conditions:
+
+- **Missing required arguments**: All required fields must be provided
+- **Invalid tool types**: Only valid types are accepted
+- **Invalid argument format**: Arguments must follow `name:type:description` format
+- **API connection errors**: If the TrustGraph API is unavailable
+- **Configuration errors**: If tool data cannot be stored
+
+Common error scenarios:
+```bash
+# Missing required field
+tg-set-tool --id tool1 --name "Tool 1"
+# Output: Exception: Must specify --type for tool
+
+# Invalid tool type
+tg-set-tool --id tool1 --name "Tool 1" --type invalid-type
+# Output: Exception: Type must be one of: knowledge-query, text-completion, mcp-tool
+
+# Invalid argument format
+tg-set-tool --id tool1 --name "Tool 1" --type knowledge-query \
+            --argument "bad-format"
+# Output: Exception: Arguments should be form name:type:description
+```
+
+## Integration with Other Commands
+
+### With Tool Management
+
+View registered tools:
+```bash
+# Register tool
+tg-set-tool --id new-tool --name "New Tool" \
+            --type knowledge-query \
+            --description "Newly registered tool"
+
+# View all tools
+tg-show-tools
+```
+
+### With Agent Invocation
+
+Use registered tools with agents:
+```bash
+# Register tool
+tg-set-tool --id weather --name "Weather" \
+            --type knowledge-query \
+            --description "Weather lookup"
+
+# Use tool in agent workflow
+tg-invoke-agent --prompt "What's the weather in London?"
+```
+
+### With Flow Configuration
+
+Tools can be used in flow configurations:
+```bash
+# Register tool for flow use
+tg-set-tool --id data-processor --name "Data Processor" \
+            --type mcp-tool \
+            --description "Process data in flows"
+
+# View flows that might use the tool
+tg-show-flows
+```
+
+## Best Practices
+
+1. **Clear Naming**: Use descriptive, unique tool IDs and names
+2. **Detailed Descriptions**: Provide comprehensive tool descriptions
+3. **Argument Documentation**: Clearly describe each argument's purpose
+4. **Type Selection**: Choose appropriate tool types for functionality
+5. **Validation**: Test tools after registration
+6. **Version Management**: Track tool configuration changes
+7. **Documentation**: Document custom tools and their usage
+
+## Troubleshooting
+
+### Tool Not Appearing
+
+If a registered tool doesn't appear in listings:
+1. Verify the tool was registered successfully
+2. Check the tool index with `tg-show-tools`
+3. Ensure the API URL is correct
+4. Verify TrustGraph API is running
+
+### Tool Registration Errors
+
+If tool registration fails:
+1. Check all required arguments are provided
+2. Verify argument format is correct
+3. Ensure tool type is valid
+4. Check API connectivity
+5. Review error messages for specific issues
+
+### Tool Configuration Issues
+
+If tools aren't working as expected:
+1. Verify tool arguments are correctly specified
+2. Check tool type matches intended functionality
+3. Ensure tool implementation is available
+4. Review agent logs for tool execution errors
+
+## Related Commands
+
+- [`tg-show-tools`](tg-show-tools.md) - Display registered tools
+- [`tg-delete-tool`](tg-delete-tool.md) - Remove tool configurations
+- [`tg-set-mcp-tool`](tg-set-mcp-tool.md) - Configure MCP tools
+- [`tg-invoke-agent`](tg-invoke-agent.md) - Use tools with agents
+
+## See Also
+
+- TrustGraph Tool Development Guide
+- Agent Configuration Documentation
+- MCP Tool Integration Guide

docs/tech-specs/ARCHITECTURE_PRINCIPLES.md

@@ -0,0 +1,106 @@
+# Knowledge Graph Architecture Foundations
+
+## Foundation 1: Subject-Predicate-Object (SPO) Graph Model
+**Decision**: Adopt SPO/RDF as the core knowledge representation model
+
+**Rationale**:
+- Provides maximum flexibility and interoperability with existing graph technologies
+- Enables seamless translation to other graph query languages (e.g., SPO → Cypher, but not vice versa)
+- Creates a foundation that "unlocks a lot" of downstream capabilities
+- Supports both node-to-node relationships (SPO) and node-to-literal relationships (RDF)
+
+**Implementation**: 
+- Core data structure: `node → edge → {node | literal}`
+- Maintain compatibility with RDF standards while supporting extended SPO operations
+
+## Foundation 2: LLM-Native Knowledge Graph Integration
+**Decision**: Optimize knowledge graph structure and operations for LLM interaction
+
+**Rationale**:
+- Primary use case involves LLMs interfacing with knowledge graphs
+- Graph technology choices must prioritize LLM compatibility over other considerations
+- Enables natural language processing workflows that leverage structured knowledge
+
+**Implementation**:
+- Design graph schemas that LLMs can effectively reason about
+- Optimize for common LLM interaction patterns
+
+## Foundation 3: Embedding-Based Graph Navigation
+**Decision**: Implement direct mapping from natural language queries to graph nodes via embeddings
+
+**Rationale**:
+- Enables the simplest possible path from NLP query to graph navigation
+- Avoids complex intermediate query generation steps
+- Provides efficient semantic search capabilities within the graph structure
+
+**Implementation**:
+- `NLP Query → Graph Embeddings → Graph Nodes`
+- Maintain embedding representations for all graph entities
+- Support direct semantic similarity matching for query resolution
+
+## Foundation 4: Distributed Entity Resolution with Deterministic Identifiers
+**Decision**: Support parallel knowledge extraction with deterministic entity identification (80% rule)
+
+**Rationale**:
+- **Ideal**: Single-process extraction with complete state visibility enables perfect entity resolution
+- **Reality**: Scalability requirements demand parallel processing capabilities
+- **Compromise**: Design for deterministic entity identification across distributed processes
+
+**Implementation**:
+- Develop mechanisms for generating consistent, unique identifiers across different knowledge extractors
+- Same entity mentioned in different processes must resolve to the same identifier
+- Acknowledge that ~20% of edge cases may require alternative processing models
+- Design fallback mechanisms for complex entity resolution scenarios
+
+## Foundation 5: Event-Driven Architecture with Publish-Subscribe
+**Decision**: Implement pub-sub messaging system for system coordination
+
+**Rationale**:
+- Enables loose coupling between knowledge extraction, storage, and query components
+- Supports real-time updates and notifications across the system
+- Facilitates scalable, distributed processing workflows
+
+**Implementation**:
+- Message-driven coordination between system components
+- Event streams for knowledge updates, extraction completion, and query results
+
+## Foundation 6: Reentrant Agent Communication
+**Decision**: Support reentrant pub-sub operations for agent-based processing
+
+**Rationale**:
+- Enables sophisticated agent workflows where agents can trigger and respond to each other
+- Supports complex, multi-step knowledge processing pipelines
+- Allows for recursive and iterative processing patterns
+
+**Implementation**:
+- Pub-sub system must handle reentrant calls safely
+- Agent coordination mechanisms that prevent infinite loops
+- Support for agent workflow orchestration
+
+## Foundation 7: Columnar Data Store Integration
+**Decision**: Ensure query compatibility with columnar storage systems
+
+**Rationale**:
+- Enables efficient analytical queries over large knowledge datasets
+- Supports business intelligence and reporting use cases
+- Bridges graph-based knowledge representation with traditional analytical workflows
+
+**Implementation**:
+- Query translation layer: Graph queries → Columnar queries
+- Hybrid storage strategy supporting both graph operations and analytical workloads
+- Maintain query performance across both paradigms
+
+---
+
+## Architecture Principles Summary
+
+1. **Flexibility First**: SPO/RDF model provides maximum adaptability
+2. **LLM Optimization**: All design decisions consider LLM interaction requirements
+3. **Semantic Efficiency**: Direct embedding-to-node mapping for optimal query performance
+4. **Pragmatic Scalability**: Balance perfect accuracy with practical distributed processing
+5. **Event-Driven Coordination**: Pub-sub enables loose coupling and scalability
+6. **Agent-Friendly**: Support complex, multi-agent processing workflows
+7. **Analytical Compatibility**: Bridge graph and columnar paradigms for comprehensive querying
+
+These foundations establish a knowledge graph architecture that balances theoretical rigor with practical scalability requirements, optimized for LLM integration and distributed processing.
+

docs/tech-specs/LOGGING_STRATEGY.md

@@ -0,0 +1,169 @@
+# TrustGraph Logging Strategy
+
+## Overview
+
+TrustGraph uses Python's built-in `logging` module for all logging operations. This provides a standardized, flexible approach to logging across all components of the system.
+
+## Default Configuration
+
+### Logging Level
+- **Default Level**: `INFO`
+- **Debug Mode**: `DEBUG` (enabled via command-line argument)
+- **Production**: `WARNING` or `ERROR` as appropriate
+
+### Output Destination
+All logs should be written to **standard output (stdout)** to ensure compatibility with containerized environments and log aggregation systems.
+
+## Implementation Guidelines
+
+### 1. Logger Initialization
+
+Each module should create its own logger using the module's `__name__`:
+
+```python
+import logging
+
+logger = logging.getLogger(__name__)
+```
+
+### 2. Centralized Configuration
+
+The logging configuration should be centralized in `async_processor.py` (or a dedicated logging configuration module) since it's inherited by much of the codebase:
+
+```python
+import logging
+import argparse
+
+def setup_logging(log_level='INFO'):
+    """Configure logging for the entire application"""
+    logging.basicConfig(
+        level=getattr(logging, log_level.upper()),
+        format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
+        handlers=[logging.StreamHandler()]
+    )
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        '--log-level',
+        default='INFO',
+        choices=['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'],
+        help='Set the logging level (default: INFO)'
+    )
+    return parser.parse_args()
+
+# In main execution
+if __name__ == '__main__':
+    args = parse_args()
+    setup_logging(args.log_level)
+```
+
+### 3. Logging Best Practices
+
+#### Log Levels Usage
+- **DEBUG**: Detailed information for diagnosing problems (variable values, function entry/exit)
+- **INFO**: General informational messages (service started, configuration loaded, processing milestones)
+- **WARNING**: Warning messages for potentially harmful situations (deprecated features, recoverable errors)
+- **ERROR**: Error messages for serious problems (failed operations, exceptions)
+- **CRITICAL**: Critical messages for system failures requiring immediate attention
+
+#### Message Format
+```python
+# Good - includes context
+logger.info(f"Processing document: {doc_id}, size: {doc_size} bytes")
+logger.error(f"Failed to connect to database: {error}", exc_info=True)
+
+# Avoid - lacks context
+logger.info("Processing document")
+logger.error("Connection failed")
+```
+
+#### Performance Considerations
+```python
+# Use lazy formatting for expensive operations
+logger.debug("Expensive operation result: %s", expensive_function())
+
+# Check log level for very expensive debug operations
+if logger.isEnabledFor(logging.DEBUG):
+    debug_data = compute_expensive_debug_info()
+    logger.debug(f"Debug data: {debug_data}")
+```
+
+### 4. Structured Logging
+
+For complex data, use structured logging:
+
+```python
+logger.info("Request processed", extra={
+    'request_id': request_id,
+    'duration_ms': duration,
+    'status_code': status_code,
+    'user_id': user_id
+})
+```
+
+### 5. Exception Logging
+
+Always include stack traces for exceptions:
+
+```python
+try:
+    process_data()
+except Exception as e:
+    logger.error(f"Failed to process data: {e}", exc_info=True)
+    raise
+```
+
+### 6. Async Logging Considerations
+
+For async code, ensure thread-safe logging:
+
+```python
+import asyncio
+import logging
+
+async def async_operation():
+    logger = logging.getLogger(__name__)
+    logger.info(f"Starting async operation in task: {asyncio.current_task().get_name()}")
+```
+
+## Environment Variables
+
+Support environment-based configuration as a fallback:
+
+```python
+import os
+
+log_level = os.environ.get('TRUSTGRAPH_LOG_LEVEL', 'INFO')
+```
+
+## Testing
+
+During tests, consider using a different logging configuration:
+
+```python
+# In test setup
+logging.getLogger().setLevel(logging.WARNING)  # Reduce noise during tests
+```
+
+## Monitoring Integration
+
+Ensure log format is compatible with monitoring tools:
+- Include timestamps in ISO format
+- Use consistent field names
+- Include correlation IDs where applicable
+- Structure logs for easy parsing (JSON format for production)
+
+## Security Considerations
+
+- Never log sensitive information (passwords, API keys, personal data)
+- Sanitize user input before logging
+- Use placeholders for sensitive fields: `user_id=****1234`
+
+## Migration Path
+
+For existing code using print statements:
+1. Replace `print()` with appropriate logger calls
+2. Choose appropriate log levels based on message importance
+3. Add context to make logs more useful
+4. Test logging output at different levels

docs/tech-specs/SCHEMA_REFACTORING_PROPOSAL.md

@@ -0,0 +1,91 @@
+# Schema Directory Refactoring Proposal
+
+## Current Issues
+
+1. **Flat structure** - All schemas in one directory makes it hard to understand relationships
+2. **Mixed concerns** - Core types, domain objects, and API contracts all mixed together
+3. **Unclear naming** - Files like "object.py", "types.py", "topic.py" don't clearly indicate their purpose
+4. **No clear layering** - Can't easily see what depends on what
+
+## Proposed Structure
+
+```
+trustgraph-base/trustgraph/schema/
+├── __init__.py
+├── core/              # Core primitive types used everywhere
+│   ├── __init__.py
+│   ├── primitives.py  # Error, Value, Triple, Field, RowSchema
+│   ├── metadata.py    # Metadata record
+│   └── topic.py       # Topic utilities
+│
+├── knowledge/         # Knowledge domain models and extraction
+│   ├── __init__.py
+│   ├── graph.py       # EntityContext, EntityEmbeddings, Triples
+│   ├── document.py    # Document, TextDocument, Chunk
+│   ├── knowledge.py   # Knowledge extraction types
+│   ├── embeddings.py  # All embedding-related types (moved from multiple files)
+│   └── nlp.py         # Definition, Topic, Relationship, Fact types
+│
+└── services/          # Service request/response contracts
+    ├── __init__.py
+    ├── llm.py         # TextCompletion, Embeddings, Tool requests/responses
+    ├── retrieval.py   # GraphRAG, DocumentRAG queries/responses
+    ├── query.py       # GraphEmbeddingsRequest/Response, DocumentEmbeddingsRequest/Response
+    ├── agent.py       # Agent requests/responses
+    ├── flow.py        # Flow requests/responses
+    ├── prompt.py      # Prompt service requests/responses
+    ├── config.py      # Configuration service
+    ├── library.py     # Librarian service
+    └── lookup.py      # Lookup service
+```
+
+## Key Changes
+
+1. **Hierarchical organization** - Clear separation between core types, knowledge models, and service contracts
+2. **Better naming**:
+   - `types.py` → `core/primitives.py` (clearer purpose)
+   - `object.py` → Split between appropriate files based on actual content
+   - `documents.py` → `knowledge/document.py` (singular, consistent)
+   - `models.py` → `services/llm.py` (clearer what kind of models)
+   - `prompt.py` → Split: service parts to `services/prompt.py`, data types to `knowledge/nlp.py`
+
+3. **Logical grouping**:
+   - All embedding types consolidated in `knowledge/embeddings.py`
+   - All LLM-related service contracts in `services/llm.py`
+   - Clear separation of request/response pairs in services directory
+   - Knowledge extraction types grouped with other knowledge domain models
+
+4. **Dependency clarity**:
+   - Core types have no dependencies
+   - Knowledge models depend only on core
+   - Service contracts can depend on both core and knowledge models
+
+## Migration Benefits
+
+1. **Easier navigation** - Developers can quickly find what they need
+2. **Better modularity** - Clear boundaries between different concerns
+3. **Simpler imports** - More intuitive import paths
+4. **Future-proof** - Easy to add new knowledge types or services without cluttering
+
+## Example Import Changes
+
+```python
+# Before
+from trustgraph.schema import Error, Triple, GraphEmbeddings, TextCompletionRequest
+
+# After
+from trustgraph.schema.core import Error, Triple
+from trustgraph.schema.knowledge import GraphEmbeddings
+from trustgraph.schema.services import TextCompletionRequest
+```
+
+## Implementation Notes
+
+1. Keep backward compatibility by maintaining imports in root `__init__.py`
+2. Move files gradually, updating imports as needed
+3. Consider adding a `legacy.py` that imports everything for transition period
+4. Update documentation to reflect new structure
+
+<function_calls>
+<invoke name="TodoWrite">
+<parameter name="todos">[{"id": "1", "content": "Examine current schema directory structure", "status": "completed", "priority": "high"}, {"id": "2", "content": "Analyze schema files and their purposes", "status": "completed", "priority": "high"}, {"id": "3", "content": "Propose improved naming and structure", "status": "completed", "priority": "high"}]

docs/tech-specs/STRUCTURED_DATA.md

@@ -0,0 +1,253 @@
+# Structured Data Technical Specification
+
+## Overview
+
+This specification describes the integration of TrustGraph with structured data flows, enabling the system to work with data that can be represented as rows in tables or objects in object stores. The integration supports four primary use cases:
+
+1. **Unstructured to Structured Extraction**: Read unstructured data sources, identify and extract object structures, and store them in a tabular format
+2. **Structured Data Ingestion**: Load data that is already in structured formats directly into the structured store alongside extracted data
+3. **Natural Language Querying**: Convert natural language questions into structured queries to extract matching data from the store
+4. **Direct Structured Querying**: Execute structured queries directly against the data store for precise data retrieval
+
+## Goals
+
+- **Unified Data Access**: Provide a single interface for accessing both structured and unstructured data within TrustGraph
+- **Seamless Integration**: Enable smooth interoperability between TrustGraph's graph-based knowledge representation and traditional structured data formats
+- **Flexible Extraction**: Support automatic extraction of structured data from various unstructured sources (documents, text, etc.)
+- **Query Versatility**: Allow users to query data using both natural language and structured query languages
+- **Data Consistency**: Maintain data integrity and consistency across different data representations
+- **Performance Optimization**: Ensure efficient storage and retrieval of structured data at scale
+- **Schema Flexibility**: Support both schema-on-write and schema-on-read approaches to accommodate diverse data sources
+- **Backwards Compatibility**: Preserve existing TrustGraph functionality while adding structured data capabilities
+
+## Background
+
+TrustGraph currently excels at processing unstructured data and building knowledge graphs from diverse sources. However, many enterprise use cases involve data that is inherently structured - customer records, transaction logs, inventory databases, and other tabular datasets. These structured datasets often need to be analyzed alongside unstructured content to provide comprehensive insights.
+
+Current limitations include:
+- No native support for ingesting pre-structured data formats (CSV, JSON arrays, database exports)
+- Inability to preserve the inherent structure when extracting tabular data from documents
+- Lack of efficient querying mechanisms for structured data patterns
+- Missing bridge between SQL-like queries and TrustGraph's graph queries
+
+This specification addresses these gaps by introducing a structured data layer that complements TrustGraph's existing capabilities. By supporting structured data natively, TrustGraph can:
+- Serve as a unified platform for both structured and unstructured data analysis
+- Enable hybrid queries that span both graph relationships and tabular data
+- Provide familiar interfaces for users accustomed to working with structured data
+- Unlock new use cases in data integration and business intelligence
+
+## Technical Design
+
+### Architecture
+
+The structured data integration requires the following technical components:
+
+1. **NLP-to-Structured-Query Service**
+   - Converts natural language questions into structured queries
+   - Supports multiple query language targets (initially SQL-like syntax)
+   - Integrates with existing TrustGraph NLP capabilities
+   
+   Module: trustgraph-flow/trustgraph/query/nlp_query/cassandra
+
+2. **Configuration Schema Support** ✅ **[COMPLETE]**
+   - Extended configuration system to store structured data schemas
+   - Support for defining table structures, field types, and relationships
+   - Schema versioning and migration capabilities
+
+3. **Object Extraction Module** ✅ **[COMPLETE]**
+   - Enhanced knowledge extractor flow integration
+   - Identifies and extracts structured objects from unstructured sources
+   - Maintains provenance and confidence scores
+   - Registers a config handler (example: trustgraph-flow/trustgraph/prompt/template/service.py) to receive config data and decode schema information
+   - Receives objects and decodes them to ExtractedObject objects for delivery on the Pulsar queue
+   - NOTE: There's existing code at `trustgraph-flow/trustgraph/extract/object/row/`. This was a previous attempt and will need to be majorly refactored as it doesn't conform to current APIs. Use it if it's useful, start from scratch if not.
+   - Requires a command-line interface: `kg-extract-objects`
+
+   Module: trustgraph-flow/trustgraph/extract/kg/objects/
+
+4. **Structured Store Writer Module** ✅ **[COMPLETE]**
+   - Receives objects in ExtractedObject format from Pulsar queues
+   - Initial implementation targeting Apache Cassandra as the structured data store
+   - Handles dynamic table creation based on schemas encountered
+   - Manages schema-to-Cassandra table mapping and data transformation
+   - Provides batch and streaming write operations for performance optimization
+   - No Pulsar outputs - this is a terminal service in the data flow
+
+   **Schema Handling**:
+   - Monitors incoming ExtractedObject messages for schema references
+   - When a new schema is encountered for the first time, automatically creates the corresponding Cassandra table
+   - Maintains a cache of known schemas to avoid redundant table creation attempts
+   - Should consider whether to receive schema definitions directly or rely on schema names in ExtractedObject messages
+
+   **Cassandra Table Mapping**:
+   - Keyspace is named after the `user` field from ExtractedObject's Metadata
+   - Table is named after the `schema_name` field from ExtractedObject
+   - Collection from Metadata becomes part of the partition key to ensure:
+     - Natural data distribution across Cassandra nodes
+     - Efficient queries within a specific collection
+     - Logical isolation between different data imports/sources
+   - Primary key structure: `PRIMARY KEY ((collection, <schema_primary_key_fields>), <clustering_keys>)`
+     - Collection is always the first component of the partition key
+     - Schema-defined primary key fields follow as part of the composite partition key
+     - This requires queries to specify the collection, ensuring predictable performance
+   - Field definitions map to Cassandra columns with type conversions:
+     - `string` → `text`
+     - `integer` → `int` or `bigint` based on size hint
+     - `float` → `float` or `double` based on precision needs
+     - `boolean` → `boolean`
+     - `timestamp` → `timestamp`
+     - `enum` → `text` with application-level validation
+   - Indexed fields create Cassandra secondary indexes (excluding fields already in the primary key)
+   - Required fields are enforced at the application level (Cassandra doesn't support NOT NULL)
+
+   **Object Storage**:
+   - Extracts values from ExtractedObject.values map
+   - Performs type conversion and validation before insertion
+   - Handles missing optional fields gracefully
+   - Maintains metadata about object provenance (source document, confidence scores)
+   - Supports idempotent writes to handle message replay scenarios
+
+   **Implementation Notes**:
+   - Existing code at `trustgraph-flow/trustgraph/storage/objects/cassandra/` is outdated and doesn't comply with current APIs
+   - Should reference `trustgraph-flow/trustgraph/storage/triples/cassandra` as an example of a working storage processor
+   - Needs evaluation of existing code for any reusable components before deciding to refactor or rewrite
+
+   Module: trustgraph-flow/trustgraph/storage/objects/cassandra
+
+5. **Structured Query Service**
+   - Accepts structured queries in defined formats
+   - Executes queries against the structured store
+   - Returns objects matching query criteria
+   - Supports pagination and result filtering
+
+   Module: trustgraph-flow/trustgraph/query/objects/cassandra
+
+6. **Agent Tool Integration**
+   - New tool class for agent frameworks
+   - Enables agents to query structured data stores
+   - Provides natural language and structured query interfaces
+   - Integrates with existing agent decision-making processes
+
+7. **Structured Data Ingestion Service**
+   - Accepts structured data in multiple formats (JSON, CSV, XML)
+   - Parses and validates incoming data against defined schemas
+   - Converts data into normalized object streams
+   - Emits objects to appropriate message queues for processing
+   - Supports bulk uploads and streaming ingestion
+
+   Module: trustgraph-flow/trustgraph/decoding/structured
+
+8. **Object Embedding Service**
+   - Generates vector embeddings for structured objects
+   - Enables semantic search across structured data
+   - Supports hybrid search combining structured queries with semantic similarity
+   - Integrates with existing vector stores
+
+   Module: trustgraph-flow/trustgraph/embeddings/object_embeddings/qdrant
+
+### Data Models
+
+#### Schema Storage Mechanism
+
+Schemas are stored in TrustGraph's configuration system using the following structure:
+
+- **Type**: `schema` (fixed value for all structured data schemas)
+- **Key**: The unique name/identifier of the schema (e.g., `customer_records`, `transaction_log`)
+- **Value**: JSON schema definition containing the structure
+
+Example configuration entry:
+```
+Type: schema
+Key: customer_records
+Value: {
+  "name": "customer_records",
+  "description": "Customer information table",
+  "fields": [
+    {
+      "name": "customer_id",
+      "type": "string",
+      "primary_key": true
+    },
+    {
+      "name": "name",
+      "type": "string",
+      "required": true
+    },
+    {
+      "name": "email",
+      "type": "string",
+      "required": true
+    },
+    {
+      "name": "registration_date",
+      "type": "timestamp"
+    },
+    {
+      "name": "status",
+      "type": "string",
+      "enum": ["active", "inactive", "suspended"]
+    }
+  ],
+  "indexes": ["email", "registration_date"]
+}
+```
+
+This approach allows:
+- Dynamic schema definition without code changes
+- Easy schema updates and versioning
+- Consistent integration with existing TrustGraph configuration management
+- Support for multiple schemas within a single deployment
+
+### APIs
+
+New APIs:
+  - Pulsar schemas for above types
+  - Pulsar interfaces in new flows
+  - Need a means to specify schema types in flows so that flows know which
+    schema types to load
+  - APIs added to gateway and rev-gateway
+
+Modified APIs:
+- Knowledge extraction endpoints - Add structured object output option
+- Agent endpoints - Add structured data tool support
+
+### Implementation Details
+
+Following existing conventions - these are just new processing modules.
+Everything is in the trustgraph-flow packages except for schema items
+in trustgraph-base.
+
+Need some UI work in the Workbench to be able to demo / pilot this
+capability.
+
+## Security Considerations
+
+No extra considerations.
+
+## Performance Considerations
+
+Some questions around using Cassandra queries and indexes so that queries
+don't slow down.
+
+## Testing Strategy
+
+Use existing test strategy, will build unit, contract and integration tests.
+
+## Migration Plan
+
+None.
+
+## Timeline
+
+Not specified.
+
+## Open Questions
+
+- Can this be made to work with other store types?  We're aiming to use
+  interfaces which make modules which work with one store applicable to
+  other stores.
+
+## References
+
+n/a.
+

docs/tech-specs/STRUCTURED_DATA_SCHEMAS.md

@@ -0,0 +1,139 @@
+# Structured Data Pulsar Schema Changes
+
+## Overview
+
+Based on the STRUCTURED_DATA.md specification, this document proposes the necessary Pulsar schema additions and modifications to support structured data capabilities in TrustGraph.
+
+## Required Schema Changes
+
+### 1. Core Schema Enhancements
+
+#### Enhanced Field Definition
+The existing `Field` class in `core/primitives.py` needs additional properties:
+
+```python
+class Field(Record):
+    name = String()
+    type = String()  # int, string, long, bool, float, double, timestamp
+    size = Integer()
+    primary = Boolean()
+    description = String()
+    # NEW FIELDS:
+    required = Boolean()  # Whether field is required
+    enum_values = Array(String())  # For enum type fields
+    indexed = Boolean()  # Whether field should be indexed
+```
+
+### 2. New Knowledge Schemas
+
+#### 2.1 Structured Data Submission
+New file: `knowledge/structured.py`
+
+```python
+from pulsar.schema import Record, String, Bytes, Map
+from ..core.metadata import Metadata
+
+class StructuredDataSubmission(Record):
+    metadata = Metadata()
+    format = String()  # "json", "csv", "xml"
+    schema_name = String()  # Reference to schema in config
+    data = Bytes()  # Raw data to ingest
+    options = Map(String())  # Format-specific options
+```
+
+### 3. New Service Schemas
+
+#### 3.1 NLP to Structured Query Service
+New file: `services/nlp_query.py`
+
+```python
+from pulsar.schema import Record, String, Array, Map, Integer, Double
+from ..core.primitives import Error
+
+class NLPToStructuredQueryRequest(Record):
+    natural_language_query = String()
+    max_results = Integer()
+    context_hints = Map(String())  # Optional context for query generation
+
+class NLPToStructuredQueryResponse(Record):
+    error = Error()
+    graphql_query = String()  # Generated GraphQL query
+    variables = Map(String())  # GraphQL variables if any
+    detected_schemas = Array(String())  # Which schemas the query targets
+    confidence = Double()
+```
+
+#### 3.2 Structured Query Service
+New file: `services/structured_query.py`
+
+```python
+from pulsar.schema import Record, String, Map, Array
+from ..core.primitives import Error
+
+class StructuredQueryRequest(Record):
+    query = String()  # GraphQL query
+    variables = Map(String())  # GraphQL variables
+    operation_name = String()  # Optional operation name for multi-operation documents
+
+class StructuredQueryResponse(Record):
+    error = Error()
+    data = String()  # JSON-encoded GraphQL response data
+    errors = Array(String())  # GraphQL errors if any
+```
+
+#### 2.2 Object Extraction Output
+New file: `knowledge/object.py`
+
+```python
+from pulsar.schema import Record, String, Map, Double
+from ..core.metadata import Metadata
+
+class ExtractedObject(Record):
+    metadata = Metadata()
+    schema_name = String()  # Which schema this object belongs to
+    values = Map(String())  # Field name -> value
+    confidence = Double()
+    source_span = String()  # Text span where object was found
+```
+
+### 4. Enhanced Knowledge Schemas
+
+#### 4.1 Object Embeddings Enhancement
+Update `knowledge/embeddings.py` to support structured object embeddings better:
+
+```python
+class StructuredObjectEmbedding(Record):
+    metadata = Metadata()
+    vectors = Array(Array(Double()))
+    schema_name = String()
+    object_id = String()  # Primary key value
+    field_embeddings = Map(Array(Double()))  # Per-field embeddings
+```
+
+## Integration Points
+
+### Flow Integration
+
+The schemas will be used by new flow modules:
+- `trustgraph-flow/trustgraph/decoding/structured` - Uses StructuredDataSubmission
+- `trustgraph-flow/trustgraph/query/nlp_query/cassandra` - Uses NLP query schemas
+- `trustgraph-flow/trustgraph/query/objects/cassandra` - Uses structured query schemas
+- `trustgraph-flow/trustgraph/extract/object/row/` - Consumes Chunk, produces ExtractedObject
+- `trustgraph-flow/trustgraph/storage/objects/cassandra` - Uses Rows schema
+- `trustgraph-flow/trustgraph/embeddings/object_embeddings/qdrant` - Uses object embedding schemas
+
+## Implementation Notes
+
+1. **Schema Versioning**: Consider adding a `version` field to RowSchema for future migration support
+2. **Type System**: The `Field.type` should support all Cassandra native types
+3. **Batch Operations**: Most services should support both single and batch operations
+4. **Error Handling**: Consistent error reporting across all new services
+5. **Backwards Compatibility**: Existing schemas remain unchanged except for minor Field enhancements
+
+## Next Steps
+
+1. Implement schema files in the new structure
+2. Update existing services to recognize new schema types
+3. Implement flow modules that use these schemas
+4. Add gateway/rev-gateway endpoints for new services
+5. Create unit tests for schema validation

grafana/provisioning/dashboard.yml

@@ -1,17 +0,0 @@
-
-apiVersion: 1
-
-providers:
-
-  - name: 'trustgraph.ai'
-    orgId: 1
-    folder: 'TrustGraph'
-    folderUid: 'b6c5be90-d432-4df8-aeab-737c7b151228'
-    type: file
-    disableDeletion: false
-    updateIntervalSeconds: 30
-    allowUiUpdates: true
-    options:
-      path: /var/lib/grafana/dashboards
-      foldersFromFilesStructure: false
-

grafana/provisioning/datasource.yml

@@ -1,21 +0,0 @@
-apiVersion: 1
-
-prune: true
-
-datasources:
-  - name: Prometheus
-    type: prometheus
-    access: proxy
-    orgId: 1
-    # <string> Sets a custom UID to reference this
-    # data source in other parts of the configuration.
-    # If not specified, Grafana generates one.
-    uid: 'f6b18033-5918-4e05-a1ca-4cb30343b129'
-
-    url: http://prometheus:9090
-
-    basicAuth: false
-    withCredentials: false
-    isDefault: true
-    editable: true
-

install_packages.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+# Install TrustGraph packages for testing
+
+echo "Installing TrustGraph packages..."
+
+# Install base package first (required by others)
+cd trustgraph-base
+pip install -e .
+cd ..
+
+# Install base package first (required by others)
+cd trustgraph-cli
+pip install -e .
+cd ..
+
+# Install vertexai package (depends on base)
+cd trustgraph-vertexai  
+pip install -e .
+cd ..
+
+# Install flow package (for additional components)
+cd trustgraph-flow
+pip install -e .
+cd ..
+
+echo "Package installation complete!"
+echo "Verify installation:"
+#python -c "import trustgraph.model.text_completion.vertexai.llm; print('VertexAI import successful')"

prometheus/prometheus.yml

@@ -1,187 +0,0 @@
-global:
-
-  scrape_interval:     15s # By default, scrape targets every 15 seconds.
-
-  # Attach these labels to any time series or alerts when communicating with
-  # external systems (federation, remote storage, Alertmanager).
-  external_labels:
-    monitor: 'trustgraph'
-
-# A scrape configuration containing exactly one endpoint to scrape:
-# Here it's Prometheus itself.
-scrape_configs:
-
-  # The job name is added as a label `job=<job_name>` to any timeseries
-  # scraped from this config.
-
-  - job_name: 'pulsar'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'pulsar:8080'
-
-  - job_name: 'bookie'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'bookie:8000'
-
-  - job_name: 'zookeeper'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'zookeeper:8000'
-
-  - job_name: 'pdf-decoder'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'pdf-decoder:8000'
-
-  - job_name: 'chunker'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'chunker:8000'
-
-  - job_name: 'document-embeddings'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'document-embeddings:8000'
-
-  - job_name: 'graph-embeddings'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'graph-embeddings:8000'
-
-  - job_name: 'embeddings'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'embeddings:8000'
-
-  - job_name: 'kg-extract-definitions'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'kg-extract-definitions:8000'
-
-  - job_name: 'kg-extract-topics'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'kg-extract-topics:8000'
-
-  - job_name: 'kg-extract-relationships'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'kg-extract-relationships:8000'
-
-  - job_name: 'metering'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'metering:8000'
-
-  - job_name: 'metering-rag'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'metering-rag:8000'
-
-  - job_name: 'store-doc-embeddings'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'store-doc-embeddings:8000'
-
-  - job_name: 'store-graph-embeddings'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'store-graph-embeddings:8000'
-
-  - job_name: 'store-triples'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'store-triples:8000'
-
-  - job_name: 'text-completion'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'text-completion:8000'
-
-  - job_name: 'text-completion-rag'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'text-completion-rag:8000'
-
-  - job_name: 'graph-rag'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'graph-rag:8000'
-
-  - job_name: 'document-rag'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'document-rag:8000'
-
-  - job_name: 'prompt'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'prompt:8000'
-
-  - job_name: 'prompt-rag'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'prompt-rag:8000'
-
-  - job_name: 'query-graph-embeddings'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'query-graph-embeddings:8000'
-
-  - job_name: 'query-doc-embeddings'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'query-doc-embeddings:8000'
-
-  - job_name: 'query-triples'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'query-triples:8000'
-
-  - job_name: 'agent-manager'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'agent-manager:8000'
-
-  - job_name: 'api-gateway'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'api-gateway:8000'
-
-  - job_name: 'workbench-ui'
-    scrape_interval: 5s
-    static_configs:
-      - targets:
-        - 'workbench-ui:8000'
-
-# Cassandra
-# qdrant
-

run_tests.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+# Test runner script for TrustGraph
+
+echo "TrustGraph Test Runner"
+echo "===================="
+
+# Check if we're in the right directory
+if [ ! -f "install_packages.sh" ]; then
+    echo "❌ Error: Please run this script from the project root directory"
+    echo "   Expected files: install_packages.sh, check_imports.py"
+    exit 1
+fi
+
+# Step 1: Check current imports
+echo "Step 1: Checking current imports..."
+python check_imports.py
+
+# Step 2: Install packages if needed
+echo ""
+echo "Step 2: Installing TrustGraph packages..."
+echo "This may take a moment..."
+./install_packages.sh
+
+# Step 3: Check imports again
+echo ""
+echo "Step 3: Verifying imports after installation..."
+python check_imports.py
+
+# Step 4: Install test dependencies
+echo ""
+echo "Step 4: Installing test dependencies..."
+cd tests/
+pip install -r requirements.txt
+cd ..
+
+# Step 5: Run the tests
+echo ""
+echo "Step 5: Running VertexAI tests..."
+echo "Command: pytest tests/unit/test_text_completion/test_vertexai_processor.py -v"
+echo ""
+
+# Set Python path just in case
+export PYTHONPATH=$PWD:$PYTHONPATH
+
+pytest tests/unit/test_text_completion/test_vertexai_processor.py -v
+
+echo ""
+echo "Test run complete!"

tests/__init__.py

@@ -0,0 +1,3 @@
+"""
+TrustGraph test suite
+"""

tests/contract/README.md

@@ -0,0 +1,243 @@
+# Contract Tests for TrustGraph
+
+This directory contains contract tests that verify service interface contracts, message schemas, and API compatibility across the TrustGraph microservices architecture.
+
+## Overview
+
+Contract tests ensure that:
+- **Message schemas remain compatible** across service versions
+- **API interfaces stay stable** for consumers
+- **Service communication contracts** are maintained
+- **Schema evolution** doesn't break existing integrations
+
+## Test Categories
+
+### 1. Pulsar Message Schema Contracts (`test_message_contracts.py`)
+
+Tests the contracts for all Pulsar message schemas used in TrustGraph service communication.
+
+#### **Coverage:**
+- ✅ **Text Completion Messages**: `TextCompletionRequest` ↔ `TextCompletionResponse`
+- ✅ **Document RAG Messages**: `DocumentRagQuery` ↔ `DocumentRagResponse`
+- ✅ **Agent Messages**: `AgentRequest` ↔ `AgentResponse` ↔ `AgentStep`
+- ✅ **Graph Messages**: `Chunk` → `Triple` → `Triples` → `EntityContext`
+- ✅ **Common Messages**: `Metadata`, `Value`, `Error` schemas
+- ✅ **Message Routing**: Properties, correlation IDs, routing keys
+- ✅ **Schema Evolution**: Backward/forward compatibility testing
+- ✅ **Serialization**: Schema validation and data integrity
+
+#### **Key Features:**
+- **Schema Validation**: Ensures all message schemas accept valid data and reject invalid data
+- **Field Contracts**: Validates required vs optional fields and type constraints
+- **Nested Schema Support**: Tests complex schemas with embedded objects and arrays
+- **Routing Contracts**: Validates message properties and routing conventions
+- **Evolution Testing**: Backward compatibility and schema versioning support
+
+## Running Contract Tests
+
+### Run All Contract Tests
+```bash
+pytest tests/contract/ -m contract
+```
+
+### Run Specific Contract Test Categories
+```bash
+# Message schema contracts
+pytest tests/contract/test_message_contracts.py -v
+
+# Specific test class
+pytest tests/contract/test_message_contracts.py::TestTextCompletionMessageContracts -v
+
+# Schema evolution tests
+pytest tests/contract/test_message_contracts.py::TestSchemaEvolutionContracts -v
+```
+
+### Run with Coverage
+```bash
+pytest tests/contract/ -m contract --cov=trustgraph.schema --cov-report=html
+```
+
+## Contract Test Patterns
+
+### 1. Schema Validation Pattern
+```python
+@pytest.mark.contract
+def test_schema_contract(self, sample_message_data):
+    """Test that schema accepts valid data and rejects invalid data"""
+    # Arrange
+    valid_data = sample_message_data["SchemaName"]
+    
+    # Act & Assert
+    assert validate_schema_contract(SchemaClass, valid_data)
+    
+    # Test field constraints
+    instance = SchemaClass(**valid_data)
+    assert hasattr(instance, 'required_field')
+    assert isinstance(instance.required_field, expected_type)
+```
+
+### 2. Serialization Contract Pattern
+```python
+@pytest.mark.contract  
+def test_serialization_contract(self, sample_message_data):
+    """Test schema serialization/deserialization contracts"""
+    # Arrange
+    data = sample_message_data["SchemaName"]
+    
+    # Act & Assert
+    assert serialize_deserialize_test(SchemaClass, data)
+```
+
+### 3. Evolution Contract Pattern
+```python
+@pytest.mark.contract
+def test_backward_compatibility_contract(self, schema_evolution_data):
+    """Test that new schema versions accept old data formats"""
+    # Arrange
+    old_version_data = schema_evolution_data["SchemaName_v1"]
+    
+    # Act - Should work with current schema
+    instance = CurrentSchema(**old_version_data)
+    
+    # Assert - Required fields maintained
+    assert instance.required_field == expected_value
+```
+
+## Schema Registry
+
+The contract tests maintain a registry of all TrustGraph schemas:
+
+```python
+schema_registry = {
+    # Text Completion
+    "TextCompletionRequest": TextCompletionRequest,
+    "TextCompletionResponse": TextCompletionResponse,
+    
+    # Document RAG  
+    "DocumentRagQuery": DocumentRagQuery,
+    "DocumentRagResponse": DocumentRagResponse,
+    
+    # Agent
+    "AgentRequest": AgentRequest,
+    "AgentResponse": AgentResponse,
+    
+    # Graph/Knowledge
+    "Chunk": Chunk,
+    "Triple": Triple,
+    "Triples": Triples,
+    "Value": Value,
+    
+    # Common
+    "Metadata": Metadata,
+    "Error": Error,
+}
+```
+
+## Message Contract Specifications
+
+### Text Completion Service Contract
+```yaml
+TextCompletionRequest:
+  required_fields: [system, prompt]
+  field_types:
+    system: string
+    prompt: string
+
+TextCompletionResponse:
+  required_fields: [error, response, model]  
+  field_types:
+    error: Error | null
+    response: string | null
+    in_token: integer | null
+    out_token: integer | null
+    model: string
+```
+
+### Document RAG Service Contract
+```yaml
+DocumentRagQuery:
+  required_fields: [query, user, collection]
+  field_types:
+    query: string
+    user: string
+    collection: string
+    doc_limit: integer
+
+DocumentRagResponse:
+  required_fields: [error, response]
+  field_types:
+    error: Error | null
+    response: string | null
+```
+
+### Agent Service Contract
+```yaml
+AgentRequest:
+  required_fields: [question, history]
+  field_types:
+    question: string
+    plan: string
+    state: string
+    history: Array<AgentStep>
+
+AgentResponse:
+  required_fields: [error]
+  field_types:
+    answer: string | null
+    error: Error | null
+    thought: string | null
+    observation: string | null
+```
+
+## Best Practices
+
+### Contract Test Design
+1. **Test Both Valid and Invalid Data**: Ensure schemas accept valid data and reject invalid data
+2. **Verify Field Constraints**: Test type constraints, required vs optional fields
+3. **Test Nested Schemas**: Validate complex objects with embedded schemas
+4. **Test Array Fields**: Ensure array serialization maintains order and content
+5. **Test Optional Fields**: Verify optional field handling in serialization
+
+### Schema Evolution
+1. **Backward Compatibility**: New schema versions must accept old message formats
+2. **Required Field Stability**: Required fields should never become optional or be removed
+3. **Additive Changes**: New fields should be optional to maintain compatibility
+4. **Deprecation Strategy**: Plan deprecation path for schema changes
+
+### Error Handling
+1. **Error Schema Consistency**: All error responses use consistent Error schema
+2. **Error Type Contracts**: Error types follow naming conventions
+3. **Error Message Format**: Error messages provide actionable information
+
+## Adding New Contract Tests
+
+When adding new message schemas or modifying existing ones:
+
+1. **Add to Schema Registry**: Update `conftest.py` schema registry
+2. **Add Sample Data**: Create valid sample data in `conftest.py`
+3. **Create Contract Tests**: Follow existing patterns for validation
+4. **Test Evolution**: Add backward compatibility tests
+5. **Update Documentation**: Document schema contracts in this README
+
+## Integration with CI/CD
+
+Contract tests should be run:
+- **On every commit** to detect breaking changes early
+- **Before releases** to ensure API stability
+- **On schema changes** to validate compatibility
+- **In dependency updates** to catch breaking changes
+
+```bash
+# CI/CD pipeline command
+pytest tests/contract/ -m contract --junitxml=contract-test-results.xml
+```
+
+## Contract Test Results
+
+Contract tests provide:
+- ✅ **Schema Compatibility Reports**: Which schemas pass/fail validation
+- ✅ **Breaking Change Detection**: Identifies contract violations
+- ✅ **Evolution Validation**: Confirms backward compatibility
+- ✅ **Field Constraint Verification**: Validates data type contracts
+
+This ensures that TrustGraph services can evolve independently while maintaining stable, compatible interfaces for all service communication.

tests/contract/conftest.py

@@ -0,0 +1,224 @@
+"""
+Contract test fixtures and configuration
+
+This file provides common fixtures for contract testing, focusing on
+message schema validation, API interface contracts, and service compatibility.
+"""
+
+import pytest
+import json
+from typing import Dict, Any, Type
+from pulsar.schema import Record
+from unittest.mock import MagicMock
+
+from trustgraph.schema import (
+    TextCompletionRequest, TextCompletionResponse,
+    DocumentRagQuery, DocumentRagResponse,
+    AgentRequest, AgentResponse, AgentStep,
+    Chunk, Triple, Triples, Value, Error,
+    EntityContext, EntityContexts,
+    GraphEmbeddings, EntityEmbeddings,
+    Metadata
+)
+
+
+@pytest.fixture
+def schema_registry():
+    """Registry of all Pulsar schemas used in TrustGraph"""
+    return {
+        # Text Completion
+        "TextCompletionRequest": TextCompletionRequest,
+        "TextCompletionResponse": TextCompletionResponse,
+        
+        # Document RAG
+        "DocumentRagQuery": DocumentRagQuery,
+        "DocumentRagResponse": DocumentRagResponse,
+        
+        # Agent
+        "AgentRequest": AgentRequest,
+        "AgentResponse": AgentResponse,
+        "AgentStep": AgentStep,
+        
+        # Graph
+        "Chunk": Chunk,
+        "Triple": Triple,
+        "Triples": Triples,
+        "Value": Value,
+        "Error": Error,
+        "EntityContext": EntityContext,
+        "EntityContexts": EntityContexts,
+        "GraphEmbeddings": GraphEmbeddings,
+        "EntityEmbeddings": EntityEmbeddings,
+        
+        # Common
+        "Metadata": Metadata,
+    }
+
+
+@pytest.fixture
+def sample_message_data():
+    """Sample message data for contract testing"""
+    return {
+        "TextCompletionRequest": {
+            "system": "You are a helpful assistant.",
+            "prompt": "What is machine learning?"
+        },
+        "TextCompletionResponse": {
+            "error": None,
+            "response": "Machine learning is a subset of artificial intelligence.",
+            "in_token": 50,
+            "out_token": 100,
+            "model": "gpt-3.5-turbo"
+        },
+        "DocumentRagQuery": {
+            "query": "What is artificial intelligence?",
+            "user": "test_user",
+            "collection": "test_collection",
+            "doc_limit": 10
+        },
+        "DocumentRagResponse": {
+            "error": None,
+            "response": "Artificial intelligence is the simulation of human intelligence in machines."
+        },
+        "AgentRequest": {
+            "question": "What is machine learning?",
+            "plan": "",
+            "state": "",
+            "history": []
+        },
+        "AgentResponse": {
+            "answer": "Machine learning is a subset of AI.",
+            "error": None,
+            "thought": "I need to provide information about machine learning.",
+            "observation": None
+        },
+        "Metadata": {
+            "id": "test-doc-123",
+            "user": "test_user",
+            "collection": "test_collection",
+            "metadata": []
+        },
+        "Value": {
+            "value": "http://example.com/entity",
+            "is_uri": True,
+            "type": ""
+        },
+        "Triple": {
+            "s": Value(
+                value="http://example.com/subject",
+                is_uri=True,
+                type=""
+            ),
+            "p": Value(
+                value="http://example.com/predicate", 
+                is_uri=True,
+                type=""
+            ),
+            "o": Value(
+                value="Object value",
+                is_uri=False,
+                type=""
+            )
+        }
+    }
+
+
+@pytest.fixture
+def invalid_message_data():
+    """Invalid message data for contract validation testing"""
+    return {
+        "TextCompletionRequest": [
+            {"system": None, "prompt": "test"},  # Invalid system (None)
+            {"system": "test", "prompt": None},  # Invalid prompt (None)
+            {"system": 123, "prompt": "test"},   # Invalid system (not string)
+            {},  # Missing required fields
+        ],
+        "DocumentRagQuery": [
+            {"query": None, "user": "test", "collection": "test", "doc_limit": 10},  # Invalid query
+            {"query": "test", "user": None, "collection": "test", "doc_limit": 10},  # Invalid user
+            {"query": "test", "user": "test", "collection": "test", "doc_limit": -1},  # Invalid doc_limit
+            {"query": "test"},  # Missing required fields
+        ],
+        "Value": [
+            {"value": None, "is_uri": True, "type": ""},  # Invalid value (None)
+            {"value": "test", "is_uri": "not_boolean", "type": ""},  # Invalid is_uri
+            {"value": 123, "is_uri": True, "type": ""},  # Invalid value (not string)
+        ]
+    }
+
+
+@pytest.fixture
+def message_properties():
+    """Standard message properties for contract testing"""
+    return {
+        "id": "test-message-123",
+        "routing_key": "test.routing.key",
+        "timestamp": "2024-01-01T00:00:00Z",
+        "source_service": "test-service",
+        "correlation_id": "correlation-123"
+    }
+
+
+@pytest.fixture
+def schema_evolution_data():
+    """Data for testing schema evolution and backward compatibility"""
+    return {
+        "TextCompletionRequest_v1": {
+            "system": "You are helpful.",
+            "prompt": "Test prompt"
+        },
+        "TextCompletionRequest_v2": {
+            "system": "You are helpful.",
+            "prompt": "Test prompt",
+            "temperature": 0.7,  # New field
+            "max_tokens": 100    # New field
+        },
+        "TextCompletionResponse_v1": {
+            "error": None,
+            "response": "Test response",
+            "model": "gpt-3.5-turbo"
+        },
+        "TextCompletionResponse_v2": {
+            "error": None,
+            "response": "Test response",
+            "in_token": 50,      # New field
+            "out_token": 100,    # New field
+            "model": "gpt-3.5-turbo"
+        }
+    }
+
+
+def validate_schema_contract(schema_class: Type[Record], data: Dict[str, Any]) -> bool:
+    """Helper function to validate schema contracts"""
+    try:
+        # Create instance from data
+        instance = schema_class(**data)
+        
+        # Verify all fields are accessible
+        for field_name in data.keys():
+            assert hasattr(instance, field_name)
+            assert getattr(instance, field_name) == data[field_name]
+        
+        return True
+    except Exception:
+        return False
+
+
+def serialize_deserialize_test(schema_class: Type[Record], data: Dict[str, Any]) -> bool:
+    """Helper function to test serialization/deserialization"""
+    try:
+        # Create instance
+        instance = schema_class(**data)
+        
+        # This would test actual Pulsar serialization if we had the client
+        # For now, we test the schema construction and field access
+        for field_name, field_value in data.items():
+            assert getattr(instance, field_name) == field_value
+        
+        return True
+    except Exception:
+        return False
+
+
+# Test markers for contract tests
+pytestmark = pytest.mark.contract

tests/contract/test_message_contracts.py

@@ -0,0 +1,614 @@
+"""
+Contract tests for Pulsar Message Schemas
+
+These tests verify the contracts for all Pulsar message schemas used in TrustGraph,
+ensuring schema compatibility, serialization contracts, and service interface stability.
+Following the TEST_STRATEGY.md approach for contract testing.
+"""
+
+import pytest
+import json
+from typing import Dict, Any, Type
+from pulsar.schema import Record
+
+from trustgraph.schema import (
+    TextCompletionRequest, TextCompletionResponse,
+    DocumentRagQuery, DocumentRagResponse,
+    AgentRequest, AgentResponse, AgentStep,
+    Chunk, Triple, Triples, Value, Error,
+    EntityContext, EntityContexts,
+    GraphEmbeddings, EntityEmbeddings,
+    Metadata, Field, RowSchema,
+    StructuredDataSubmission, ExtractedObject,
+    NLPToStructuredQueryRequest, NLPToStructuredQueryResponse,
+    StructuredQueryRequest, StructuredQueryResponse,
+    StructuredObjectEmbedding
+)
+from .conftest import validate_schema_contract, serialize_deserialize_test
+
+
+@pytest.mark.contract
+class TestTextCompletionMessageContracts:
+    """Contract tests for Text Completion message schemas"""
+
+    def test_text_completion_request_schema_contract(self, sample_message_data):
+        """Test TextCompletionRequest schema contract"""
+        # Arrange
+        request_data = sample_message_data["TextCompletionRequest"]
+
+        # Act & Assert
+        assert validate_schema_contract(TextCompletionRequest, request_data)
+        
+        # Test required fields
+        request = TextCompletionRequest(**request_data)
+        assert hasattr(request, 'system')
+        assert hasattr(request, 'prompt')
+        assert isinstance(request.system, str)
+        assert isinstance(request.prompt, str)
+
+    def test_text_completion_response_schema_contract(self, sample_message_data):
+        """Test TextCompletionResponse schema contract"""
+        # Arrange
+        response_data = sample_message_data["TextCompletionResponse"]
+
+        # Act & Assert
+        assert validate_schema_contract(TextCompletionResponse, response_data)
+        
+        # Test required fields
+        response = TextCompletionResponse(**response_data)
+        assert hasattr(response, 'error')
+        assert hasattr(response, 'response')
+        assert hasattr(response, 'in_token')
+        assert hasattr(response, 'out_token')
+        assert hasattr(response, 'model')
+
+    def test_text_completion_request_serialization_contract(self, sample_message_data):
+        """Test TextCompletionRequest serialization/deserialization contract"""
+        # Arrange
+        request_data = sample_message_data["TextCompletionRequest"]
+
+        # Act & Assert
+        assert serialize_deserialize_test(TextCompletionRequest, request_data)
+
+    def test_text_completion_response_serialization_contract(self, sample_message_data):
+        """Test TextCompletionResponse serialization/deserialization contract"""
+        # Arrange
+        response_data = sample_message_data["TextCompletionResponse"]
+
+        # Act & Assert
+        assert serialize_deserialize_test(TextCompletionResponse, response_data)
+
+    def test_text_completion_request_field_constraints(self):
+        """Test TextCompletionRequest field type constraints"""
+        # Test valid data
+        valid_request = TextCompletionRequest(
+            system="You are helpful.",
+            prompt="Test prompt"
+        )
+        assert valid_request.system == "You are helpful."
+        assert valid_request.prompt == "Test prompt"
+
+    def test_text_completion_response_field_constraints(self):
+        """Test TextCompletionResponse field type constraints"""
+        # Test valid response with no error
+        valid_response = TextCompletionResponse(
+            error=None,
+            response="Test response",
+            in_token=50,
+            out_token=100,
+            model="gpt-3.5-turbo"
+        )
+        assert valid_response.error is None
+        assert valid_response.response == "Test response"
+        assert valid_response.in_token == 50
+        assert valid_response.out_token == 100
+        assert valid_response.model == "gpt-3.5-turbo"
+
+        # Test response with error
+        error_response = TextCompletionResponse(
+            error=Error(type="rate-limit", message="Rate limit exceeded"),
+            response=None,
+            in_token=None,
+            out_token=None,
+            model=None
+        )
+        assert error_response.error is not None
+        assert error_response.error.type == "rate-limit"
+        assert error_response.response is None
+
+
+@pytest.mark.contract
+class TestDocumentRagMessageContracts:
+    """Contract tests for Document RAG message schemas"""
+
+    def test_document_rag_query_schema_contract(self, sample_message_data):
+        """Test DocumentRagQuery schema contract"""
+        # Arrange
+        query_data = sample_message_data["DocumentRagQuery"]
+
+        # Act & Assert
+        assert validate_schema_contract(DocumentRagQuery, query_data)
+        
+        # Test required fields
+        query = DocumentRagQuery(**query_data)
+        assert hasattr(query, 'query')
+        assert hasattr(query, 'user')
+        assert hasattr(query, 'collection')
+        assert hasattr(query, 'doc_limit')
+
+    def test_document_rag_response_schema_contract(self, sample_message_data):
+        """Test DocumentRagResponse schema contract"""
+        # Arrange
+        response_data = sample_message_data["DocumentRagResponse"]
+
+        # Act & Assert
+        assert validate_schema_contract(DocumentRagResponse, response_data)
+        
+        # Test required fields
+        response = DocumentRagResponse(**response_data)
+        assert hasattr(response, 'error')
+        assert hasattr(response, 'response')
+
+    def test_document_rag_query_field_constraints(self):
+        """Test DocumentRagQuery field constraints"""
+        # Test valid query
+        valid_query = DocumentRagQuery(
+            query="What is AI?",
+            user="test_user",
+            collection="test_collection",
+            doc_limit=5
+        )
+        assert valid_query.query == "What is AI?"
+        assert valid_query.user == "test_user"
+        assert valid_query.collection == "test_collection"
+        assert valid_query.doc_limit == 5
+
+    def test_document_rag_response_error_contract(self):
+        """Test DocumentRagResponse error handling contract"""
+        # Test successful response
+        success_response = DocumentRagResponse(
+            error=None,
+            response="AI is artificial intelligence."
+        )
+        assert success_response.error is None
+        assert success_response.response == "AI is artificial intelligence."
+
+        # Test error response
+        error_response = DocumentRagResponse(
+            error=Error(type="no-documents", message="No documents found"),
+            response=None
+        )
+        assert error_response.error is not None
+        assert error_response.error.type == "no-documents"
+        assert error_response.response is None
+
+
+@pytest.mark.contract
+class TestAgentMessageContracts:
+    """Contract tests for Agent message schemas"""
+
+    def test_agent_request_schema_contract(self, sample_message_data):
+        """Test AgentRequest schema contract"""
+        # Arrange
+        request_data = sample_message_data["AgentRequest"]
+
+        # Act & Assert
+        assert validate_schema_contract(AgentRequest, request_data)
+        
+        # Test required fields
+        request = AgentRequest(**request_data)
+        assert hasattr(request, 'question')
+        assert hasattr(request, 'plan')
+        assert hasattr(request, 'state')
+        assert hasattr(request, 'history')
+
+    def test_agent_response_schema_contract(self, sample_message_data):
+        """Test AgentResponse schema contract"""
+        # Arrange
+        response_data = sample_message_data["AgentResponse"]
+
+        # Act & Assert
+        assert validate_schema_contract(AgentResponse, response_data)
+        
+        # Test required fields
+        response = AgentResponse(**response_data)
+        assert hasattr(response, 'answer')
+        assert hasattr(response, 'error')
+        assert hasattr(response, 'thought')
+        assert hasattr(response, 'observation')
+
+    def test_agent_step_schema_contract(self):
+        """Test AgentStep schema contract"""
+        # Arrange
+        step_data = {
+            "thought": "I need to search for information",
+            "action": "knowledge_query",
+            "arguments": {"question": "What is AI?"},
+            "observation": "AI is artificial intelligence"
+        }
+
+        # Act & Assert
+        assert validate_schema_contract(AgentStep, step_data)
+        
+        step = AgentStep(**step_data)
+        assert step.thought == "I need to search for information"
+        assert step.action == "knowledge_query"
+        assert step.arguments == {"question": "What is AI?"}
+        assert step.observation == "AI is artificial intelligence"
+
+    def test_agent_request_with_history_contract(self):
+        """Test AgentRequest with conversation history contract"""
+        # Arrange
+        history_steps = [
+            AgentStep(
+                thought="First thought",
+                action="first_action",
+                arguments={"param": "value"},
+                observation="First observation"
+            ),
+            AgentStep(
+                thought="Second thought",
+                action="second_action", 
+                arguments={"param2": "value2"},
+                observation="Second observation"
+            )
+        ]
+
+        # Act
+        request = AgentRequest(
+            question="What comes next?",
+            plan="Multi-step plan",
+            state="processing",
+            history=history_steps
+        )
+
+        # Assert
+        assert len(request.history) == 2
+        assert request.history[0].thought == "First thought"
+        assert request.history[1].action == "second_action"
+
+
+@pytest.mark.contract
+class TestGraphMessageContracts:
+    """Contract tests for Graph/Knowledge message schemas"""
+
+    def test_value_schema_contract(self, sample_message_data):
+        """Test Value schema contract"""
+        # Arrange
+        value_data = sample_message_data["Value"]
+
+        # Act & Assert
+        assert validate_schema_contract(Value, value_data)
+        
+        # Test URI value
+        uri_value = Value(**value_data)
+        assert uri_value.value == "http://example.com/entity"
+        assert uri_value.is_uri is True
+
+        # Test literal value
+        literal_value = Value(
+            value="Literal text value",
+            is_uri=False,
+            type=""
+        )
+        assert literal_value.value == "Literal text value"
+        assert literal_value.is_uri is False
+
+    def test_triple_schema_contract(self, sample_message_data):
+        """Test Triple schema contract"""
+        # Arrange
+        triple_data = sample_message_data["Triple"]
+
+        # Act & Assert - Triple uses Value objects, not dict validation
+        triple = Triple(
+            s=triple_data["s"],
+            p=triple_data["p"], 
+            o=triple_data["o"]
+        )
+        assert triple.s.value == "http://example.com/subject"
+        assert triple.p.value == "http://example.com/predicate"
+        assert triple.o.value == "Object value"
+        assert triple.s.is_uri is True
+        assert triple.p.is_uri is True
+        assert triple.o.is_uri is False
+
+    def test_triples_schema_contract(self, sample_message_data):
+        """Test Triples (batch) schema contract"""
+        # Arrange
+        metadata = Metadata(**sample_message_data["Metadata"])
+        triple = Triple(**sample_message_data["Triple"])
+        
+        triples_data = {
+            "metadata": metadata,
+            "triples": [triple]
+        }
+
+        # Act & Assert
+        assert validate_schema_contract(Triples, triples_data)
+        
+        triples = Triples(**triples_data)
+        assert triples.metadata.id == "test-doc-123"
+        assert len(triples.triples) == 1
+        assert triples.triples[0].s.value == "http://example.com/subject"
+
+    def test_chunk_schema_contract(self, sample_message_data):
+        """Test Chunk schema contract"""
+        # Arrange
+        metadata = Metadata(**sample_message_data["Metadata"])
+        chunk_data = {
+            "metadata": metadata,
+            "chunk": b"This is a text chunk for processing"
+        }
+
+        # Act & Assert
+        assert validate_schema_contract(Chunk, chunk_data)
+        
+        chunk = Chunk(**chunk_data)
+        assert chunk.metadata.id == "test-doc-123"
+        assert chunk.chunk == b"This is a text chunk for processing"
+
+    def test_entity_context_schema_contract(self):
+        """Test EntityContext schema contract"""
+        # Arrange
+        entity_value = Value(value="http://example.com/entity", is_uri=True, type="")
+        entity_context_data = {
+            "entity": entity_value,
+            "context": "Context information about the entity"
+        }
+
+        # Act & Assert
+        assert validate_schema_contract(EntityContext, entity_context_data)
+        
+        entity_context = EntityContext(**entity_context_data)
+        assert entity_context.entity.value == "http://example.com/entity"
+        assert entity_context.context == "Context information about the entity"
+
+    def test_entity_contexts_batch_schema_contract(self, sample_message_data):
+        """Test EntityContexts (batch) schema contract"""
+        # Arrange
+        metadata = Metadata(**sample_message_data["Metadata"])
+        entity_value = Value(value="http://example.com/entity", is_uri=True, type="")
+        entity_context = EntityContext(
+            entity=entity_value,
+            context="Entity context"
+        )
+        
+        entity_contexts_data = {
+            "metadata": metadata,
+            "entities": [entity_context]
+        }
+
+        # Act & Assert
+        assert validate_schema_contract(EntityContexts, entity_contexts_data)
+        
+        entity_contexts = EntityContexts(**entity_contexts_data)
+        assert entity_contexts.metadata.id == "test-doc-123"
+        assert len(entity_contexts.entities) == 1
+        assert entity_contexts.entities[0].context == "Entity context"
+
+
+@pytest.mark.contract
+class TestMetadataMessageContracts:
+    """Contract tests for Metadata and common message schemas"""
+
+    def test_metadata_schema_contract(self, sample_message_data):
+        """Test Metadata schema contract"""
+        # Arrange
+        metadata_data = sample_message_data["Metadata"]
+
+        # Act & Assert
+        assert validate_schema_contract(Metadata, metadata_data)
+        
+        metadata = Metadata(**metadata_data)
+        assert metadata.id == "test-doc-123"
+        assert metadata.user == "test_user"
+        assert metadata.collection == "test_collection"
+        assert isinstance(metadata.metadata, list)
+
+    def test_metadata_with_triples_contract(self, sample_message_data):
+        """Test Metadata with embedded triples contract"""
+        # Arrange
+        triple = Triple(**sample_message_data["Triple"])
+        metadata_data = {
+            "id": "doc-with-triples",
+            "user": "test_user",
+            "collection": "test_collection",
+            "metadata": [triple]
+        }
+
+        # Act & Assert
+        assert validate_schema_contract(Metadata, metadata_data)
+        
+        metadata = Metadata(**metadata_data)
+        assert len(metadata.metadata) == 1
+        assert metadata.metadata[0].s.value == "http://example.com/subject"
+
+    def test_error_schema_contract(self):
+        """Test Error schema contract"""
+        # Arrange
+        error_data = {
+            "type": "validation-error",
+            "message": "Invalid input data provided"
+        }
+
+        # Act & Assert
+        assert validate_schema_contract(Error, error_data)
+        
+        error = Error(**error_data)
+        assert error.type == "validation-error"
+        assert error.message == "Invalid input data provided"
+
+
+@pytest.mark.contract
+class TestMessageRoutingContracts:
+    """Contract tests for message routing and properties"""
+
+    def test_message_property_contracts(self, message_properties):
+        """Test standard message property contracts"""
+        # Act & Assert
+        required_properties = ["id", "routing_key", "timestamp", "source_service"]
+        
+        for prop in required_properties:
+            assert prop in message_properties
+            assert message_properties[prop] is not None
+            assert isinstance(message_properties[prop], str)
+
+    def test_message_id_format_contract(self, message_properties):
+        """Test message ID format contract"""
+        # Act & Assert
+        message_id = message_properties["id"]
+        assert isinstance(message_id, str)
+        assert len(message_id) > 0
+        # Message IDs should follow a consistent format
+        assert "test-message-" in message_id
+
+    def test_routing_key_format_contract(self, message_properties):
+        """Test routing key format contract"""
+        # Act & Assert
+        routing_key = message_properties["routing_key"]
+        assert isinstance(routing_key, str)
+        assert "." in routing_key  # Should use dot notation
+        assert routing_key.count(".") >= 2  # Should have at least 3 parts
+
+    def test_correlation_id_contract(self, message_properties):
+        """Test correlation ID contract for request/response tracking"""
+        # Act & Assert
+        correlation_id = message_properties.get("correlation_id")
+        if correlation_id is not None:
+            assert isinstance(correlation_id, str)
+            assert len(correlation_id) > 0
+
+
+@pytest.mark.contract
+class TestSchemaEvolutionContracts:
+    """Contract tests for schema evolution and backward compatibility"""
+
+    def test_schema_backward_compatibility(self, schema_evolution_data):
+        """Test schema backward compatibility"""
+        # Test that v1 data can still be processed
+        v1_request = schema_evolution_data["TextCompletionRequest_v1"]
+        
+        # Should work with current schema (optional fields default)
+        request = TextCompletionRequest(**v1_request)
+        assert request.system == "You are helpful."
+        assert request.prompt == "Test prompt"
+
+    def test_schema_forward_compatibility(self, schema_evolution_data):
+        """Test schema forward compatibility with new fields"""
+        # Test that v2 data works with additional fields
+        v2_request = schema_evolution_data["TextCompletionRequest_v2"]
+        
+        # Current schema should handle new fields gracefully
+        # (This would require actual schema versioning implementation)
+        base_fields = {"system": v2_request["system"], "prompt": v2_request["prompt"]}
+        request = TextCompletionRequest(**base_fields)
+        assert request.system == "You are helpful."
+        assert request.prompt == "Test prompt"
+
+    def test_required_field_stability_contract(self):
+        """Test that required fields remain stable across versions"""
+        # These fields should never become optional or be removed
+        required_fields = {
+            "TextCompletionRequest": ["system", "prompt"],
+            "TextCompletionResponse": ["error", "response", "model"],
+            "DocumentRagQuery": ["query", "user", "collection"],
+            "DocumentRagResponse": ["error", "response"],
+            "AgentRequest": ["question", "history"],
+            "AgentResponse": ["error"],
+        }
+
+        # Verify required fields are present in schema definitions
+        for schema_name, fields in required_fields.items():
+            # This would be implemented with actual schema introspection
+            # For now, we verify by attempting to create instances
+            assert len(fields) > 0  # Ensure we have defined required fields
+
+
+@pytest.mark.contract
+class TestSerializationContracts:
+    """Contract tests for message serialization/deserialization"""
+
+    def test_all_schemas_serialization_contract(self, schema_registry, sample_message_data):
+        """Test serialization contract for all schemas"""
+        # Test each schema in the registry
+        for schema_name, schema_class in schema_registry.items():
+            if schema_name in sample_message_data:
+                # Skip Triple schema as it requires special handling with Value objects
+                if schema_name == "Triple":
+                    continue
+                    
+                # Act & Assert
+                data = sample_message_data[schema_name]
+                assert serialize_deserialize_test(schema_class, data), f"Serialization failed for {schema_name}"
+    
+    def test_triple_serialization_contract(self, sample_message_data):
+        """Test Triple schema serialization contract with Value objects"""
+        # Arrange
+        triple_data = sample_message_data["Triple"]
+        
+        # Act
+        triple = Triple(
+            s=triple_data["s"],
+            p=triple_data["p"], 
+            o=triple_data["o"]
+        )
+        
+        # Assert - Test that Value objects are properly constructed and accessible
+        assert triple.s.value == "http://example.com/subject"
+        assert triple.p.value == "http://example.com/predicate"
+        assert triple.o.value == "Object value"
+        assert isinstance(triple.s, Value)
+        assert isinstance(triple.p, Value)
+        assert isinstance(triple.o, Value)
+
+    def test_nested_schema_serialization_contract(self, sample_message_data):
+        """Test serialization of nested schemas"""
+        # Test Triples (contains Metadata and Triple objects)
+        metadata = Metadata(**sample_message_data["Metadata"])
+        triple = Triple(**sample_message_data["Triple"])
+        
+        triples = Triples(metadata=metadata, triples=[triple])
+        
+        # Verify nested objects maintain their contracts
+        assert triples.metadata.id == "test-doc-123"
+        assert triples.triples[0].s.value == "http://example.com/subject"
+
+    def test_array_field_serialization_contract(self):
+        """Test serialization of array fields"""
+        # Test AgentRequest with history array
+        steps = [
+            AgentStep(
+                thought=f"Step {i}",
+                action=f"action_{i}",
+                arguments={f"param_{i}": f"value_{i}"},
+                observation=f"Observation {i}"
+            )
+            for i in range(3)
+        ]
+        
+        request = AgentRequest(
+            question="Test with array",
+            plan="Test plan",
+            state="Test state",
+            history=steps
+        )
+        
+        # Verify array serialization maintains order and content
+        assert len(request.history) == 3
+        assert request.history[0].thought == "Step 0"
+        assert request.history[2].action == "action_2"
+
+    def test_optional_field_serialization_contract(self):
+        """Test serialization contract for optional fields"""
+        # Test with minimal required fields
+        minimal_response = TextCompletionResponse(
+            error=None,
+            response="Test",
+            in_token=None,  # Optional field
+            out_token=None,  # Optional field
+            model="test-model"
+        )
+        
+        assert minimal_response.response == "Test"
+        assert minimal_response.in_token is None
+        assert minimal_response.out_token is None

tests/contract/test_objects_cassandra_contracts.py

@@ -0,0 +1,306 @@
+"""
+Contract tests for Cassandra Object Storage
+
+These tests verify the message contracts and schema compatibility
+for the objects storage processor.
+"""
+
+import pytest
+import json
+from pulsar.schema import AvroSchema
+
+from trustgraph.schema import ExtractedObject, Metadata, RowSchema, Field
+from trustgraph.storage.objects.cassandra.write import Processor
+
+
+@pytest.mark.contract
+class TestObjectsCassandraContracts:
+    """Contract tests for Cassandra object storage messages"""
+
+    def test_extracted_object_input_contract(self):
+        """Test that ExtractedObject schema matches expected input format"""
+        # Create test object with all required fields
+        test_metadata = Metadata(
+            id="test-doc-001",
+            user="test_user",
+            collection="test_collection",
+            metadata=[]
+        )
+        
+        test_object = ExtractedObject(
+            metadata=test_metadata,
+            schema_name="customer_records",
+            values={
+                "customer_id": "CUST123",
+                "name": "Test Customer",
+                "email": "test@example.com"
+            },
+            confidence=0.95,
+            source_span="Customer data from document..."
+        )
+        
+        # Verify all required fields are present
+        assert hasattr(test_object, 'metadata')
+        assert hasattr(test_object, 'schema_name')
+        assert hasattr(test_object, 'values')
+        assert hasattr(test_object, 'confidence')
+        assert hasattr(test_object, 'source_span')
+        
+        # Verify metadata structure
+        assert hasattr(test_object.metadata, 'id')
+        assert hasattr(test_object.metadata, 'user')
+        assert hasattr(test_object.metadata, 'collection')
+        assert hasattr(test_object.metadata, 'metadata')
+        
+        # Verify types
+        assert isinstance(test_object.schema_name, str)
+        assert isinstance(test_object.values, dict)
+        assert isinstance(test_object.confidence, float)
+        assert isinstance(test_object.source_span, str)
+
+    def test_row_schema_structure_contract(self):
+        """Test RowSchema structure used for table definitions"""
+        # Create test schema
+        test_fields = [
+            Field(
+                name="id",
+                type="string",
+                size=50,
+                primary=True,
+                description="Primary key",
+                required=True,
+                enum_values=[],
+                indexed=False
+            ),
+            Field(
+                name="status",
+                type="string",
+                size=20,
+                primary=False,
+                description="Status field",
+                required=False,
+                enum_values=["active", "inactive", "pending"],
+                indexed=True
+            )
+        ]
+        
+        test_schema = RowSchema(
+            name="test_table",
+            description="Test table schema",
+            fields=test_fields
+        )
+        
+        # Verify schema structure
+        assert hasattr(test_schema, 'name')
+        assert hasattr(test_schema, 'description')
+        assert hasattr(test_schema, 'fields')
+        assert isinstance(test_schema.fields, list)
+        
+        # Verify field structure
+        for field in test_schema.fields:
+            assert hasattr(field, 'name')
+            assert hasattr(field, 'type')
+            assert hasattr(field, 'size')
+            assert hasattr(field, 'primary')
+            assert hasattr(field, 'description')
+            assert hasattr(field, 'required')
+            assert hasattr(field, 'enum_values')
+            assert hasattr(field, 'indexed')
+
+    def test_schema_config_format_contract(self):
+        """Test the expected configuration format for schemas"""
+        # Define expected config structure
+        config_format = {
+            "schema": {
+                "table_name": json.dumps({
+                    "name": "table_name",
+                    "description": "Table description",
+                    "fields": [
+                        {
+                            "name": "field_name",
+                            "type": "string",
+                            "size": 0,
+                            "primary_key": True,
+                            "description": "Field description",
+                            "required": True,
+                            "enum": [],
+                            "indexed": False
+                        }
+                    ]
+                })
+            }
+        }
+        
+        # Verify config can be parsed
+        schema_json = json.loads(config_format["schema"]["table_name"])
+        assert "name" in schema_json
+        assert "fields" in schema_json
+        assert isinstance(schema_json["fields"], list)
+        
+        # Verify field format
+        field = schema_json["fields"][0]
+        required_field_keys = {"name", "type"}
+        optional_field_keys = {"size", "primary_key", "description", "required", "enum", "indexed"}
+        
+        assert required_field_keys.issubset(field.keys())
+        assert set(field.keys()).issubset(required_field_keys | optional_field_keys)
+
+    def test_cassandra_type_mapping_contract(self):
+        """Test that all supported field types have Cassandra mappings"""
+        processor = Processor.__new__(Processor)
+        
+        # All field types that should be supported
+        supported_types = [
+            ("string", "text"),
+            ("integer", "int"),  # or bigint based on size
+            ("float", "float"),  # or double based on size
+            ("boolean", "boolean"),
+            ("timestamp", "timestamp"),
+            ("date", "date"),
+            ("time", "time"),
+            ("uuid", "uuid")
+        ]
+        
+        for field_type, expected_cassandra_type in supported_types:
+            cassandra_type = processor.get_cassandra_type(field_type)
+            # For integer and float, the exact type depends on size
+            if field_type in ["integer", "float"]:
+                assert cassandra_type in ["int", "bigint", "float", "double"]
+            else:
+                assert cassandra_type == expected_cassandra_type
+
+    def test_value_conversion_contract(self):
+        """Test value conversion for all supported types"""
+        processor = Processor.__new__(Processor)
+        
+        # Test conversions maintain data integrity
+        test_cases = [
+            # (input_value, field_type, expected_output, expected_type)
+            ("123", "integer", 123, int),
+            ("123.45", "float", 123.45, float),
+            ("true", "boolean", True, bool),
+            ("false", "boolean", False, bool),
+            ("test string", "string", "test string", str),
+            (None, "string", None, type(None)),
+        ]
+        
+        for input_val, field_type, expected_val, expected_type in test_cases:
+            result = processor.convert_value(input_val, field_type)
+            assert result == expected_val
+            assert isinstance(result, expected_type) or result is None
+
+    def test_extracted_object_serialization_contract(self):
+        """Test that ExtractedObject can be serialized/deserialized correctly"""
+        # Create test object
+        original = ExtractedObject(
+            metadata=Metadata(
+                id="serial-001",
+                user="test_user",
+                collection="test_coll",
+                metadata=[]
+            ),
+            schema_name="test_schema",
+            values={"field1": "value1", "field2": "123"},
+            confidence=0.85,
+            source_span="Test span"
+        )
+        
+        # Test serialization using schema
+        schema = AvroSchema(ExtractedObject)
+        
+        # Encode and decode
+        encoded = schema.encode(original)
+        decoded = schema.decode(encoded)
+        
+        # Verify round-trip
+        assert decoded.metadata.id == original.metadata.id
+        assert decoded.metadata.user == original.metadata.user
+        assert decoded.metadata.collection == original.metadata.collection
+        assert decoded.schema_name == original.schema_name
+        assert decoded.values == original.values
+        assert decoded.confidence == original.confidence
+        assert decoded.source_span == original.source_span
+
+    def test_cassandra_table_naming_contract(self):
+        """Test Cassandra naming conventions and constraints"""
+        processor = Processor.__new__(Processor)
+        
+        # Test table naming (always gets o_ prefix)
+        table_test_names = [
+            ("simple_name", "o_simple_name"),
+            ("Name-With-Dashes", "o_name_with_dashes"),
+            ("name.with.dots", "o_name_with_dots"),
+            ("123_numbers", "o_123_numbers"),
+            ("special!@#chars", "o_special___chars"),  # 3 special chars become 3 underscores
+            ("UPPERCASE", "o_uppercase"),
+            ("CamelCase", "o_camelcase"),
+            ("", "o_"),  # Edge case - empty string becomes o_
+        ]
+        
+        for input_name, expected_name in table_test_names:
+            result = processor.sanitize_table(input_name)
+            assert result == expected_name
+            # Verify result is valid Cassandra identifier (starts with letter)
+            assert result.startswith('o_')
+            assert result.replace('o_', '').replace('_', '').isalnum() or result == 'o_'
+        
+        # Test regular name sanitization (only adds o_ prefix if starts with number)
+        name_test_cases = [
+            ("simple_name", "simple_name"),
+            ("Name-With-Dashes", "name_with_dashes"),
+            ("name.with.dots", "name_with_dots"),
+            ("123_numbers", "o_123_numbers"),  # Only this gets o_ prefix
+            ("special!@#chars", "special___chars"),  # 3 special chars become 3 underscores
+            ("UPPERCASE", "uppercase"),
+            ("CamelCase", "camelcase"),
+        ]
+        
+        for input_name, expected_name in name_test_cases:
+            result = processor.sanitize_name(input_name)
+            assert result == expected_name
+
+    def test_primary_key_structure_contract(self):
+        """Test that primary key structure follows Cassandra best practices"""
+        # Verify partition key always includes collection
+        processor = Processor.__new__(Processor)
+        processor.schemas = {}
+        processor.known_keyspaces = set()
+        processor.known_tables = {}
+        processor.session = None
+        
+        # Test schema with primary key
+        schema_with_pk = RowSchema(
+            name="test",
+            fields=[
+                Field(name="id", type="string", primary=True),
+                Field(name="data", type="string")
+            ]
+        )
+        
+        # The primary key should be ((collection, id))
+        # This is verified in the implementation where collection
+        # is always first in the partition key
+
+    def test_metadata_field_usage_contract(self):
+        """Test that metadata fields are used correctly in storage"""
+        # Create test object
+        test_obj = ExtractedObject(
+            metadata=Metadata(
+                id="meta-001",
+                user="user123",  # -> keyspace
+                collection="coll456",  # -> partition key
+                metadata=[{"key": "value"}]
+            ),
+            schema_name="table789",  # -> table name
+            values={"field": "value"},
+            confidence=0.9,
+            source_span="Source"
+        )
+        
+        # Verify mapping contract:
+        # - metadata.user -> Cassandra keyspace
+        # - schema_name -> Cassandra table
+        # - metadata.collection -> Part of primary key
+        assert test_obj.metadata.user  # Required for keyspace
+        assert test_obj.schema_name  # Required for table
+        assert test_obj.metadata.collection  # Required for partition key

tests/contract/test_structured_data_contracts.py

@@ -0,0 +1,308 @@
+"""
+Contract tests for Structured Data Pulsar Message Schemas
+
+These tests verify the contracts for all structured data Pulsar message schemas,
+ensuring schema compatibility, serialization contracts, and service interface stability.
+Following the TEST_STRATEGY.md approach for contract testing.
+"""
+
+import pytest
+import json
+from typing import Dict, Any
+
+from trustgraph.schema import (
+    StructuredDataSubmission, ExtractedObject,
+    NLPToStructuredQueryRequest, NLPToStructuredQueryResponse,
+    StructuredQueryRequest, StructuredQueryResponse,
+    StructuredObjectEmbedding, Field, RowSchema,
+    Metadata, Error, Value
+)
+from .conftest import serialize_deserialize_test
+
+
+@pytest.mark.contract
+class TestStructuredDataSchemaContracts:
+    """Contract tests for structured data schemas"""
+
+    def test_field_schema_contract(self):
+        """Test enhanced Field schema contract"""
+        # Arrange & Act - create Field instance directly
+        field = Field(
+            name="customer_id",
+            type="string",
+            size=0,
+            primary=True,
+            description="Unique customer identifier",
+            required=True,
+            enum_values=[],
+            indexed=True
+        )
+
+        # Assert - test field properties
+        assert field.name == "customer_id"
+        assert field.type == "string"
+        assert field.primary is True
+        assert field.indexed is True
+        assert isinstance(field.enum_values, list)
+        assert len(field.enum_values) == 0
+        
+        # Test with enum values
+        field_with_enum = Field(
+            name="status",
+            type="string",
+            size=0,
+            primary=False,
+            description="Status field",
+            required=False,
+            enum_values=["active", "inactive"],
+            indexed=True
+        )
+        
+        assert len(field_with_enum.enum_values) == 2
+        assert "active" in field_with_enum.enum_values
+
+    def test_row_schema_contract(self):
+        """Test RowSchema contract"""
+        # Arrange & Act
+        field = Field(
+            name="email",
+            type="string",
+            size=255,
+            primary=False,
+            description="Customer email",
+            required=True,
+            enum_values=[],
+            indexed=True
+        )
+        
+        schema = RowSchema(
+            name="customers",
+            description="Customer records schema",
+            fields=[field]
+        )
+
+        # Assert
+        assert schema.name == "customers"
+        assert schema.description == "Customer records schema"
+        assert len(schema.fields) == 1
+        assert schema.fields[0].name == "email"
+        assert schema.fields[0].indexed is True
+
+    def test_structured_data_submission_contract(self):
+        """Test StructuredDataSubmission schema contract"""
+        # Arrange
+        metadata = Metadata(
+            id="structured-data-001",
+            user="test_user",
+            collection="test_collection",
+            metadata=[]
+        )
+        
+        # Act
+        submission = StructuredDataSubmission(
+            metadata=metadata,
+            format="csv",
+            schema_name="customer_records",
+            data=b"id,name,email\n1,John,john@example.com",
+            options={"delimiter": ",", "header": "true"}
+        )
+
+        # Assert
+        assert submission.format == "csv"
+        assert submission.schema_name == "customer_records"
+        assert submission.options["delimiter"] == ","
+        assert submission.metadata.id == "structured-data-001"
+        assert len(submission.data) > 0
+
+    def test_extracted_object_contract(self):
+        """Test ExtractedObject schema contract"""
+        # Arrange
+        metadata = Metadata(
+            id="extracted-obj-001",
+            user="test_user",
+            collection="test_collection",
+            metadata=[]
+        )
+        
+        # Act
+        obj = ExtractedObject(
+            metadata=metadata,
+            schema_name="customer_records",
+            values={"id": "123", "name": "John Doe", "email": "john@example.com"},
+            confidence=0.95,
+            source_span="John Doe (john@example.com) customer ID 123"
+        )
+
+        # Assert
+        assert obj.schema_name == "customer_records"
+        assert obj.values["name"] == "John Doe"
+        assert obj.confidence == 0.95
+        assert len(obj.source_span) > 0
+        assert obj.metadata.id == "extracted-obj-001"
+
+
+@pytest.mark.contract
+class TestStructuredQueryServiceContracts:
+    """Contract tests for structured query services"""
+
+    def test_nlp_to_structured_query_request_contract(self):
+        """Test NLPToStructuredQueryRequest schema contract"""
+        # Act
+        request = NLPToStructuredQueryRequest(
+            natural_language_query="Show me all customers who registered last month",
+            max_results=100,
+            context_hints={"time_range": "last_month", "entity_type": "customer"}
+        )
+
+        # Assert
+        assert "customers" in request.natural_language_query
+        assert request.max_results == 100
+        assert request.context_hints["time_range"] == "last_month"
+
+    def test_nlp_to_structured_query_response_contract(self):
+        """Test NLPToStructuredQueryResponse schema contract"""
+        # Act
+        response = NLPToStructuredQueryResponse(
+            error=None,
+            graphql_query="query { customers(filter: {registered: {gte: \"2024-01-01\"}}) { id name email } }",
+            variables={"start_date": "2024-01-01"},
+            detected_schemas=["customers"],
+            confidence=0.92
+        )
+
+        # Assert
+        assert response.error is None
+        assert "customers" in response.graphql_query
+        assert response.detected_schemas[0] == "customers"
+        assert response.confidence > 0.9
+
+    def test_structured_query_request_contract(self):
+        """Test StructuredQueryRequest schema contract"""
+        # Act
+        request = StructuredQueryRequest(
+            query="query GetCustomers($limit: Int) { customers(limit: $limit) { id name email } }",
+            variables={"limit": "10"},
+            operation_name="GetCustomers"
+        )
+
+        # Assert
+        assert "customers" in request.query
+        assert request.variables["limit"] == "10"
+        assert request.operation_name == "GetCustomers"
+
+    def test_structured_query_response_contract(self):
+        """Test StructuredQueryResponse schema contract"""
+        # Act
+        response = StructuredQueryResponse(
+            error=None,
+            data='{"customers": [{"id": "1", "name": "John", "email": "john@example.com"}]}',
+            errors=[]
+        )
+
+        # Assert
+        assert response.error is None
+        assert "customers" in response.data
+        assert len(response.errors) == 0
+
+    def test_structured_query_response_with_errors_contract(self):
+        """Test StructuredQueryResponse with GraphQL errors contract"""
+        # Act
+        response = StructuredQueryResponse(
+            error=None,
+            data=None,
+            errors=["Field 'invalid_field' not found in schema 'customers'"]
+        )
+
+        # Assert
+        assert response.data is None
+        assert len(response.errors) == 1
+        assert "invalid_field" in response.errors[0]
+
+
+@pytest.mark.contract
+class TestStructuredEmbeddingsContracts:
+    """Contract tests for structured object embeddings"""
+
+    def test_structured_object_embedding_contract(self):
+        """Test StructuredObjectEmbedding schema contract"""
+        # Arrange
+        metadata = Metadata(
+            id="struct-embed-001",
+            user="test_user",
+            collection="test_collection",
+            metadata=[]
+        )
+        
+        # Act
+        embedding = StructuredObjectEmbedding(
+            metadata=metadata,
+            vectors=[[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]],
+            schema_name="customer_records",
+            object_id="customer_123",
+            field_embeddings={
+                "name": [0.1, 0.2, 0.3],
+                "email": [0.4, 0.5, 0.6]
+            }
+        )
+
+        # Assert
+        assert embedding.schema_name == "customer_records"
+        assert embedding.object_id == "customer_123"
+        assert len(embedding.vectors) == 2
+        assert len(embedding.field_embeddings) == 2
+        assert "name" in embedding.field_embeddings
+
+
+@pytest.mark.contract
+class TestStructuredDataSerializationContracts:
+    """Contract tests for structured data serialization/deserialization"""
+
+    def test_structured_data_submission_serialization(self):
+        """Test StructuredDataSubmission serialization contract"""
+        # Arrange
+        metadata = Metadata(id="test", user="user", collection="col", metadata=[])
+        submission_data = {
+            "metadata": metadata,
+            "format": "json",
+            "schema_name": "test_schema",
+            "data": b'{"test": "data"}',
+            "options": {"encoding": "utf-8"}
+        }
+
+        # Act & Assert
+        assert serialize_deserialize_test(StructuredDataSubmission, submission_data)
+
+    def test_extracted_object_serialization(self):
+        """Test ExtractedObject serialization contract"""
+        # Arrange
+        metadata = Metadata(id="test", user="user", collection="col", metadata=[])
+        object_data = {
+            "metadata": metadata,
+            "schema_name": "test_schema",
+            "values": {"field1": "value1"},
+            "confidence": 0.8,
+            "source_span": "test span"
+        }
+
+        # Act & Assert
+        assert serialize_deserialize_test(ExtractedObject, object_data)
+
+    def test_nlp_query_serialization(self):
+        """Test NLP query request/response serialization contract"""
+        # Test request
+        request_data = {
+            "natural_language_query": "test query",
+            "max_results": 10,
+            "context_hints": {}
+        }
+        assert serialize_deserialize_test(NLPToStructuredQueryRequest, request_data)
+
+        # Test response
+        response_data = {
+            "error": None,
+            "graphql_query": "query { test }",
+            "variables": {},
+            "detected_schemas": ["test"],
+            "confidence": 0.9
+        }
+        assert serialize_deserialize_test(NLPToStructuredQueryResponse, response_data)

tests/integration/README.md

@@ -0,0 +1,269 @@
+# Integration Test Pattern for TrustGraph
+
+This directory contains integration tests that verify the coordination between multiple TrustGraph services and components, following the patterns outlined in [TEST_STRATEGY.md](../../TEST_STRATEGY.md).
+
+## Integration Test Approach
+
+Integration tests focus on **service-to-service communication patterns** and **end-to-end message flows** while still using mocks for external infrastructure.
+
+### Key Principles
+
+1. **Test Service Coordination**: Verify that services work together correctly
+2. **Mock External Dependencies**: Use mocks for databases, APIs, and infrastructure
+3. **Real Business Logic**: Exercise actual service logic and data transformations
+4. **Error Propagation**: Test how errors flow through the system
+5. **Configuration Testing**: Verify services respond correctly to different configurations
+
+## Test Structure
+
+### Fixtures (conftest.py)
+
+Common fixtures for integration tests:
+- `mock_pulsar_client`: Mock Pulsar messaging client
+- `mock_flow_context`: Mock flow context for service coordination
+- `integration_config`: Standard configuration for integration tests
+- `sample_documents`: Test document collections
+- `sample_embeddings`: Test embedding vectors
+- `sample_queries`: Test query sets
+
+### Test Patterns
+
+#### 1. End-to-End Flow Testing
+
+```python
+@pytest.mark.integration
+@pytest.mark.asyncio
+async def test_service_end_to_end_flow(self, service_instance, mock_clients):
+    """Test complete service pipeline from input to output"""
+    # Arrange - Set up realistic test data
+    # Act - Execute the full service workflow
+    # Assert - Verify coordination between all components
+```
+
+#### 2. Error Propagation Testing
+
+```python
+@pytest.mark.integration
+@pytest.mark.asyncio
+async def test_service_error_handling(self, service_instance, mock_clients):
+    """Test how errors propagate through service coordination"""
+    # Arrange - Set up failure scenarios
+    # Act - Execute service with failing dependency
+    # Assert - Verify proper error handling and cleanup
+```
+
+#### 3. Configuration Testing
+
+```python
+@pytest.mark.integration
+@pytest.mark.asyncio
+async def test_service_configuration_scenarios(self, service_instance):
+    """Test service behavior with different configurations"""
+    # Test multiple configuration scenarios
+    # Verify service adapts correctly to each configuration
+```
+
+## Running Integration Tests
+
+### Run All Integration Tests
+```bash
+pytest tests/integration/ -m integration
+```
+
+### Run Specific Test
+```bash
+pytest tests/integration/test_document_rag_integration.py::TestDocumentRagIntegration::test_document_rag_end_to_end_flow -v
+```
+
+### Run with Coverage (Skip Coverage Requirement)
+```bash
+pytest tests/integration/ -m integration --cov=trustgraph --cov-fail-under=0
+```
+
+### Run Slow Tests
+```bash
+pytest tests/integration/ -m "integration and slow"
+```
+
+### Skip Slow Tests
+```bash
+pytest tests/integration/ -m "integration and not slow"
+```
+
+## Examples: Integration Test Implementations
+
+### 1. Document RAG Integration Test
+
+The `test_document_rag_integration.py` demonstrates the integration test pattern:
+
+### What It Tests
+- **Service Coordination**: Embeddings → Document Retrieval → Prompt Generation
+- **Error Handling**: Failure scenarios for each service dependency
+- **Configuration**: Different document limits, users, and collections
+- **Performance**: Large document set handling
+
+### Key Features
+- **Realistic Data Flow**: Uses actual service logic with mocked dependencies
+- **Multiple Scenarios**: Success, failure, and edge cases
+- **Verbose Logging**: Tests logging functionality
+- **Multi-User Support**: Tests user and collection isolation
+
+### Test Coverage
+- ✅ End-to-end happy path
+- ✅ No documents found scenario
+- ✅ Service failure scenarios (embeddings, documents, prompt)
+- ✅ Configuration variations
+- ✅ Multi-user isolation
+- ✅ Performance testing
+- ✅ Verbose logging
+
+### 2. Text Completion Integration Test
+
+The `test_text_completion_integration.py` demonstrates external API integration testing:
+
+### What It Tests
+- **External API Integration**: OpenAI API connectivity and authentication
+- **Rate Limiting**: Proper handling of API rate limits and retries
+- **Error Handling**: API failures, connection timeouts, and error propagation
+- **Token Tracking**: Accurate input/output token counting and metrics
+- **Configuration**: Different model parameters and settings
+- **Concurrency**: Multiple simultaneous API requests
+
+### Key Features
+- **Realistic Mock Responses**: Uses actual OpenAI API response structures
+- **Authentication Testing**: API key validation and base URL configuration
+- **Error Scenarios**: Rate limits, connection failures, invalid requests
+- **Performance Metrics**: Timing and token usage validation
+- **Model Flexibility**: Tests different GPT models and parameters
+
+### Test Coverage
+- ✅ Successful text completion generation
+- ✅ Multiple model configurations (GPT-3.5, GPT-4, GPT-4-turbo)
+- ✅ Rate limit handling (RateLimitError → TooManyRequests)
+- ✅ API error handling and propagation
+- ✅ Token counting accuracy
+- ✅ Prompt construction and parameter validation
+- ✅ Authentication patterns and API key validation
+- ✅ Concurrent request processing
+- ✅ Response content extraction and validation
+- ✅ Performance timing measurements
+
+### 3. Agent Manager Integration Test
+
+The `test_agent_manager_integration.py` demonstrates complex service coordination testing:
+
+### What It Tests
+- **ReAct Pattern**: Think-Act-Observe cycles with multi-step reasoning
+- **Tool Coordination**: Selection and execution of different tools (knowledge query, text completion, MCP tools)
+- **Conversation State**: Management of conversation history and context
+- **Multi-Service Integration**: Coordination between prompt, graph RAG, and tool services
+- **Error Handling**: Tool failures, unknown tools, and error propagation
+- **Configuration Management**: Dynamic tool loading and configuration
+
+### Key Features
+- **Complex Coordination**: Tests agent reasoning with multiple tool options
+- **Stateful Processing**: Maintains conversation history across interactions
+- **Dynamic Tool Selection**: Tests tool selection based on context and reasoning
+- **Callback Pattern**: Tests think/observe callback mechanisms
+- **JSON Serialization**: Handles complex data structures in prompts
+- **Performance Testing**: Large conversation history handling
+
+### Test Coverage
+- ✅ Basic reasoning cycle with tool selection
+- ✅ Final answer generation (ending ReAct cycle)
+- ✅ Full ReAct cycle with tool execution
+- ✅ Conversation history management
+- ✅ Multiple tool coordination and selection
+- ✅ Tool argument validation and processing
+- ✅ Error handling (unknown tools, execution failures)
+- ✅ Context integration and additional prompting
+- ✅ Empty tool configuration handling
+- ✅ Tool response processing and cleanup
+- ✅ Performance with large conversation history
+- ✅ JSON serialization in complex prompts
+
+### 4. Knowledge Graph Extract → Store Pipeline Integration Test
+
+The `test_kg_extract_store_integration.py` demonstrates multi-stage pipeline testing:
+
+### What It Tests
+- **Text-to-Graph Transformation**: Complete pipeline from text chunks to graph triples
+- **Entity Extraction**: Definition extraction with proper URI generation
+- **Relationship Extraction**: Subject-predicate-object relationship extraction
+- **Graph Database Integration**: Storage coordination with Cassandra knowledge store
+- **Data Validation**: Entity filtering, validation, and consistency checks
+- **Pipeline Coordination**: Multi-stage processing with proper data flow
+
+### Key Features
+- **Multi-Stage Pipeline**: Tests definitions → relationships → storage coordination
+- **Graph Data Structures**: RDF triples, entity contexts, and graph embeddings
+- **URI Generation**: Consistent entity URI creation across pipeline stages
+- **Data Transformation**: Complex text analysis to structured graph data
+- **Batch Processing**: Large document set processing performance
+- **Error Resilience**: Graceful handling of extraction failures
+
+### Test Coverage
+- ✅ Definitions extraction pipeline (text → entities + definitions)
+- ✅ Relationships extraction pipeline (text → subject-predicate-object)
+- ✅ URI generation consistency between processors
+- ✅ Triple generation from definitions and relationships
+- ✅ Knowledge store integration (triples and embeddings storage)
+- ✅ End-to-end pipeline coordination
+- ✅ Error handling in extraction services
+- ✅ Empty and invalid extraction results handling
+- ✅ Entity filtering and validation
+- ✅ Large batch processing performance
+- ✅ Metadata propagation through pipeline stages
+
+## Best Practices
+
+### Test Organization
+- Group related tests in classes
+- Use descriptive test names that explain the scenario
+- Follow the Arrange-Act-Assert pattern
+- Use appropriate pytest markers (`@pytest.mark.integration`, `@pytest.mark.slow`)
+
+### Mock Strategy
+- Mock external services (databases, APIs, message brokers)
+- Use real service logic and data transformations
+- Create realistic mock responses that match actual service behavior
+- Reset mocks between tests to ensure isolation
+
+### Test Data
+- Use realistic test data that reflects actual usage patterns
+- Create reusable fixtures for common test scenarios
+- Test with various data sizes and edge cases
+- Include both success and failure scenarios
+
+### Error Testing
+- Test each dependency failure scenario
+- Verify proper error propagation and cleanup
+- Test timeout and retry mechanisms
+- Validate error response formats
+
+### Performance Testing
+- Mark performance tests with `@pytest.mark.slow`
+- Test with realistic data volumes
+- Set reasonable performance expectations
+- Monitor resource usage during tests
+
+## Adding New Integration Tests
+
+1. **Identify Service Dependencies**: Map out which services your target service coordinates with
+2. **Create Mock Fixtures**: Set up mocks for each dependency in conftest.py
+3. **Design Test Scenarios**: Plan happy path, error cases, and edge conditions
+4. **Implement Tests**: Follow the established patterns in this directory
+5. **Add Documentation**: Update this README with your new test patterns
+
+## Test Markers
+
+- `@pytest.mark.integration`: Marks tests as integration tests
+- `@pytest.mark.slow`: Marks tests that take longer to run
+- `@pytest.mark.asyncio`: Required for async test functions
+
+## Future Enhancements
+
+- Add tests with real test containers for database integration
+- Implement contract testing for service interfaces
+- Add performance benchmarking for critical paths
+- Create integration test templates for common service patterns

tests/integration/cassandra_test_helper.py

@@ -0,0 +1,112 @@
+"""
+Helper for managing Cassandra containers in integration tests
+Alternative to testcontainers for Fedora/Podman compatibility
+"""
+
+import subprocess
+import time
+import socket
+from contextlib import contextmanager
+from cassandra.cluster import Cluster
+from cassandra.policies import RetryPolicy
+
+
+class CassandraTestContainer:
+    """Simple Cassandra container manager using Podman"""
+    
+    def __init__(self, image="docker.io/library/cassandra:4.1", port=9042):
+        self.image = image
+        self.port = port
+        self.container_name = f"test-cassandra-{int(time.time())}"
+        self.container_id = None
+    
+    def start(self):
+        """Start Cassandra container"""
+        # Remove any existing container with same name
+        subprocess.run([
+            "podman", "rm", "-f", self.container_name
+        ], capture_output=True)
+        
+        # Start new container with faster startup options
+        result = subprocess.run([
+            "podman", "run", "-d", 
+            "--name", self.container_name,
+            "-p", f"{self.port}:9042",
+            "-e", "JVM_OPTS=-Dcassandra.skip_wait_for_gossip_to_settle=0",
+            self.image
+        ], capture_output=True, text=True)
+        
+        if result.returncode != 0:
+            raise RuntimeError(f"Failed to start container: {result.stderr}")
+        
+        self.container_id = result.stdout.strip()
+        
+        # Wait for Cassandra to be ready
+        self._wait_for_ready()
+        return self
+    
+    def stop(self):
+        """Stop and remove container"""
+        import time
+        if self.container_name:
+            # Small delay before stopping to ensure connections are closed
+            time.sleep(0.5)
+            subprocess.run([
+                "podman", "rm", "-f", self.container_name
+            ], capture_output=True)
+    
+    def get_connection_host_port(self):
+        """Get host and port for connection"""
+        return "localhost", self.port
+    
+    def _wait_for_ready(self, timeout=120):
+        """Wait for Cassandra to be ready for CQL queries"""
+        start_time = time.time()
+        
+        print(f"Waiting for Cassandra to be ready on port {self.port}...")
+        
+        while time.time() - start_time < timeout:
+            try:
+                # First check if port is open
+                sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+                sock.settimeout(1)
+                result = sock.connect_ex(("localhost", self.port))
+                sock.close()
+                
+                if result == 0:
+                    # Port is open, now try to connect with Cassandra driver
+                    try:
+                        cluster = Cluster(['localhost'], port=self.port)
+                        cluster.connect_timeout = 5
+                        session = cluster.connect()
+                        
+                        # Try a simple query to verify Cassandra is ready
+                        session.execute("SELECT release_version FROM system.local")
+                        session.shutdown()
+                        cluster.shutdown()
+                        
+                        print("Cassandra is ready!")
+                        return
+                        
+                    except Exception as e:
+                        print(f"Cassandra not ready yet: {e}")
+                        pass
+                        
+            except Exception as e:
+                print(f"Connection check failed: {e}")
+                pass
+            
+            time.sleep(3)
+        
+        raise RuntimeError(f"Cassandra not ready after {timeout} seconds")
+
+
+@contextmanager
+def cassandra_container(image="docker.io/library/cassandra:4.1", port=9042):
+    """Context manager for Cassandra container"""
+    container = CassandraTestContainer(image, port)
+    try:
+        container.start()
+        yield container
+    finally:
+        container.stop()

tests/integration/conftest.py

@@ -0,0 +1,404 @@
+"""
+Shared fixtures and configuration for integration tests
+
+This file provides common fixtures and test configuration for integration tests.
+Following the TEST_STRATEGY.md patterns for integration testing.
+"""
+
+import pytest
+from unittest.mock import AsyncMock, MagicMock
+
+
+@pytest.fixture
+def mock_pulsar_client():
+    """Mock Pulsar client for integration tests"""
+    client = MagicMock()
+    client.create_producer.return_value = AsyncMock()
+    client.subscribe.return_value = AsyncMock()
+    return client
+
+
+@pytest.fixture
+def mock_flow_context():
+    """Mock flow context for testing service coordination"""
+    context = MagicMock()
+    
+    # Mock flow producers/consumers
+    context.return_value.send = AsyncMock()
+    context.return_value.receive = AsyncMock()
+    
+    return context
+
+
+@pytest.fixture
+def integration_config():
+    """Common configuration for integration tests"""
+    return {
+        "pulsar_host": "localhost",
+        "pulsar_port": 6650,
+        "test_timeout": 30.0,
+        "max_retries": 3,
+        "doc_limit": 10,
+        "embedding_dim": 5,
+    }
+
+
+@pytest.fixture
+def sample_documents():
+    """Sample document collection for testing"""
+    return [
+        {
+            "id": "doc1",
+            "content": "Machine learning is a subset of artificial intelligence that focuses on algorithms that learn from data.",
+            "collection": "ml_knowledge",
+            "user": "test_user"
+        },
+        {
+            "id": "doc2", 
+            "content": "Deep learning uses neural networks with multiple layers to model complex patterns in data.",
+            "collection": "ml_knowledge",
+            "user": "test_user"
+        },
+        {
+            "id": "doc3",
+            "content": "Supervised learning algorithms learn from labeled training data to make predictions on new data.",
+            "collection": "ml_knowledge", 
+            "user": "test_user"
+        }
+    ]
+
+
+@pytest.fixture
+def sample_embeddings():
+    """Sample embedding vectors for testing"""
+    return [
+        [0.1, 0.2, 0.3, 0.4, 0.5],
+        [0.6, 0.7, 0.8, 0.9, 1.0],
+        [0.2, 0.3, 0.4, 0.5, 0.6],
+        [0.7, 0.8, 0.9, 1.0, 0.1],
+        [0.3, 0.4, 0.5, 0.6, 0.7]
+    ]
+
+
+@pytest.fixture
+def sample_queries():
+    """Sample queries for testing"""
+    return [
+        "What is machine learning?",
+        "How does deep learning work?",
+        "Explain supervised learning",
+        "What are neural networks?",
+        "How do algorithms learn from data?"
+    ]
+
+
+@pytest.fixture
+def sample_text_completion_requests():
+    """Sample text completion requests for testing"""
+    return [
+        {
+            "system": "You are a helpful assistant.",
+            "prompt": "What is artificial intelligence?",
+            "expected_keywords": ["artificial intelligence", "AI", "machine learning"]
+        },
+        {
+            "system": "You are a technical expert.",
+            "prompt": "Explain neural networks",
+            "expected_keywords": ["neural networks", "neurons", "layers"]
+        },
+        {
+            "system": "You are a teacher.",
+            "prompt": "What is supervised learning?",
+            "expected_keywords": ["supervised learning", "training", "labels"]
+        }
+    ]
+
+
+@pytest.fixture
+def mock_openai_response():
+    """Mock OpenAI API response structure"""
+    return {
+        "id": "chatcmpl-test123",
+        "object": "chat.completion",
+        "created": 1234567890,
+        "model": "gpt-3.5-turbo",
+        "choices": [
+            {
+                "index": 0,
+                "message": {
+                    "role": "assistant",
+                    "content": "This is a test response from the AI model."
+                },
+                "finish_reason": "stop"
+            }
+        ],
+        "usage": {
+            "prompt_tokens": 50,
+            "completion_tokens": 100,
+            "total_tokens": 150
+        }
+    }
+
+
+@pytest.fixture
+def text_completion_configs():
+    """Various text completion configurations for testing"""
+    return [
+        {
+            "model": "gpt-3.5-turbo",
+            "temperature": 0.0,
+            "max_output": 1024,
+            "description": "Conservative settings"
+        },
+        {
+            "model": "gpt-4",
+            "temperature": 0.7,
+            "max_output": 2048,
+            "description": "Balanced settings"
+        },
+        {
+            "model": "gpt-4-turbo",
+            "temperature": 1.0,
+            "max_output": 4096,
+            "description": "Creative settings"
+        }
+    ]
+
+
+@pytest.fixture
+def sample_agent_tools():
+    """Sample agent tools configuration for testing"""
+    return {
+        "knowledge_query": {
+            "name": "knowledge_query",
+            "description": "Query the knowledge graph for information",
+            "type": "knowledge-query",
+            "arguments": [
+                {
+                    "name": "question",
+                    "type": "string",
+                    "description": "The question to ask the knowledge graph"
+                }
+            ]
+        },
+        "text_completion": {
+            "name": "text_completion", 
+            "description": "Generate text completion using LLM",
+            "type": "text-completion",
+            "arguments": [
+                {
+                    "name": "question",
+                    "type": "string",
+                    "description": "The question to ask the LLM"
+                }
+            ]
+        },
+        "web_search": {
+            "name": "web_search",
+            "description": "Search the web for information",
+            "type": "mcp-tool",
+            "arguments": [
+                {
+                    "name": "query",
+                    "type": "string",
+                    "description": "The search query"
+                }
+            ]
+        }
+    }
+
+
+@pytest.fixture
+def sample_agent_requests():
+    """Sample agent requests for testing"""
+    return [
+        {
+            "question": "What is machine learning?",
+            "plan": "",
+            "state": "",
+            "history": [],
+            "expected_tool": "knowledge_query"
+        },
+        {
+            "question": "Can you explain neural networks in simple terms?",
+            "plan": "",
+            "state": "",
+            "history": [],
+            "expected_tool": "text_completion"
+        },
+        {
+            "question": "Search for the latest AI research papers",
+            "plan": "",
+            "state": "",
+            "history": [],
+            "expected_tool": "web_search"
+        }
+    ]
+
+
+@pytest.fixture
+def sample_agent_responses():
+    """Sample agent responses for testing"""
+    return [
+        {
+            "thought": "I need to search for information about machine learning",
+            "action": "knowledge_query",
+            "arguments": {"question": "What is machine learning?"}
+        },
+        {
+            "thought": "I can provide a direct answer about neural networks",
+            "final-answer": "Neural networks are computing systems inspired by biological neural networks."
+        },
+        {
+            "thought": "I should search the web for recent research",
+            "action": "web_search",
+            "arguments": {"query": "latest AI research papers 2024"}
+        }
+    ]
+
+
+@pytest.fixture
+def sample_conversation_history():
+    """Sample conversation history for testing"""
+    return [
+        {
+            "thought": "I need to search for basic information first",
+            "action": "knowledge_query",
+            "arguments": {"question": "What is artificial intelligence?"},
+            "observation": "AI is the simulation of human intelligence in machines."
+        },
+        {
+            "thought": "Now I can provide more specific information",
+            "action": "text_completion",
+            "arguments": {"question": "Explain machine learning within AI"},
+            "observation": "Machine learning is a subset of AI that enables computers to learn from data."
+        }
+    ]
+
+
+@pytest.fixture
+def sample_kg_extraction_data():
+    """Sample knowledge graph extraction data for testing"""
+    return {
+        "text_chunks": [
+            "Machine Learning is a subset of Artificial Intelligence that enables computers to learn from data.",
+            "Neural Networks are computing systems inspired by biological neural networks.",
+            "Deep Learning uses neural networks with multiple layers to model complex patterns."
+        ],
+        "expected_entities": [
+            "Machine Learning",
+            "Artificial Intelligence", 
+            "Neural Networks",
+            "Deep Learning"
+        ],
+        "expected_relationships": [
+            {
+                "subject": "Machine Learning",
+                "predicate": "is_subset_of",
+                "object": "Artificial Intelligence"
+            },
+            {
+                "subject": "Deep Learning",
+                "predicate": "uses",
+                "object": "Neural Networks"
+            }
+        ]
+    }
+
+
+@pytest.fixture
+def sample_kg_definitions():
+    """Sample knowledge graph definitions for testing"""
+    return [
+        {
+            "entity": "Machine Learning",
+            "definition": "A subset of artificial intelligence that enables computers to learn from data without explicit programming."
+        },
+        {
+            "entity": "Artificial Intelligence",
+            "definition": "The simulation of human intelligence in machines that are programmed to think and act like humans."
+        },
+        {
+            "entity": "Neural Networks",
+            "definition": "Computing systems inspired by biological neural networks that process information using interconnected nodes."
+        },
+        {
+            "entity": "Deep Learning",
+            "definition": "A subset of machine learning that uses neural networks with multiple layers to model complex patterns in data."
+        }
+    ]
+
+
+@pytest.fixture
+def sample_kg_relationships():
+    """Sample knowledge graph relationships for testing"""
+    return [
+        {
+            "subject": "Machine Learning",
+            "predicate": "is_subset_of",
+            "object": "Artificial Intelligence",
+            "object-entity": True
+        },
+        {
+            "subject": "Deep Learning",
+            "predicate": "is_subset_of",
+            "object": "Machine Learning",
+            "object-entity": True
+        },
+        {
+            "subject": "Neural Networks",
+            "predicate": "is_used_in",
+            "object": "Deep Learning",
+            "object-entity": True
+        },
+        {
+            "subject": "Machine Learning",
+            "predicate": "processes",
+            "object": "data patterns",
+            "object-entity": False
+        }
+    ]
+
+
+@pytest.fixture
+def sample_kg_triples():
+    """Sample knowledge graph triples for testing"""
+    return [
+        {
+            "subject": "http://trustgraph.ai/e/machine-learning",
+            "predicate": "http://www.w3.org/2000/01/rdf-schema#label",
+            "object": "Machine Learning"
+        },
+        {
+            "subject": "http://trustgraph.ai/e/machine-learning",
+            "predicate": "http://trustgraph.ai/definition",
+            "object": "A subset of artificial intelligence that enables computers to learn from data."
+        },
+        {
+            "subject": "http://trustgraph.ai/e/machine-learning",
+            "predicate": "http://trustgraph.ai/e/is_subset_of",
+            "object": "http://trustgraph.ai/e/artificial-intelligence"
+        }
+    ]
+
+
+# Test markers for integration tests
+pytestmark = pytest.mark.integration
+
+
+def pytest_sessionfinish(session, exitstatus):
+    """
+    Called after whole test run finished, right before returning the exit status.
+    
+    This hook is used to ensure Cassandra driver threads have time to shut down
+    properly before pytest exits, preventing "cannot schedule new futures after
+    shutdown" errors.
+    """
+    import time
+    import gc
+    
+    # Force garbage collection to clean up any remaining objects
+    gc.collect()
+    
+    # Give Cassandra driver threads more time to clean up
+    time.sleep(2)

tests/integration/test_agent_kg_extraction_integration.py

@@ -0,0 +1,481 @@
+"""
+Integration tests for Agent-based Knowledge Graph Extraction
+
+These tests verify the end-to-end functionality of the agent-driven knowledge graph
+extraction pipeline, testing the integration between agent communication, prompt
+rendering, JSON response processing, and knowledge graph generation.
+Following the TEST_STRATEGY.md approach for integration testing.
+"""
+
+import pytest
+import json
+from unittest.mock import AsyncMock, MagicMock, patch
+
+from trustgraph.extract.kg.agent.extract import Processor as AgentKgExtractor
+from trustgraph.schema import Chunk, Triple, Triples, Metadata, Value, Error
+from trustgraph.schema import EntityContext, EntityContexts, AgentRequest, AgentResponse
+from trustgraph.rdf import TRUSTGRAPH_ENTITIES, DEFINITION, RDF_LABEL, SUBJECT_OF
+from trustgraph.template.prompt_manager import PromptManager
+
+
+@pytest.mark.integration
+class TestAgentKgExtractionIntegration:
+    """Integration tests for Agent-based Knowledge Graph Extraction"""
+
+    @pytest.fixture
+    def mock_flow_context(self):
+        """Mock flow context for agent communication and output publishing"""
+        context = MagicMock()
+        
+        # Mock agent client
+        agent_client = AsyncMock()
+        
+        # Mock successful agent response
+        def mock_agent_response(recipient, question):
+            # Simulate agent processing and return structured response
+            mock_response = MagicMock()
+            mock_response.error = None
+            mock_response.answer = '''```json
+{
+    "definitions": [
+        {
+            "entity": "Machine Learning",
+            "definition": "A subset of artificial intelligence that enables computers to learn from data without explicit programming."
+        },
+        {
+            "entity": "Neural Networks", 
+            "definition": "Computing systems inspired by biological neural networks that process information."
+        }
+    ],
+    "relationships": [
+        {
+            "subject": "Machine Learning",
+            "predicate": "is_subset_of", 
+            "object": "Artificial Intelligence",
+            "object-entity": true
+        },
+        {
+            "subject": "Neural Networks",
+            "predicate": "used_in",
+            "object": "Machine Learning", 
+            "object-entity": true
+        }
+    ]
+}
+```'''
+            return mock_response.answer
+        
+        agent_client.invoke = mock_agent_response
+        
+        # Mock output publishers
+        triples_publisher = AsyncMock()
+        entity_contexts_publisher = AsyncMock()
+        
+        def context_router(service_name):
+            if service_name == "agent-request":
+                return agent_client
+            elif service_name == "triples":
+                return triples_publisher
+            elif service_name == "entity-contexts":
+                return entity_contexts_publisher
+            else:
+                return AsyncMock()
+        
+        context.side_effect = context_router
+        return context
+
+    @pytest.fixture
+    def sample_chunk(self):
+        """Sample text chunk for knowledge extraction"""
+        text = """
+        Machine Learning is a subset of Artificial Intelligence that enables computers 
+        to learn from data without explicit programming. Neural Networks are computing 
+        systems inspired by biological neural networks that process information. 
+        Neural Networks are commonly used in Machine Learning applications.
+        """
+        
+        return Chunk(
+            chunk=text.encode('utf-8'),
+            metadata=Metadata(
+                id="doc123",
+                metadata=[
+                    Triple(
+                        s=Value(value="doc123", is_uri=True),
+                        p=Value(value="http://example.org/type", is_uri=True),
+                        o=Value(value="document", is_uri=False)
+                    )
+                ]
+            )
+        )
+
+    @pytest.fixture
+    def configured_agent_extractor(self):
+        """Mock agent extractor with loaded configuration for integration testing"""
+        # Create a mock extractor that simulates the real behavior
+        from trustgraph.extract.kg.agent.extract import Processor
+        
+        # Create mock without calling __init__ to avoid FlowProcessor issues
+        extractor = MagicMock()
+        real_extractor = Processor.__new__(Processor)
+        
+        # Copy the methods we want to test
+        extractor.to_uri = real_extractor.to_uri
+        extractor.parse_json = real_extractor.parse_json
+        extractor.process_extraction_data = real_extractor.process_extraction_data
+        extractor.emit_triples = real_extractor.emit_triples
+        extractor.emit_entity_contexts = real_extractor.emit_entity_contexts
+        
+        # Set up the configuration and manager
+        extractor.manager = PromptManager()
+        extractor.template_id = "agent-kg-extract"
+        extractor.config_key = "prompt"
+        
+        # Mock configuration
+        config = {
+            "system": json.dumps("You are a knowledge extraction agent."),
+            "template-index": json.dumps(["agent-kg-extract"]),
+            "template.agent-kg-extract": json.dumps({
+                "prompt": "Extract entities and relationships from: {{ text }}",
+                "response-type": "json"
+            })
+        }
+        
+        # Load configuration
+        extractor.manager.load_config(config)
+        
+        # Mock the on_message method to simulate real behavior
+        async def mock_on_message(msg, consumer, flow):
+            v = msg.value()
+            chunk_text = v.chunk.decode('utf-8')
+            
+            # Render prompt
+            prompt = extractor.manager.render(extractor.template_id, {"text": chunk_text})
+            
+            # Get agent response (the mock returns a string directly)
+            agent_client = flow("agent-request")
+            agent_response = agent_client.invoke(recipient=lambda x: True, question=prompt)
+            
+            # Parse and process
+            extraction_data = extractor.parse_json(agent_response)
+            triples, entity_contexts = extractor.process_extraction_data(extraction_data, v.metadata)
+            
+            # Add metadata triples
+            for t in v.metadata.metadata:
+                triples.append(t)
+            
+            # Emit outputs
+            if triples:
+                await extractor.emit_triples(flow("triples"), v.metadata, triples)
+            if entity_contexts:
+                await extractor.emit_entity_contexts(flow("entity-contexts"), v.metadata, entity_contexts)
+        
+        extractor.on_message = mock_on_message
+        
+        return extractor
+
+    @pytest.mark.asyncio
+    async def test_end_to_end_knowledge_extraction(self, configured_agent_extractor, sample_chunk, mock_flow_context):
+        """Test complete end-to-end knowledge extraction workflow"""
+        # Arrange
+        mock_message = MagicMock()
+        mock_message.value.return_value = sample_chunk
+        mock_consumer = MagicMock()
+
+        # Act
+        await configured_agent_extractor.on_message(mock_message, mock_consumer, mock_flow_context)
+
+        # Assert
+        # Verify agent was called with rendered prompt
+        agent_client = mock_flow_context("agent-request")
+        # Check that the mock function was replaced and called
+        assert hasattr(agent_client, 'invoke')
+
+        # Verify triples were emitted
+        triples_publisher = mock_flow_context("triples")
+        triples_publisher.send.assert_called_once()
+        
+        sent_triples = triples_publisher.send.call_args[0][0]
+        assert isinstance(sent_triples, Triples)
+        assert sent_triples.metadata.id == "doc123"
+        assert len(sent_triples.triples) > 0
+        
+        # Check that we have definition triples
+        definition_triples = [t for t in sent_triples.triples if t.p.value == DEFINITION]
+        assert len(definition_triples) >= 2  # Should have definitions for ML and Neural Networks
+        
+        # Check that we have label triples
+        label_triples = [t for t in sent_triples.triples if t.p.value == RDF_LABEL]
+        assert len(label_triples) >= 2  # Should have labels for entities
+        
+        # Check subject-of relationships
+        subject_of_triples = [t for t in sent_triples.triples if t.p.value == SUBJECT_OF]
+        assert len(subject_of_triples) >= 2  # Entities should be linked to document
+
+        # Verify entity contexts were emitted
+        entity_contexts_publisher = mock_flow_context("entity-contexts")
+        entity_contexts_publisher.send.assert_called_once()
+        
+        sent_contexts = entity_contexts_publisher.send.call_args[0][0]
+        assert isinstance(sent_contexts, EntityContexts)
+        assert len(sent_contexts.entities) >= 2  # Should have contexts for both entities
+        
+        # Verify entity URIs are properly formed
+        entity_uris = [ec.entity.value for ec in sent_contexts.entities]
+        assert f"{TRUSTGRAPH_ENTITIES}Machine%20Learning" in entity_uris
+        assert f"{TRUSTGRAPH_ENTITIES}Neural%20Networks" in entity_uris
+
+    @pytest.mark.asyncio
+    async def test_agent_error_handling(self, configured_agent_extractor, sample_chunk, mock_flow_context):
+        """Test handling of agent errors"""
+        # Arrange - mock agent error response
+        agent_client = mock_flow_context("agent-request")
+        
+        def mock_error_response(recipient, question):
+            # Simulate agent error by raising an exception
+            raise RuntimeError("Agent processing failed")
+        
+        agent_client.invoke = mock_error_response
+        
+        mock_message = MagicMock()
+        mock_message.value.return_value = sample_chunk
+        mock_consumer = MagicMock()
+
+        # Act & Assert
+        with pytest.raises(RuntimeError) as exc_info:
+            await configured_agent_extractor.on_message(mock_message, mock_consumer, mock_flow_context)
+        
+        assert "Agent processing failed" in str(exc_info.value)
+
+    @pytest.mark.asyncio
+    async def test_invalid_json_response_handling(self, configured_agent_extractor, sample_chunk, mock_flow_context):
+        """Test handling of invalid JSON responses from agent"""
+        # Arrange - mock invalid JSON response
+        agent_client = mock_flow_context("agent-request")
+        
+        def mock_invalid_json_response(recipient, question):
+            return "This is not valid JSON at all"
+        
+        agent_client.invoke = mock_invalid_json_response
+        
+        mock_message = MagicMock()
+        mock_message.value.return_value = sample_chunk
+        mock_consumer = MagicMock()
+
+        # Act & Assert
+        with pytest.raises((ValueError, json.JSONDecodeError)):
+            await configured_agent_extractor.on_message(mock_message, mock_consumer, mock_flow_context)
+
+    @pytest.mark.asyncio
+    async def test_empty_extraction_results(self, configured_agent_extractor, sample_chunk, mock_flow_context):
+        """Test handling of empty extraction results"""
+        # Arrange - mock empty extraction response
+        agent_client = mock_flow_context("agent-request")
+        
+        def mock_empty_response(recipient, question):
+            return '{"definitions": [], "relationships": []}'
+        
+        agent_client.invoke = mock_empty_response
+        
+        mock_message = MagicMock()
+        mock_message.value.return_value = sample_chunk
+        mock_consumer = MagicMock()
+
+        # Act
+        await configured_agent_extractor.on_message(mock_message, mock_consumer, mock_flow_context)
+
+        # Assert
+        # Should still emit outputs (even if empty) to maintain flow consistency
+        triples_publisher = mock_flow_context("triples")
+        entity_contexts_publisher = mock_flow_context("entity-contexts")
+        
+        # Triples should include metadata triples at minimum
+        triples_publisher.send.assert_called_once()
+        sent_triples = triples_publisher.send.call_args[0][0]
+        assert isinstance(sent_triples, Triples)
+        
+        # Entity contexts should not be sent if empty
+        entity_contexts_publisher.send.assert_not_called()
+
+    @pytest.mark.asyncio
+    async def test_malformed_extraction_data(self, configured_agent_extractor, sample_chunk, mock_flow_context):
+        """Test handling of malformed extraction data"""
+        # Arrange - mock malformed extraction response
+        agent_client = mock_flow_context("agent-request")
+        
+        def mock_malformed_response(recipient, question):
+            return '''{"definitions": [{"entity": "Missing Definition"}], "relationships": [{"subject": "Missing Object"}]}'''
+        
+        agent_client.invoke = mock_malformed_response
+        
+        mock_message = MagicMock()
+        mock_message.value.return_value = sample_chunk
+        mock_consumer = MagicMock()
+
+        # Act & Assert
+        with pytest.raises(KeyError):
+            await configured_agent_extractor.on_message(mock_message, mock_consumer, mock_flow_context)
+
+    @pytest.mark.asyncio
+    async def test_prompt_rendering_integration(self, configured_agent_extractor, mock_flow_context):
+        """Test integration with prompt template rendering"""
+        # Create a chunk with specific text
+        test_text = "Test text for prompt rendering"
+        chunk = Chunk(
+            chunk=test_text.encode('utf-8'),
+            metadata=Metadata(id="test-doc", metadata=[])
+        )
+        
+        agent_client = mock_flow_context("agent-request")
+        
+        def capture_prompt(recipient, question):
+            # Verify the prompt contains the test text
+            assert test_text in question
+            return '{"definitions": [], "relationships": []}'
+        
+        agent_client.invoke = capture_prompt
+        
+        mock_message = MagicMock()
+        mock_message.value.return_value = chunk
+        mock_consumer = MagicMock()
+
+        # Act
+        await configured_agent_extractor.on_message(mock_message, mock_consumer, mock_flow_context)
+        
+        # Assert - prompt should have been rendered with the text
+        # The agent_client.invoke is a function, not a mock, so we verify it was called by checking the flow worked
+        assert hasattr(agent_client, 'invoke')
+
+    @pytest.mark.asyncio
+    async def test_concurrent_processing_simulation(self, configured_agent_extractor, mock_flow_context):
+        """Test simulation of concurrent chunk processing"""
+        # Create multiple chunks
+        chunks = []
+        for i in range(3):
+            text = f"Test document {i} content"
+            chunks.append(Chunk(
+                chunk=text.encode('utf-8'),
+                metadata=Metadata(id=f"doc{i}", metadata=[])
+            ))
+        
+        agent_client = mock_flow_context("agent-request")
+        responses = []
+        
+        def mock_response(recipient, question):
+            response = f'{{"definitions": [{{"entity": "Entity {len(responses)}", "definition": "Definition {len(responses)}"}}], "relationships": []}}'
+            responses.append(response)
+            return response
+        
+        agent_client.invoke = mock_response
+
+        # Process chunks sequentially (simulating concurrent processing)
+        for chunk in chunks:
+            mock_message = MagicMock()
+            mock_message.value.return_value = chunk
+            mock_consumer = MagicMock()
+            
+            await configured_agent_extractor.on_message(mock_message, mock_consumer, mock_flow_context)
+
+        # Assert
+        assert len(responses) == 3
+        
+        # Verify all chunks were processed
+        triples_publisher = mock_flow_context("triples")
+        assert triples_publisher.send.call_count == 3
+
+    @pytest.mark.asyncio
+    async def test_unicode_text_handling(self, configured_agent_extractor, mock_flow_context):
+        """Test handling of text with unicode characters"""
+        # Create chunk with unicode text
+        unicode_text = "Machine Learning (学习机器) は人工知能の一分野です。"
+        chunk = Chunk(
+            chunk=unicode_text.encode('utf-8'),
+            metadata=Metadata(id="unicode-doc", metadata=[])
+        )
+        
+        agent_client = mock_flow_context("agent-request")
+        
+        def mock_unicode_response(recipient, question):
+            # Verify unicode text was properly decoded and included
+            assert "学习机器" in question
+            assert "人工知能" in question
+            return '''{"definitions": [{"entity": "機械学習", "definition": "人工知能の一分野"}], "relationships": []}'''
+        
+        agent_client.invoke = mock_unicode_response
+        
+        mock_message = MagicMock()
+        mock_message.value.return_value = chunk
+        mock_consumer = MagicMock()
+
+        # Act
+        await configured_agent_extractor.on_message(mock_message, mock_consumer, mock_flow_context)
+
+        # Assert - should handle unicode properly
+        triples_publisher = mock_flow_context("triples")
+        triples_publisher.send.assert_called_once()
+        
+        sent_triples = triples_publisher.send.call_args[0][0]
+        # Check that unicode entity was properly processed
+        entity_labels = [t for t in sent_triples.triples if t.p.value == RDF_LABEL and t.o.value == "機械学習"]
+        assert len(entity_labels) > 0
+
+    @pytest.mark.asyncio
+    async def test_large_text_chunk_processing(self, configured_agent_extractor, mock_flow_context):
+        """Test processing of large text chunks"""
+        # Create a large text chunk
+        large_text = "Machine Learning is important. " * 1000  # Repeat to create large text
+        chunk = Chunk(
+            chunk=large_text.encode('utf-8'),
+            metadata=Metadata(id="large-doc", metadata=[])
+        )
+        
+        agent_client = mock_flow_context("agent-request")
+        
+        def mock_large_text_response(recipient, question):
+            # Verify large text was included
+            assert len(question) > 10000
+            return '''{"definitions": [{"entity": "Machine Learning", "definition": "Important AI technique"}], "relationships": []}'''
+        
+        agent_client.invoke = mock_large_text_response
+        
+        mock_message = MagicMock()
+        mock_message.value.return_value = chunk
+        mock_consumer = MagicMock()
+
+        # Act
+        await configured_agent_extractor.on_message(mock_message, mock_consumer, mock_flow_context)
+
+        # Assert - should handle large text without issues
+        triples_publisher = mock_flow_context("triples")
+        triples_publisher.send.assert_called_once()
+
+    def test_configuration_parameter_validation(self):
+        """Test parameter validation logic"""
+        # Test that default parameter logic would work
+        default_template_id = "agent-kg-extract"
+        default_config_type = "prompt" 
+        default_concurrency = 1
+        
+        # Simulate parameter handling
+        params = {}
+        template_id = params.get("template-id", default_template_id)
+        config_key = params.get("config-type", default_config_type)
+        concurrency = params.get("concurrency", default_concurrency)
+        
+        assert template_id == "agent-kg-extract"
+        assert config_key == "prompt"
+        assert concurrency == 1
+        
+        # Test with custom parameters
+        custom_params = {
+            "template-id": "custom-template",
+            "config-type": "custom-config", 
+            "concurrency": 10
+        }
+        
+        template_id = custom_params.get("template-id", default_template_id)
+        config_key = custom_params.get("config-type", default_config_type)
+        concurrency = custom_params.get("concurrency", default_concurrency)
+        
+        assert template_id == "custom-template"
+        assert config_key == "custom-config"
+        assert concurrency == 10

tests/integration/test_agent_manager_integration.py

@@ -0,0 +1,716 @@
+"""
+Integration tests for Agent Manager (ReAct Pattern) Service
+
+These tests verify the end-to-end functionality of the Agent Manager service,
+testing the ReAct pattern (Think-Act-Observe), tool coordination, multi-step reasoning,
+and conversation state management.
+Following the TEST_STRATEGY.md approach for integration testing.
+"""
+
+import pytest
+import json
+from unittest.mock import AsyncMock, MagicMock, patch
+
+from trustgraph.agent.react.agent_manager import AgentManager
+from trustgraph.agent.react.tools import KnowledgeQueryImpl, TextCompletionImpl, McpToolImpl
+from trustgraph.agent.react.types import Action, Final, Tool, Argument
+from trustgraph.schema import AgentRequest, AgentResponse, AgentStep, Error
+
+
+@pytest.mark.integration
+class TestAgentManagerIntegration:
+    """Integration tests for Agent Manager ReAct pattern coordination"""
+
+    @pytest.fixture
+    def mock_flow_context(self):
+        """Mock flow context for service coordination"""
+        context = MagicMock()
+        
+        # Mock prompt client
+        prompt_client = AsyncMock()
+        prompt_client.agent_react.return_value = """Thought: I need to search for information about machine learning
+Action: knowledge_query
+Args: {
+    "question": "What is machine learning?"
+}"""
+        
+        # Mock graph RAG client
+        graph_rag_client = AsyncMock()
+        graph_rag_client.rag.return_value = "Machine learning is a subset of AI that enables computers to learn from data."
+        
+        # Mock text completion client
+        text_completion_client = AsyncMock()
+        text_completion_client.question.return_value = "Machine learning involves algorithms that improve through experience."
+        
+        # Mock MCP tool client
+        mcp_tool_client = AsyncMock()
+        mcp_tool_client.invoke.return_value = "Tool execution successful"
+        
+        # Configure context to return appropriate clients
+        def context_router(service_name):
+            if service_name == "prompt-request":
+                return prompt_client
+            elif service_name == "graph-rag-request":
+                return graph_rag_client
+            elif service_name == "prompt-request":
+                return text_completion_client
+            elif service_name == "mcp-tool-request":
+                return mcp_tool_client
+            else:
+                return AsyncMock()
+        
+        context.side_effect = context_router
+        return context
+
+    @pytest.fixture
+    def sample_tools(self):
+        """Sample tool configuration for testing"""
+        return {
+            "knowledge_query": Tool(
+                name="knowledge_query",
+                description="Query the knowledge graph for information",
+                arguments=[
+                    Argument(
+                        name="question",
+                        type="string", 
+                        description="The question to ask the knowledge graph"
+                    )
+                ],
+                implementation=KnowledgeQueryImpl,
+                config={}
+            ),
+            "text_completion": Tool(
+                name="text_completion",
+                description="Generate text completion using LLM",
+                arguments=[
+                    Argument(
+                        name="question",
+                        type="string",
+                        description="The question to ask the LLM"
+                    )
+                ],
+                implementation=TextCompletionImpl,
+                config={}
+            ),
+            "web_search": Tool(
+                name="web_search",
+                description="Search the web for information",
+                arguments=[
+                    Argument(
+                        name="query",
+                        type="string",
+                        description="The search query"
+                    )
+                ],
+                implementation=lambda context: AsyncMock(invoke=AsyncMock(return_value="Web search results")),
+                config={}
+            )
+        }
+
+    @pytest.fixture
+    def agent_manager(self, sample_tools):
+        """Create agent manager with sample tools"""
+        return AgentManager(
+            tools=sample_tools,
+            additional_context="You are a helpful AI assistant with access to knowledge and tools."
+        )
+
+    @pytest.mark.asyncio
+    async def test_agent_manager_reasoning_cycle(self, agent_manager, mock_flow_context):
+        """Test basic reasoning cycle with tool selection"""
+        # Arrange
+        question = "What is machine learning?"
+        history = []
+
+        # Act
+        action = await agent_manager.reason(question, history, mock_flow_context)
+
+        # Assert
+        assert isinstance(action, Action)
+        assert action.thought == "I need to search for information about machine learning"
+        assert action.name == "knowledge_query"
+        assert action.arguments == {"question": "What is machine learning?"}
+        assert action.observation == ""
+
+        # Verify prompt client was called correctly
+        prompt_client = mock_flow_context("prompt-request")
+        prompt_client.agent_react.assert_called_once()
+        
+        # Verify the prompt variables passed to agent_react
+        call_args = prompt_client.agent_react.call_args
+        variables = call_args[0][0]
+        assert variables["question"] == question
+        assert len(variables["tools"]) == 3  # knowledge_query, text_completion, web_search
+        assert variables["context"] == "You are a helpful AI assistant with access to knowledge and tools."
+
+    @pytest.mark.asyncio
+    async def test_agent_manager_final_answer(self, agent_manager, mock_flow_context):
+        """Test agent manager returning final answer"""
+        # Arrange
+        mock_flow_context("prompt-request").agent_react.return_value = """Thought: I have enough information to answer the question
+Final Answer: Machine learning is a field of AI that enables computers to learn from data."""
+        
+        question = "What is machine learning?"
+        history = []
+
+        # Act
+        action = await agent_manager.reason(question, history, mock_flow_context)
+
+        # Assert
+        assert isinstance(action, Final)
+        assert action.thought == "I have enough information to answer the question"
+        assert action.final == "Machine learning is a field of AI that enables computers to learn from data."
+
+    @pytest.mark.asyncio
+    async def test_agent_manager_react_with_tool_execution(self, agent_manager, mock_flow_context):
+        """Test full ReAct cycle with tool execution"""
+        # Arrange
+        question = "What is machine learning?"
+        history = []
+        
+        think_callback = AsyncMock()
+        observe_callback = AsyncMock()
+
+        # Act
+        action = await agent_manager.react(question, history, think_callback, observe_callback, mock_flow_context)
+
+        # Assert
+        assert isinstance(action, Action)
+        assert action.thought == "I need to search for information about machine learning"
+        assert action.name == "knowledge_query"
+        assert action.arguments == {"question": "What is machine learning?"}
+        assert action.observation == "Machine learning is a subset of AI that enables computers to learn from data."
+
+        # Verify callbacks were called
+        think_callback.assert_called_once_with("I need to search for information about machine learning")
+        observe_callback.assert_called_once_with("Machine learning is a subset of AI that enables computers to learn from data.")
+
+        # Verify tool was executed
+        graph_rag_client = mock_flow_context("graph-rag-request")
+        graph_rag_client.rag.assert_called_once_with("What is machine learning?")
+
+    @pytest.mark.asyncio
+    async def test_agent_manager_react_with_final_answer(self, agent_manager, mock_flow_context):
+        """Test ReAct cycle ending with final answer"""
+        # Arrange
+        mock_flow_context("prompt-request").agent_react.return_value = """Thought: I can provide a direct answer
+Final Answer: Machine learning is a branch of artificial intelligence."""
+        
+        question = "What is machine learning?"
+        history = []
+        
+        think_callback = AsyncMock()
+        observe_callback = AsyncMock()
+
+        # Act
+        action = await agent_manager.react(question, history, think_callback, observe_callback, mock_flow_context)
+
+        # Assert
+        assert isinstance(action, Final)
+        assert action.thought == "I can provide a direct answer"
+        assert action.final == "Machine learning is a branch of artificial intelligence."
+
+        # Verify only think callback was called (no observation for final answer)
+        think_callback.assert_called_once_with("I can provide a direct answer")
+        observe_callback.assert_not_called()
+
+    @pytest.mark.asyncio
+    async def test_agent_manager_with_conversation_history(self, agent_manager, mock_flow_context):
+        """Test agent manager with conversation history"""
+        # Arrange
+        question = "Can you tell me more about neural networks?"
+        history = [
+            Action(
+                thought="I need to search for information about machine learning",
+                name="knowledge_query",
+                arguments={"question": "What is machine learning?"},
+                observation="Machine learning is a subset of AI that enables computers to learn from data."
+            )
+        ]
+
+        # Act
+        action = await agent_manager.reason(question, history, mock_flow_context)
+
+        # Assert
+        assert isinstance(action, Action)
+        
+        # Verify history was included in prompt variables
+        prompt_client = mock_flow_context("prompt-request")
+        call_args = prompt_client.agent_react.call_args
+        variables = call_args[0][0]
+        assert len(variables["history"]) == 1
+        assert variables["history"][0]["thought"] == "I need to search for information about machine learning"
+        assert variables["history"][0]["action"] == "knowledge_query"
+        assert variables["history"][0]["observation"] == "Machine learning is a subset of AI that enables computers to learn from data."
+
+    @pytest.mark.asyncio
+    async def test_agent_manager_tool_selection(self, agent_manager, mock_flow_context):
+        """Test agent manager selecting different tools"""
+        # Test different tool selections
+        tool_scenarios = [
+            ("knowledge_query", "graph-rag-request"),
+            ("text_completion", "prompt-request"),
+        ]
+
+        for tool_name, expected_service in tool_scenarios:
+            # Arrange
+            mock_flow_context("prompt-request").agent_react.return_value = f"""Thought: I need to use {tool_name}
+Action: {tool_name}
+Args: {{
+    "question": "test question"
+}}"""
+            
+            think_callback = AsyncMock()
+            observe_callback = AsyncMock()
+
+            # Act
+            action = await agent_manager.react("test question", [], think_callback, observe_callback, mock_flow_context)
+
+            # Assert
+            assert isinstance(action, Action)
+            assert action.name == tool_name
+            
+            # Verify correct service was called
+            if tool_name == "knowledge_query":
+                mock_flow_context("graph-rag-request").rag.assert_called()
+            elif tool_name == "text_completion":
+                mock_flow_context("prompt-request").question.assert_called()
+
+            # Reset mocks for next iteration
+            for service in ["prompt-request", "graph-rag-request", "prompt-request"]:
+                mock_flow_context(service).reset_mock()
+
+    @pytest.mark.asyncio
+    async def test_agent_manager_unknown_tool_error(self, agent_manager, mock_flow_context):
+        """Test agent manager error handling for unknown tool"""
+        # Arrange
+        mock_flow_context("prompt-request").agent_react.return_value = """Thought: I need to use an unknown tool
+Action: unknown_tool
+Args: {
+    "param": "value"
+}"""
+        
+        think_callback = AsyncMock()
+        observe_callback = AsyncMock()
+
+        # Act & Assert
+        with pytest.raises(RuntimeError) as exc_info:
+            await agent_manager.react("test question", [], think_callback, observe_callback, mock_flow_context)
+        
+        assert "No action for unknown_tool!" in str(exc_info.value)
+
+    @pytest.mark.asyncio
+    async def test_agent_manager_tool_execution_error(self, agent_manager, mock_flow_context):
+        """Test agent manager handling tool execution errors"""
+        # Arrange
+        mock_flow_context("graph-rag-request").rag.side_effect = Exception("Tool execution failed")
+        
+        think_callback = AsyncMock()
+        observe_callback = AsyncMock()
+
+        # Act & Assert
+        with pytest.raises(Exception) as exc_info:
+            await agent_manager.react("test question", [], think_callback, observe_callback, mock_flow_context)
+        
+        assert "Tool execution failed" in str(exc_info.value)
+
+    @pytest.mark.asyncio
+    async def test_agent_manager_multiple_tools_coordination(self, agent_manager, mock_flow_context):
+        """Test agent manager coordination with multiple available tools"""
+        # Arrange
+        question = "Find information about AI and summarize it"
+        
+        # Mock multi-step reasoning
+        mock_flow_context("prompt-request").agent_react.return_value = """Thought: I need to search for AI information first
+Action: knowledge_query
+Args: {
+    "question": "What is artificial intelligence?"
+}"""
+
+        # Act
+        action = await agent_manager.reason(question, [], mock_flow_context)
+
+        # Assert
+        assert isinstance(action, Action)
+        assert action.name == "knowledge_query"
+        
+        # Verify tool information was passed to prompt
+        prompt_client = mock_flow_context("prompt-request")
+        call_args = prompt_client.agent_react.call_args
+        variables = call_args[0][0]
+        
+        # Should have all 3 tools available
+        tool_names = [tool["name"] for tool in variables["tools"]]
+        assert "knowledge_query" in tool_names
+        assert "text_completion" in tool_names
+        assert "web_search" in tool_names
+
+    @pytest.mark.asyncio
+    async def test_agent_manager_tool_argument_validation(self, agent_manager, mock_flow_context):
+        """Test agent manager with various tool argument patterns"""
+        # Arrange
+        test_cases = [
+            {
+                "action": "knowledge_query",
+                "arguments": {"question": "What is deep learning?"},
+                "expected_service": "graph-rag-request"
+            },
+            {
+                "action": "text_completion", 
+                "arguments": {"question": "Explain neural networks"},
+                "expected_service": "prompt-request"
+            },
+            {
+                "action": "web_search",
+                "arguments": {"query": "latest AI research"},
+                "expected_service": None  # Custom mock
+            }
+        ]
+
+        for test_case in test_cases:
+            # Arrange
+            # Format arguments as JSON
+            import json
+            args_json = json.dumps(test_case['arguments'], indent=4)
+            mock_flow_context("prompt-request").agent_react.return_value = f"""Thought: Using {test_case['action']}
+Action: {test_case['action']}
+Args: {args_json}"""
+            
+            think_callback = AsyncMock()
+            observe_callback = AsyncMock()
+
+            # Act
+            action = await agent_manager.react("test", [], think_callback, observe_callback, mock_flow_context)
+
+            # Assert
+            assert isinstance(action, Action)
+            assert action.name == test_case['action']
+            assert action.arguments == test_case['arguments']
+            
+            # Reset mocks
+            for service in ["prompt-request", "graph-rag-request", "prompt-request"]:
+                mock_flow_context(service).reset_mock()
+
+    @pytest.mark.asyncio
+    async def test_agent_manager_context_integration(self, agent_manager, mock_flow_context):
+        """Test agent manager integration with additional context"""
+        # Arrange
+        agent_with_context = AgentManager(
+            tools={"knowledge_query": agent_manager.tools["knowledge_query"]},
+            additional_context="You are an expert in machine learning research."
+        )
+        
+        question = "What are the latest developments in AI?"
+
+        # Act
+        action = await agent_with_context.reason(question, [], mock_flow_context)
+
+        # Assert
+        prompt_client = mock_flow_context("prompt-request")
+        call_args = prompt_client.agent_react.call_args
+        variables = call_args[0][0]
+        
+        assert variables["context"] == "You are an expert in machine learning research."
+        assert variables["question"] == question
+
+    @pytest.mark.asyncio
+    async def test_agent_manager_empty_tools(self, mock_flow_context):
+        """Test agent manager with no tools available"""
+        # Arrange
+        agent_no_tools = AgentManager(tools={}, additional_context="")
+        
+        question = "What is machine learning?"
+
+        # Act
+        action = await agent_no_tools.reason(question, [], mock_flow_context)
+
+        # Assert
+        prompt_client = mock_flow_context("prompt-request")
+        call_args = prompt_client.agent_react.call_args
+        variables = call_args[0][0]
+        
+        assert len(variables["tools"]) == 0
+        assert variables["tool_names"] == ""
+
+    @pytest.mark.asyncio
+    async def test_agent_manager_tool_response_processing(self, agent_manager, mock_flow_context):
+        """Test agent manager processing different tool response types"""
+        # Arrange
+        response_scenarios = [
+            "Simple text response",
+            "Multi-line response\nwith several lines\nof information",
+            "Response with special characters: @#$%^&*()_+-=[]{}|;':\",./<>?",
+            "   Response with whitespace   ",
+            ""  # Empty response
+        ]
+
+        for expected_response in response_scenarios:
+            # Set up mock response
+            mock_flow_context("graph-rag-request").rag.return_value = expected_response
+            
+            think_callback = AsyncMock()
+            observe_callback = AsyncMock()
+
+            # Act
+            action = await agent_manager.react("test question", [], think_callback, observe_callback, mock_flow_context)
+
+            # Assert
+            assert isinstance(action, Action)
+            assert action.observation == expected_response.strip()
+            observe_callback.assert_called_with(expected_response.strip())
+            
+            # Reset mocks
+            mock_flow_context("graph-rag-request").reset_mock()
+
+    @pytest.mark.asyncio
+    async def test_agent_manager_malformed_response_handling(self, agent_manager, mock_flow_context):
+        """Test agent manager handling of malformed text responses"""
+        # Test cases with expected error messages
+        test_cases = [
+            # Missing action/final answer
+            {
+                "response": "Thought: I need to do something",
+                "error_contains": "Response has thought but no action or final answer"
+            },
+            # Invalid JSON in Args
+            {
+                "response": """Thought: I need to search
+Action: knowledge_query
+Args: {invalid json}""",
+                "error_contains": "Invalid JSON in Args"
+            },
+            # Empty response
+            {
+                "response": "",
+                "error_contains": "Could not parse response"
+            },
+            # Only whitespace
+            {
+                "response": "   \n\t  ",
+                "error_contains": "Could not parse response"
+            },
+            # Missing Args for action (should create empty args dict)
+            {
+                "response": """Thought: I need to search
+Action: knowledge_query""",
+                "error_contains": None  # This should actually succeed with empty args
+            },
+            # Incomplete JSON
+            {
+                "response": """Thought: I need to search
+Action: knowledge_query
+Args: {
+    "question": "test"
+""",
+                "error_contains": "Invalid JSON in Args"
+            },
+        ]
+        
+        for test_case in test_cases:
+            mock_flow_context("prompt-request").agent_react.return_value = test_case["response"]
+            
+            if test_case["error_contains"]:
+                # Should raise an error
+                with pytest.raises(RuntimeError) as exc_info:
+                    await agent_manager.reason("test question", [], mock_flow_context)
+                
+                assert "Failed to parse agent response" in str(exc_info.value)
+                assert test_case["error_contains"] in str(exc_info.value)
+            else:
+                # Should succeed
+                action = await agent_manager.reason("test question", [], mock_flow_context)
+                assert isinstance(action, Action)
+                assert action.name == "knowledge_query"
+                assert action.arguments == {}
+
+    @pytest.mark.asyncio
+    async def test_agent_manager_text_parsing_edge_cases(self, agent_manager, mock_flow_context):
+        """Test edge cases in text parsing"""
+        # Test response with markdown code blocks
+        mock_flow_context("prompt-request").agent_react.return_value = """```
+Thought: I need to search for information
+Action: knowledge_query
+Args: {
+    "question": "What is AI?"
+}
+```"""
+        
+        action = await agent_manager.reason("test", [], mock_flow_context)
+        assert isinstance(action, Action)
+        assert action.thought == "I need to search for information"
+        assert action.name == "knowledge_query"
+        
+        # Test response with extra whitespace
+        mock_flow_context("prompt-request").agent_react.return_value = """
+
+Thought:    I need to think about this    
+Action:   knowledge_query   
+Args:    {
+    "question": "test"
+}
+
+"""
+        
+        action = await agent_manager.reason("test", [], mock_flow_context)
+        assert isinstance(action, Action)
+        assert action.thought == "I need to think about this"
+        assert action.name == "knowledge_query"
+
+    @pytest.mark.asyncio
+    async def test_agent_manager_multiline_content(self, agent_manager, mock_flow_context):
+        """Test handling of multi-line thoughts and final answers"""
+        # Multi-line thought
+        mock_flow_context("prompt-request").agent_react.return_value = """Thought: I need to consider multiple factors:
+1. The user's question is complex
+2. I should search for comprehensive information
+3. This requires using the knowledge query tool
+Action: knowledge_query
+Args: {
+    "question": "complex query"
+}"""
+        
+        action = await agent_manager.reason("test", [], mock_flow_context)
+        assert isinstance(action, Action)
+        assert "multiple factors" in action.thought
+        assert "knowledge query tool" in action.thought
+        
+        # Multi-line final answer
+        mock_flow_context("prompt-request").agent_react.return_value = """Thought: I have gathered enough information
+Final Answer: Here is a comprehensive answer:
+1. First point about the topic
+2. Second point with details
+3. Final conclusion
+
+This covers all aspects of the question."""
+        
+        action = await agent_manager.reason("test", [], mock_flow_context)
+        assert isinstance(action, Final)
+        assert "First point" in action.final
+        assert "Final conclusion" in action.final
+        assert "all aspects" in action.final
+
+    @pytest.mark.asyncio
+    async def test_agent_manager_json_args_special_characters(self, agent_manager, mock_flow_context):
+        """Test JSON arguments with special characters and edge cases"""
+        # Test with special characters in JSON (properly escaped)
+        mock_flow_context("prompt-request").agent_react.return_value = """Thought: Processing special characters
+Action: knowledge_query
+Args: {
+    "question": "What about \\"quotes\\" and 'apostrophes'?",
+    "context": "Line 1\\nLine 2\\tTabbed",
+    "special": "Symbols: @#$%^&*()_+-=[]{}|;':,.<>?"
+}"""
+        
+        action = await agent_manager.reason("test", [], mock_flow_context)
+        assert isinstance(action, Action)
+        assert action.arguments["question"] == 'What about "quotes" and \'apostrophes\'?'
+        assert action.arguments["context"] == "Line 1\nLine 2\tTabbed"
+        assert "@#$%^&*" in action.arguments["special"]
+        
+        # Test with nested JSON
+        mock_flow_context("prompt-request").agent_react.return_value = """Thought: Complex arguments
+Action: web_search
+Args: {
+    "query": "test",
+    "options": {
+        "limit": 10,
+        "filters": ["recent", "relevant"],
+        "metadata": {
+            "source": "user",
+            "timestamp": "2024-01-01"
+        }
+    }
+}"""
+        
+        action = await agent_manager.reason("test", [], mock_flow_context)
+        assert isinstance(action, Action)
+        assert action.arguments["options"]["limit"] == 10
+        assert "recent" in action.arguments["options"]["filters"]
+        assert action.arguments["options"]["metadata"]["source"] == "user"
+
+    @pytest.mark.asyncio
+    async def test_agent_manager_final_answer_json_format(self, agent_manager, mock_flow_context):
+        """Test final answers that contain JSON-like content"""
+        # Final answer with JSON content
+        mock_flow_context("prompt-request").agent_react.return_value = """Thought: I can provide the data in JSON format
+Final Answer: {
+    "result": "success",
+    "data": {
+        "name": "Machine Learning",
+        "type": "AI Technology",
+        "applications": ["NLP", "Computer Vision", "Robotics"]
+    },
+    "confidence": 0.95
+}"""
+        
+        action = await agent_manager.reason("test", [], mock_flow_context)
+        assert isinstance(action, Final)
+        # The final answer should preserve the JSON structure as a string
+        assert '"result": "success"' in action.final
+        assert '"applications":' in action.final
+
+    @pytest.mark.asyncio
+    @pytest.mark.slow
+    async def test_agent_manager_performance_with_large_history(self, agent_manager, mock_flow_context):
+        """Test agent manager performance with large conversation history"""
+        # Arrange
+        large_history = [
+            Action(
+                thought=f"Step {i} thinking",
+                name="knowledge_query",
+                arguments={"question": f"Question {i}"},
+                observation=f"Observation {i}"
+            )
+            for i in range(50)  # Large history
+        ]
+        
+        question = "Final question"
+
+        # Act
+        import time
+        start_time = time.time()
+        
+        action = await agent_manager.reason(question, large_history, mock_flow_context)
+        
+        end_time = time.time()
+        execution_time = end_time - start_time
+
+        # Assert
+        assert isinstance(action, Action)
+        assert execution_time < 5.0  # Should complete within reasonable time
+        
+        # Verify history was processed correctly
+        prompt_client = mock_flow_context("prompt-request")
+        call_args = prompt_client.agent_react.call_args
+        variables = call_args[0][0]
+        assert len(variables["history"]) == 50
+
+    @pytest.mark.asyncio
+    async def test_agent_manager_json_serialization(self, agent_manager, mock_flow_context):
+        """Test agent manager handling of JSON serialization in prompts"""
+        # Arrange
+        complex_history = [
+            Action(
+                thought="Complex thinking with special characters: \"quotes\", 'apostrophes', and symbols",
+                name="knowledge_query",
+                arguments={"question": "What about JSON serialization?", "complex": {"nested": "value"}},
+                observation="Response with JSON: {\"key\": \"value\"}"
+            )
+        ]
+        
+        question = "Handle JSON properly"
+
+        # Act
+        action = await agent_manager.reason(question, complex_history, mock_flow_context)
+
+        # Assert
+        assert isinstance(action, Action)
+        
+        # Verify JSON was properly serialized in prompt
+        prompt_client = mock_flow_context("prompt-request")
+        call_args = prompt_client.agent_react.call_args
+        variables = call_args[0][0]
+        
+        # Should not raise JSON serialization errors
+        json_str = json.dumps(variables, indent=4)
+        assert len(json_str) > 0

tests/integration/test_cassandra_integration.py

@@ -0,0 +1,411 @@
+"""
+Cassandra integration tests using Podman containers
+
+These tests verify end-to-end functionality of Cassandra storage and query processors
+with real database instances. Compatible with Fedora Linux and Podman.
+
+Uses a single container for all tests to minimize startup time.
+"""
+
+import pytest
+import asyncio
+import time
+from unittest.mock import MagicMock
+
+from .cassandra_test_helper import cassandra_container
+from trustgraph.direct.cassandra import TrustGraph
+from trustgraph.storage.triples.cassandra.write import Processor as StorageProcessor
+from trustgraph.query.triples.cassandra.service import Processor as QueryProcessor
+from trustgraph.schema import Triple, Value, Metadata, Triples, TriplesQueryRequest
+
+
+@pytest.mark.integration
+@pytest.mark.slow
+class TestCassandraIntegration:
+    """Integration tests for Cassandra using a single shared container"""
+
+    @pytest.fixture(scope="class")
+    def cassandra_shared_container(self):
+        """Class-level fixture: single Cassandra container for all tests"""
+        with cassandra_container() as container:
+            yield container
+    
+    def setup_method(self):
+        """Track all created clients for cleanup"""
+        self.clients_to_close = []
+    
+    def teardown_method(self):
+        """Clean up all Cassandra connections"""
+        import gc
+        
+        for client in self.clients_to_close:
+            try:
+                client.close()
+            except Exception:
+                pass  # Ignore errors during cleanup
+        
+        # Clear the list and force garbage collection
+        self.clients_to_close.clear()
+        gc.collect()
+        
+        # Small delay to let threads finish
+        time.sleep(0.5)
+
+    @pytest.mark.asyncio
+    async def test_complete_cassandra_integration(self, cassandra_shared_container):
+        """Complete integration test covering all Cassandra functionality"""
+        container = cassandra_shared_container
+        host, port = container.get_connection_host_port()
+        
+        print("=" * 60)
+        print("RUNNING COMPLETE CASSANDRA INTEGRATION TEST")
+        print("=" * 60)
+        
+        # =====================================================
+        # Test 1: Basic TrustGraph Operations
+        # =====================================================
+        print("\n1. Testing basic TrustGraph operations...")
+        
+        client = TrustGraph(
+            hosts=[host],
+            keyspace="test_basic",
+            table="test_table"
+        )
+        self.clients_to_close.append(client)
+        
+        # Insert test data
+        client.insert("http://example.org/alice", "knows", "http://example.org/bob")
+        client.insert("http://example.org/alice", "age", "25")
+        client.insert("http://example.org/bob", "age", "30")
+        
+        # Test get_all
+        all_results = list(client.get_all(limit=10))
+        assert len(all_results) == 3
+        print(f"✓ Stored and retrieved {len(all_results)} triples")
+        
+        # Test get_s (subject query)
+        alice_results = list(client.get_s("http://example.org/alice", limit=10))
+        assert len(alice_results) == 2
+        alice_predicates = [r.p for r in alice_results]
+        assert "knows" in alice_predicates
+        assert "age" in alice_predicates
+        print("✓ Subject queries working")
+        
+        # Test get_p (predicate query)
+        age_results = list(client.get_p("age", limit=10))
+        assert len(age_results) == 2
+        age_subjects = [r.s for r in age_results]
+        assert "http://example.org/alice" in age_subjects
+        assert "http://example.org/bob" in age_subjects
+        print("✓ Predicate queries working")
+        
+        # =====================================================
+        # Test 2: Storage Processor Integration
+        # =====================================================
+        print("\n2. Testing storage processor integration...")
+        
+        storage_processor = StorageProcessor(
+            taskgroup=MagicMock(),
+            hosts=[host],
+            keyspace="test_storage",
+            table="test_triples"
+        )
+        # Track the TrustGraph instance that will be created
+        self.storage_processor = storage_processor
+        
+        # Create test message
+        storage_message = Triples(
+            metadata=Metadata(user="testuser", collection="testcol"),
+            triples=[
+                Triple(
+                    s=Value(value="http://example.org/person1", is_uri=True),
+                    p=Value(value="http://example.org/name", is_uri=True),
+                    o=Value(value="Alice Smith", is_uri=False)
+                ),
+                Triple(
+                    s=Value(value="http://example.org/person1", is_uri=True),
+                    p=Value(value="http://example.org/age", is_uri=True),
+                    o=Value(value="25", is_uri=False)
+                ),
+                Triple(
+                    s=Value(value="http://example.org/person1", is_uri=True),
+                    p=Value(value="http://example.org/department", is_uri=True),
+                    o=Value(value="Engineering", is_uri=False)
+                )
+            ]
+        )
+        
+        # Store triples via processor
+        await storage_processor.store_triples(storage_message)
+        # Track the created TrustGraph instance
+        if hasattr(storage_processor, 'tg'):
+            self.clients_to_close.append(storage_processor.tg)
+        
+        # Verify data was stored
+        storage_results = list(storage_processor.tg.get_s("http://example.org/person1", limit=10))
+        assert len(storage_results) == 3
+        
+        predicates = [row.p for row in storage_results]
+        objects = [row.o for row in storage_results]
+        
+        assert "http://example.org/name" in predicates
+        assert "http://example.org/age" in predicates
+        assert "http://example.org/department" in predicates
+        assert "Alice Smith" in objects
+        assert "25" in objects
+        assert "Engineering" in objects
+        print("✓ Storage processor working")
+        
+        # =====================================================
+        # Test 3: Query Processor Integration
+        # =====================================================
+        print("\n3. Testing query processor integration...")
+        
+        query_processor = QueryProcessor(
+            taskgroup=MagicMock(),
+            hosts=[host],
+            keyspace="test_query",
+            table="test_triples"
+        )
+        
+        # Use same storage processor for the query keyspace
+        query_storage_processor = StorageProcessor(
+            taskgroup=MagicMock(),
+            hosts=[host],
+            keyspace="test_query",
+            table="test_triples"
+        )
+        
+        # Store test data for querying
+        query_test_message = Triples(
+            metadata=Metadata(user="testuser", collection="testcol"),
+            triples=[
+                Triple(
+                    s=Value(value="http://example.org/alice", is_uri=True),
+                    p=Value(value="http://example.org/knows", is_uri=True),
+                    o=Value(value="http://example.org/bob", is_uri=True)
+                ),
+                Triple(
+                    s=Value(value="http://example.org/alice", is_uri=True),
+                    p=Value(value="http://example.org/age", is_uri=True),
+                    o=Value(value="30", is_uri=False)
+                ),
+                Triple(
+                    s=Value(value="http://example.org/bob", is_uri=True),
+                    p=Value(value="http://example.org/knows", is_uri=True),
+                    o=Value(value="http://example.org/charlie", is_uri=True)
+                )
+            ]
+        )
+        await query_storage_processor.store_triples(query_test_message)
+        
+        # Debug: Check what was actually stored
+        print("Debug: Checking what was stored for Alice...")
+        direct_results = list(query_storage_processor.tg.get_s("http://example.org/alice", limit=10))
+        print(f"Direct TrustGraph results: {len(direct_results)}")
+        for result in direct_results:
+            print(f"  S=http://example.org/alice, P={result.p}, O={result.o}")
+        
+        # Test S query (find all relationships for Alice)
+        s_query = TriplesQueryRequest(
+            s=Value(value="http://example.org/alice", is_uri=True),
+            p=None,  # None for wildcard
+            o=None,  # None for wildcard
+            limit=10,
+            user="testuser",
+            collection="testcol"
+        )
+        s_results = await query_processor.query_triples(s_query)
+        print(f"Query processor results: {len(s_results)}")
+        for result in s_results:
+            print(f"  S={result.s.value}, P={result.p.value}, O={result.o.value}")
+        assert len(s_results) == 2
+        
+        s_predicates = [t.p.value for t in s_results]
+        assert "http://example.org/knows" in s_predicates
+        assert "http://example.org/age" in s_predicates
+        print("✓ Subject queries via processor working")
+        
+        # Test P query (find all "knows" relationships)
+        p_query = TriplesQueryRequest(
+            s=None,  # None for wildcard
+            p=Value(value="http://example.org/knows", is_uri=True),
+            o=None,  # None for wildcard
+            limit=10,
+            user="testuser",
+            collection="testcol"
+        )
+        p_results = await query_processor.query_triples(p_query)
+        print(p_results)
+        assert len(p_results) == 2  # Alice knows Bob, Bob knows Charlie
+        
+        p_subjects = [t.s.value for t in p_results]
+        assert "http://example.org/alice" in p_subjects
+        assert "http://example.org/bob" in p_subjects
+        print("✓ Predicate queries via processor working")
+        
+        # =====================================================
+        # Test 4: Concurrent Operations
+        # =====================================================
+        print("\n4. Testing concurrent operations...")
+        
+        concurrent_processor = StorageProcessor(
+            taskgroup=MagicMock(),
+            hosts=[host],
+            keyspace="test_concurrent",
+            table="test_triples"
+        )
+        
+        # Create multiple coroutines for concurrent storage
+        async def store_person_data(person_id, name, age, department):
+            message = Triples(
+                metadata=Metadata(user="concurrent_test", collection="people"),
+                triples=[
+                    Triple(
+                        s=Value(value=f"http://example.org/{person_id}", is_uri=True),
+                        p=Value(value="http://example.org/name", is_uri=True),
+                        o=Value(value=name, is_uri=False)
+                    ),
+                    Triple(
+                        s=Value(value=f"http://example.org/{person_id}", is_uri=True),
+                        p=Value(value="http://example.org/age", is_uri=True),
+                        o=Value(value=str(age), is_uri=False)
+                    ),
+                    Triple(
+                        s=Value(value=f"http://example.org/{person_id}", is_uri=True),
+                        p=Value(value="http://example.org/department", is_uri=True),
+                        o=Value(value=department, is_uri=False)
+                    )
+                ]
+            )
+            await concurrent_processor.store_triples(message)
+        
+        # Store data for multiple people concurrently
+        people_data = [
+            ("person1", "John Doe", 25, "Engineering"),
+            ("person2", "Jane Smith", 30, "Marketing"),
+            ("person3", "Bob Wilson", 35, "Engineering"),
+            ("person4", "Alice Brown", 28, "Sales"),
+        ]
+        
+        # Run storage operations concurrently
+        store_tasks = [store_person_data(pid, name, age, dept) for pid, name, age, dept in people_data]
+        await asyncio.gather(*store_tasks)
+        # Track the created TrustGraph instance
+        if hasattr(concurrent_processor, 'tg'):
+            self.clients_to_close.append(concurrent_processor.tg)
+        
+        # Verify all names were stored
+        name_results = list(concurrent_processor.tg.get_p("http://example.org/name", limit=10))
+        assert len(name_results) == 4
+        
+        stored_names = [r.o for r in name_results]
+        expected_names = ["John Doe", "Jane Smith", "Bob Wilson", "Alice Brown"]
+        
+        for name in expected_names:
+            assert name in stored_names
+        
+        # Verify department data
+        dept_results = list(concurrent_processor.tg.get_p("http://example.org/department", limit=10))
+        assert len(dept_results) == 4
+        
+        stored_depts = [r.o for r in dept_results]
+        assert "Engineering" in stored_depts
+        assert "Marketing" in stored_depts
+        assert "Sales" in stored_depts
+        print("✓ Concurrent operations working")
+        
+        # =====================================================
+        # Test 5: Complex Queries and Data Integrity
+        # =====================================================
+        print("\n5. Testing complex queries and data integrity...")
+        
+        complex_processor = StorageProcessor(
+            taskgroup=MagicMock(),
+            hosts=[host],
+            keyspace="test_complex",
+            table="test_triples"
+        )
+        
+        # Create a knowledge graph about a company
+        company_graph = Triples(
+            metadata=Metadata(user="integration_test", collection="company"),
+            triples=[
+                # People and their types
+                Triple(
+                    s=Value(value="http://company.org/alice", is_uri=True),
+                    p=Value(value="http://www.w3.org/1999/02/22-rdf-syntax-ns#type", is_uri=True),
+                    o=Value(value="http://company.org/Employee", is_uri=True)
+                ),
+                Triple(
+                    s=Value(value="http://company.org/bob", is_uri=True),
+                    p=Value(value="http://www.w3.org/1999/02/22-rdf-syntax-ns#type", is_uri=True),
+                    o=Value(value="http://company.org/Employee", is_uri=True)
+                ),
+                # Relationships
+                Triple(
+                    s=Value(value="http://company.org/alice", is_uri=True),
+                    p=Value(value="http://company.org/reportsTo", is_uri=True),
+                    o=Value(value="http://company.org/bob", is_uri=True)
+                ),
+                Triple(
+                    s=Value(value="http://company.org/alice", is_uri=True),
+                    p=Value(value="http://company.org/worksIn", is_uri=True),
+                    o=Value(value="http://company.org/engineering", is_uri=True)
+                ),
+                # Personal info
+                Triple(
+                    s=Value(value="http://company.org/alice", is_uri=True),
+                    p=Value(value="http://company.org/fullName", is_uri=True),
+                    o=Value(value="Alice Johnson", is_uri=False)
+                ),
+                Triple(
+                    s=Value(value="http://company.org/alice", is_uri=True),
+                    p=Value(value="http://company.org/email", is_uri=True),
+                    o=Value(value="alice@company.org", is_uri=False)
+                ),
+            ]
+        )
+        
+        # Store the company knowledge graph
+        await complex_processor.store_triples(company_graph)
+        # Track the created TrustGraph instance
+        if hasattr(complex_processor, 'tg'):
+            self.clients_to_close.append(complex_processor.tg)
+        
+        # Verify all Alice's data
+        alice_data = list(complex_processor.tg.get_s("http://company.org/alice", limit=20))
+        assert len(alice_data) == 5
+        
+        alice_predicates = [r.p for r in alice_data]
+        expected_predicates = [
+            "http://www.w3.org/1999/02/22-rdf-syntax-ns#type",
+            "http://company.org/reportsTo",
+            "http://company.org/worksIn",
+            "http://company.org/fullName",
+            "http://company.org/email"
+        ]
+        for pred in expected_predicates:
+            assert pred in alice_predicates
+        
+        # Test type-based queries
+        employee_results = list(complex_processor.tg.get_p("http://www.w3.org/1999/02/22-rdf-syntax-ns#type", limit=10))
+        print(employee_results)
+        assert len(employee_results) == 2
+        
+        employees = [r.s for r in employee_results]
+        assert "http://company.org/alice" in employees
+        assert "http://company.org/bob" in employees
+        print("✓ Complex queries and data integrity working")
+        
+        # =====================================================
+        # Summary
+        # =====================================================
+        print("\n" + "=" * 60)
+        print("✅ ALL CASSANDRA INTEGRATION TESTS PASSED!")
+        print("✅ Basic operations: PASSED")
+        print("✅ Storage processor: PASSED") 
+        print("✅ Query processor: PASSED")
+        print("✅ Concurrent operations: PASSED")
+        print("✅ Complex queries: PASSED")
+        print("=" * 60)

tests/integration/test_document_rag_integration.py

@@ -0,0 +1,312 @@
+"""
+Integration tests for DocumentRAG retrieval system
+
+These tests verify the end-to-end functionality of the DocumentRAG system,
+testing the coordination between embeddings, document retrieval, and prompt services.
+Following the TEST_STRATEGY.md approach for integration testing.
+"""
+
+import pytest
+from unittest.mock import AsyncMock, MagicMock
+from trustgraph.retrieval.document_rag.document_rag import DocumentRag
+
+
+@pytest.mark.integration
+class TestDocumentRagIntegration:
+    """Integration tests for DocumentRAG system coordination"""
+
+    @pytest.fixture
+    def mock_embeddings_client(self):
+        """Mock embeddings client that returns realistic vector embeddings"""
+        client = AsyncMock()
+        client.embed.return_value = [
+            [0.1, 0.2, 0.3, 0.4, 0.5],  # Realistic 5-dimensional embedding
+            [0.6, 0.7, 0.8, 0.9, 1.0]   # Second embedding for testing
+        ]
+        return client
+
+    @pytest.fixture
+    def mock_doc_embeddings_client(self):
+        """Mock document embeddings client that returns realistic document chunks"""
+        client = AsyncMock()
+        client.query.return_value = [
+            "Machine learning is a subset of artificial intelligence that focuses on algorithms that learn from data.",
+            "Deep learning uses neural networks with multiple layers to model complex patterns in data.",
+            "Supervised learning algorithms learn from labeled training data to make predictions on new data."
+        ]
+        return client
+
+    @pytest.fixture
+    def mock_prompt_client(self):
+        """Mock prompt client that generates realistic responses"""
+        client = AsyncMock()
+        client.document_prompt.return_value = (
+            "Machine learning is a field of artificial intelligence that enables computers to learn "
+            "and improve from experience without being explicitly programmed. It uses algorithms "
+            "to find patterns in data and make predictions or decisions."
+        )
+        return client
+
+    @pytest.fixture
+    def document_rag(self, mock_embeddings_client, mock_doc_embeddings_client, mock_prompt_client):
+        """Create DocumentRag instance with mocked dependencies"""
+        return DocumentRag(
+            embeddings_client=mock_embeddings_client,
+            doc_embeddings_client=mock_doc_embeddings_client,
+            prompt_client=mock_prompt_client,
+            verbose=True
+        )
+
+    @pytest.mark.asyncio
+    async def test_document_rag_end_to_end_flow(self, document_rag, mock_embeddings_client, 
+                                                mock_doc_embeddings_client, mock_prompt_client):
+        """Test complete DocumentRAG pipeline from query to response"""
+        # Arrange
+        query = "What is machine learning?"
+        user = "test_user"
+        collection = "ml_knowledge"
+        doc_limit = 10
+
+        # Act
+        result = await document_rag.query(
+            query=query,
+            user=user,
+            collection=collection,
+            doc_limit=doc_limit
+        )
+
+        # Assert - Verify service coordination
+        mock_embeddings_client.embed.assert_called_once_with(query)
+        
+        mock_doc_embeddings_client.query.assert_called_once_with(
+            [[0.1, 0.2, 0.3, 0.4, 0.5], [0.6, 0.7, 0.8, 0.9, 1.0]],
+            limit=doc_limit,
+            user=user,
+            collection=collection
+        )
+        
+        mock_prompt_client.document_prompt.assert_called_once_with(
+            query=query,
+            documents=[
+                "Machine learning is a subset of artificial intelligence that focuses on algorithms that learn from data.",
+                "Deep learning uses neural networks with multiple layers to model complex patterns in data.",
+                "Supervised learning algorithms learn from labeled training data to make predictions on new data."
+            ]
+        )
+
+        # Verify final response
+        assert result is not None
+        assert isinstance(result, str)
+        assert "machine learning" in result.lower()
+        assert "artificial intelligence" in result.lower()
+
+    @pytest.mark.asyncio
+    async def test_document_rag_with_no_documents_found(self, mock_embeddings_client, 
+                                                        mock_doc_embeddings_client, mock_prompt_client):
+        """Test DocumentRAG behavior when no documents are retrieved"""
+        # Arrange
+        mock_doc_embeddings_client.query.return_value = []  # No documents found
+        mock_prompt_client.document_prompt.return_value = "I couldn't find any relevant documents for your query."
+        
+        document_rag = DocumentRag(
+            embeddings_client=mock_embeddings_client,
+            doc_embeddings_client=mock_doc_embeddings_client,
+            prompt_client=mock_prompt_client,
+            verbose=False
+        )
+
+        # Act
+        result = await document_rag.query("very obscure query")
+
+        # Assert
+        mock_embeddings_client.embed.assert_called_once()
+        mock_doc_embeddings_client.query.assert_called_once()
+        mock_prompt_client.document_prompt.assert_called_once_with(
+            query="very obscure query",
+            documents=[]
+        )
+        
+        assert result == "I couldn't find any relevant documents for your query."
+
+    @pytest.mark.asyncio
+    async def test_document_rag_embeddings_service_failure(self, mock_embeddings_client, 
+                                                          mock_doc_embeddings_client, mock_prompt_client):
+        """Test DocumentRAG error handling when embeddings service fails"""
+        # Arrange
+        mock_embeddings_client.embed.side_effect = Exception("Embeddings service unavailable")
+        
+        document_rag = DocumentRag(
+            embeddings_client=mock_embeddings_client,
+            doc_embeddings_client=mock_doc_embeddings_client,
+            prompt_client=mock_prompt_client,
+            verbose=False
+        )
+
+        # Act & Assert
+        with pytest.raises(Exception) as exc_info:
+            await document_rag.query("test query")
+        
+        assert "Embeddings service unavailable" in str(exc_info.value)
+        mock_embeddings_client.embed.assert_called_once()
+        mock_doc_embeddings_client.query.assert_not_called()
+        mock_prompt_client.document_prompt.assert_not_called()
+
+    @pytest.mark.asyncio
+    async def test_document_rag_document_service_failure(self, mock_embeddings_client, 
+                                                        mock_doc_embeddings_client, mock_prompt_client):
+        """Test DocumentRAG error handling when document service fails"""
+        # Arrange
+        mock_doc_embeddings_client.query.side_effect = Exception("Document service connection failed")
+        
+        document_rag = DocumentRag(
+            embeddings_client=mock_embeddings_client,
+            doc_embeddings_client=mock_doc_embeddings_client,
+            prompt_client=mock_prompt_client,
+            verbose=False
+        )
+
+        # Act & Assert
+        with pytest.raises(Exception) as exc_info:
+            await document_rag.query("test query")
+        
+        assert "Document service connection failed" in str(exc_info.value)
+        mock_embeddings_client.embed.assert_called_once()
+        mock_doc_embeddings_client.query.assert_called_once()
+        mock_prompt_client.document_prompt.assert_not_called()
+
+    @pytest.mark.asyncio
+    async def test_document_rag_prompt_service_failure(self, mock_embeddings_client, 
+                                                      mock_doc_embeddings_client, mock_prompt_client):
+        """Test DocumentRAG error handling when prompt service fails"""
+        # Arrange
+        mock_prompt_client.document_prompt.side_effect = Exception("LLM service rate limited")
+        
+        document_rag = DocumentRag(
+            embeddings_client=mock_embeddings_client,
+            doc_embeddings_client=mock_doc_embeddings_client,
+            prompt_client=mock_prompt_client,
+            verbose=False
+        )
+
+        # Act & Assert
+        with pytest.raises(Exception) as exc_info:
+            await document_rag.query("test query")
+        
+        assert "LLM service rate limited" in str(exc_info.value)
+        mock_embeddings_client.embed.assert_called_once()
+        mock_doc_embeddings_client.query.assert_called_once()
+        mock_prompt_client.document_prompt.assert_called_once()
+
+    @pytest.mark.asyncio
+    async def test_document_rag_with_different_document_limits(self, document_rag, 
+                                                              mock_doc_embeddings_client):
+        """Test DocumentRAG with various document limit configurations"""
+        # Test different document limits
+        test_cases = [1, 5, 10, 25, 50]
+        
+        for limit in test_cases:
+            # Reset mock call history
+            mock_doc_embeddings_client.reset_mock()
+            
+            # Act
+            await document_rag.query(f"query with limit {limit}", doc_limit=limit)
+            
+            # Assert
+            mock_doc_embeddings_client.query.assert_called_once()
+            call_args = mock_doc_embeddings_client.query.call_args
+            assert call_args.kwargs['limit'] == limit
+
+    @pytest.mark.asyncio
+    async def test_document_rag_multi_user_isolation(self, document_rag, mock_doc_embeddings_client):
+        """Test DocumentRAG properly isolates queries by user and collection"""
+        # Arrange
+        test_scenarios = [
+            ("user1", "collection1"),
+            ("user2", "collection2"),
+            ("user1", "collection2"),  # Same user, different collection
+            ("user2", "collection1"),  # Different user, same collection
+        ]
+
+        for user, collection in test_scenarios:
+            # Reset mock call history
+            mock_doc_embeddings_client.reset_mock()
+            
+            # Act
+            await document_rag.query(
+                f"query from {user} in {collection}",
+                user=user,
+                collection=collection
+            )
+            
+            # Assert
+            mock_doc_embeddings_client.query.assert_called_once()
+            call_args = mock_doc_embeddings_client.query.call_args
+            assert call_args.kwargs['user'] == user
+            assert call_args.kwargs['collection'] == collection
+
+    @pytest.mark.asyncio
+    async def test_document_rag_verbose_logging(self, mock_embeddings_client, 
+                                               mock_doc_embeddings_client, mock_prompt_client, 
+                                               caplog):
+        """Test DocumentRAG verbose logging functionality"""
+        import logging
+        
+        # Arrange - Configure logging to capture debug messages
+        caplog.set_level(logging.DEBUG)
+        
+        document_rag = DocumentRag(
+            embeddings_client=mock_embeddings_client,
+            doc_embeddings_client=mock_doc_embeddings_client,
+            prompt_client=mock_prompt_client,
+            verbose=True
+        )
+
+        # Act
+        await document_rag.query("test query for verbose logging")
+
+        # Assert - Check for new logging messages
+        log_messages = caplog.text
+        assert "DocumentRag initialized" in log_messages
+        assert "Constructing prompt..." in log_messages
+        assert "Computing embeddings..." in log_messages
+        assert "Getting documents..." in log_messages
+        assert "Invoking LLM..." in log_messages
+        assert "Query processing complete" in log_messages
+
+    @pytest.mark.asyncio
+    @pytest.mark.slow
+    async def test_document_rag_performance_with_large_document_set(self, document_rag, 
+                                                                   mock_doc_embeddings_client):
+        """Test DocumentRAG performance with large document retrieval"""
+        # Arrange - Mock large document set (100 documents)
+        large_doc_set = [f"Document {i} content about machine learning and AI" for i in range(100)]
+        mock_doc_embeddings_client.query.return_value = large_doc_set
+
+        # Act
+        import time
+        start_time = time.time()
+        
+        result = await document_rag.query("performance test query", doc_limit=100)
+        
+        end_time = time.time()
+        execution_time = end_time - start_time
+
+        # Assert
+        assert result is not None
+        assert execution_time < 5.0  # Should complete within 5 seconds
+        mock_doc_embeddings_client.query.assert_called_once()
+        call_args = mock_doc_embeddings_client.query.call_args
+        assert call_args.kwargs['limit'] == 100
+
+    @pytest.mark.asyncio
+    async def test_document_rag_default_parameters(self, document_rag, mock_doc_embeddings_client):
+        """Test DocumentRAG uses correct default parameters"""
+        # Act
+        await document_rag.query("test query with defaults")
+
+        # Assert
+        mock_doc_embeddings_client.query.assert_called_once()
+        call_args = mock_doc_embeddings_client.query.call_args
+        assert call_args.kwargs['user'] == "trustgraph"
+        assert call_args.kwargs['collection'] == "default"
+        assert call_args.kwargs['limit'] == 20

tests/integration/test_kg_extract_store_integration.py

@@ -0,0 +1,642 @@
+"""
+Integration tests for Knowledge Graph Extract → Store Pipeline
+
+These tests verify the end-to-end functionality of the knowledge graph extraction
+and storage pipeline, testing text-to-graph transformation, entity extraction,
+relationship extraction, and graph database storage.
+Following the TEST_STRATEGY.md approach for integration testing.
+"""
+
+import pytest
+import json
+import urllib.parse
+from unittest.mock import AsyncMock, MagicMock, patch
+
+from trustgraph.extract.kg.definitions.extract import Processor as DefinitionsProcessor
+from trustgraph.extract.kg.relationships.extract import Processor as RelationshipsProcessor
+from trustgraph.storage.knowledge.store import Processor as KnowledgeStoreProcessor
+from trustgraph.schema import Chunk, Triple, Triples, Metadata, Value, Error
+from trustgraph.schema import EntityContext, EntityContexts, GraphEmbeddings
+from trustgraph.rdf import TRUSTGRAPH_ENTITIES, DEFINITION, RDF_LABEL, SUBJECT_OF
+
+
+@pytest.mark.integration
+class TestKnowledgeGraphPipelineIntegration:
+    """Integration tests for Knowledge Graph Extract → Store Pipeline"""
+
+    @pytest.fixture
+    def mock_flow_context(self):
+        """Mock flow context for service coordination"""
+        context = MagicMock()
+        
+        # Mock prompt client for definitions extraction
+        prompt_client = AsyncMock()
+        prompt_client.extract_definitions.return_value = [
+            {
+                "entity": "Machine Learning",
+                "definition": "A subset of artificial intelligence that enables computers to learn from data without explicit programming."
+            },
+            {
+                "entity": "Neural Networks",
+                "definition": "Computing systems inspired by biological neural networks that process information."
+            }
+        ]
+        
+        # Mock prompt client for relationships extraction
+        prompt_client.extract_relationships.return_value = [
+            {
+                "subject": "Machine Learning",
+                "predicate": "is_subset_of",
+                "object": "Artificial Intelligence",
+                "object-entity": True
+            },
+            {
+                "subject": "Neural Networks",
+                "predicate": "is_used_in",
+                "object": "Machine Learning",
+                "object-entity": True
+            }
+        ]
+        
+        # Mock producers for output streams
+        triples_producer = AsyncMock()
+        entity_contexts_producer = AsyncMock()
+        
+        # Configure context routing
+        def context_router(service_name):
+            if service_name == "prompt-request":
+                return prompt_client
+            elif service_name == "triples":
+                return triples_producer
+            elif service_name == "entity-contexts":
+                return entity_contexts_producer
+            else:
+                return AsyncMock()
+        
+        context.side_effect = context_router
+        return context
+
+    @pytest.fixture
+    def mock_cassandra_store(self):
+        """Mock Cassandra knowledge table store"""
+        store = AsyncMock()
+        store.add_triples.return_value = None
+        store.add_graph_embeddings.return_value = None
+        return store
+
+    @pytest.fixture
+    def sample_chunk(self):
+        """Sample text chunk for processing"""
+        return Chunk(
+            metadata=Metadata(
+                id="doc-123",
+                user="test_user",
+                collection="test_collection",
+                metadata=[]
+            ),
+            chunk=b"Machine Learning is a subset of Artificial Intelligence. Neural Networks are used in Machine Learning to process complex patterns."
+        )
+
+    @pytest.fixture
+    def sample_definitions_response(self):
+        """Sample definitions extraction response"""
+        return [
+            {
+                "entity": "Machine Learning",
+                "definition": "A subset of artificial intelligence that enables computers to learn from data."
+            },
+            {
+                "entity": "Artificial Intelligence",
+                "definition": "The simulation of human intelligence in machines."
+            },
+            {
+                "entity": "Neural Networks",
+                "definition": "Computing systems inspired by biological neural networks."
+            }
+        ]
+
+    @pytest.fixture
+    def sample_relationships_response(self):
+        """Sample relationships extraction response"""
+        return [
+            {
+                "subject": "Machine Learning",
+                "predicate": "is_subset_of",
+                "object": "Artificial Intelligence",
+                "object-entity": True
+            },
+            {
+                "subject": "Neural Networks",
+                "predicate": "is_used_in",
+                "object": "Machine Learning",
+                "object-entity": True
+            },
+            {
+                "subject": "Machine Learning",
+                "predicate": "processes",
+                "object": "data patterns",
+                "object-entity": False
+            }
+        ]
+
+    @pytest.fixture
+    def definitions_processor(self):
+        """Create definitions processor with minimal configuration"""
+        processor = MagicMock()
+        processor.to_uri = DefinitionsProcessor.to_uri.__get__(processor, DefinitionsProcessor)
+        processor.emit_triples = DefinitionsProcessor.emit_triples.__get__(processor, DefinitionsProcessor)
+        processor.emit_ecs = DefinitionsProcessor.emit_ecs.__get__(processor, DefinitionsProcessor)
+        processor.on_message = DefinitionsProcessor.on_message.__get__(processor, DefinitionsProcessor)
+        return processor
+
+    @pytest.fixture
+    def relationships_processor(self):
+        """Create relationships processor with minimal configuration"""
+        processor = MagicMock()
+        processor.to_uri = RelationshipsProcessor.to_uri.__get__(processor, RelationshipsProcessor)
+        processor.emit_triples = RelationshipsProcessor.emit_triples.__get__(processor, RelationshipsProcessor)
+        processor.on_message = RelationshipsProcessor.on_message.__get__(processor, RelationshipsProcessor)
+        return processor
+
+    @pytest.mark.asyncio
+    async def test_definitions_extraction_pipeline(self, definitions_processor, mock_flow_context, sample_chunk):
+        """Test definitions extraction from text chunk to graph triples"""
+        # Arrange
+        mock_msg = MagicMock()
+        mock_msg.value.return_value = sample_chunk
+        mock_consumer = MagicMock()
+
+        # Act
+        await definitions_processor.on_message(mock_msg, mock_consumer, mock_flow_context)
+
+        # Assert
+        # Verify prompt client was called for definitions extraction
+        prompt_client = mock_flow_context("prompt-request")
+        prompt_client.extract_definitions.assert_called_once()
+        call_args = prompt_client.extract_definitions.call_args
+        assert "Machine Learning" in call_args.kwargs['text']
+        assert "Neural Networks" in call_args.kwargs['text']
+
+        # Verify triples producer was called
+        triples_producer = mock_flow_context("triples")
+        triples_producer.send.assert_called_once()
+        
+        # Verify entity contexts producer was called
+        entity_contexts_producer = mock_flow_context("entity-contexts")
+        entity_contexts_producer.send.assert_called_once()
+
+    @pytest.mark.asyncio
+    async def test_relationships_extraction_pipeline(self, relationships_processor, mock_flow_context, sample_chunk):
+        """Test relationships extraction from text chunk to graph triples"""
+        # Arrange
+        mock_msg = MagicMock()
+        mock_msg.value.return_value = sample_chunk
+        mock_consumer = MagicMock()
+
+        # Act
+        await relationships_processor.on_message(mock_msg, mock_consumer, mock_flow_context)
+
+        # Assert
+        # Verify prompt client was called for relationships extraction
+        prompt_client = mock_flow_context("prompt-request")
+        prompt_client.extract_relationships.assert_called_once()
+        call_args = prompt_client.extract_relationships.call_args
+        assert "Machine Learning" in call_args.kwargs['text']
+
+        # Verify triples producer was called
+        triples_producer = mock_flow_context("triples")
+        triples_producer.send.assert_called_once()
+
+    @pytest.mark.asyncio
+    async def test_uri_generation_consistency(self, definitions_processor, relationships_processor):
+        """Test URI generation consistency between processors"""
+        # Arrange
+        test_entities = [
+            "Machine Learning",
+            "Artificial Intelligence",
+            "Neural Networks",
+            "Deep Learning",
+            "Natural Language Processing"
+        ]
+
+        # Act & Assert
+        for entity in test_entities:
+            def_uri = definitions_processor.to_uri(entity)
+            rel_uri = relationships_processor.to_uri(entity)
+            
+            # URIs should be identical between processors
+            assert def_uri == rel_uri
+            
+            # URI should be properly encoded
+            assert def_uri.startswith(TRUSTGRAPH_ENTITIES)
+            assert " " not in def_uri
+            assert def_uri.endswith(urllib.parse.quote(entity.replace(" ", "-").lower().encode("utf-8")))
+
+    @pytest.mark.asyncio
+    async def test_definitions_triple_generation(self, definitions_processor, sample_definitions_response):
+        """Test triple generation from definitions extraction"""
+        # Arrange
+        metadata = Metadata(
+            id="test-doc",
+            user="test_user",
+            collection="test_collection",
+            metadata=[]
+        )
+
+        # Act
+        triples = []
+        entities = []
+        
+        for defn in sample_definitions_response:
+            s = defn["entity"]
+            o = defn["definition"]
+            
+            if s and o:
+                s_uri = definitions_processor.to_uri(s)
+                s_value = Value(value=str(s_uri), is_uri=True)
+                o_value = Value(value=str(o), is_uri=False)
+                
+                # Generate triples as the processor would
+                triples.append(Triple(
+                    s=s_value,
+                    p=Value(value=RDF_LABEL, is_uri=True),
+                    o=Value(value=s, is_uri=False)
+                ))
+                
+                triples.append(Triple(
+                    s=s_value,
+                    p=Value(value=DEFINITION, is_uri=True),
+                    o=o_value
+                ))
+                
+                entities.append(EntityContext(
+                    entity=s_value,
+                    context=defn["definition"]
+                ))
+
+        # Assert
+        assert len(triples) == 6  # 2 triples per entity * 3 entities
+        assert len(entities) == 3  # 1 entity context per entity
+        
+        # Verify triple structure
+        label_triples = [t for t in triples if t.p.value == RDF_LABEL]
+        definition_triples = [t for t in triples if t.p.value == DEFINITION]
+        
+        assert len(label_triples) == 3
+        assert len(definition_triples) == 3
+        
+        # Verify entity contexts
+        for entity in entities:
+            assert entity.entity.is_uri is True
+            assert entity.entity.value.startswith(TRUSTGRAPH_ENTITIES)
+            assert len(entity.context) > 0
+
+    @pytest.mark.asyncio
+    async def test_relationships_triple_generation(self, relationships_processor, sample_relationships_response):
+        """Test triple generation from relationships extraction"""
+        # Arrange
+        metadata = Metadata(
+            id="test-doc",
+            user="test_user",
+            collection="test_collection",
+            metadata=[]
+        )
+
+        # Act
+        triples = []
+        
+        for rel in sample_relationships_response:
+            s = rel["subject"]
+            p = rel["predicate"]
+            o = rel["object"]
+            
+            if s and p and o:
+                s_uri = relationships_processor.to_uri(s)
+                s_value = Value(value=str(s_uri), is_uri=True)
+                
+                p_uri = relationships_processor.to_uri(p)
+                p_value = Value(value=str(p_uri), is_uri=True)
+                
+                if rel["object-entity"]:
+                    o_uri = relationships_processor.to_uri(o)
+                    o_value = Value(value=str(o_uri), is_uri=True)
+                else:
+                    o_value = Value(value=str(o), is_uri=False)
+                
+                # Main relationship triple
+                triples.append(Triple(s=s_value, p=p_value, o=o_value))
+                
+                # Label triples
+                triples.append(Triple(
+                    s=s_value,
+                    p=Value(value=RDF_LABEL, is_uri=True),
+                    o=Value(value=str(s), is_uri=False)
+                ))
+                
+                triples.append(Triple(
+                    s=p_value,
+                    p=Value(value=RDF_LABEL, is_uri=True),
+                    o=Value(value=str(p), is_uri=False)
+                ))
+                
+                if rel["object-entity"]:
+                    triples.append(Triple(
+                        s=o_value,
+                        p=Value(value=RDF_LABEL, is_uri=True),
+                        o=Value(value=str(o), is_uri=False)
+                    ))
+
+        # Assert
+        assert len(triples) > 0
+        
+        # Verify relationship triples exist
+        relationship_triples = [t for t in triples if t.p.value.endswith("is_subset_of") or t.p.value.endswith("is_used_in")]
+        assert len(relationship_triples) >= 2
+        
+        # Verify label triples
+        label_triples = [t for t in triples if t.p.value == RDF_LABEL]
+        assert len(label_triples) > 0
+
+    @pytest.mark.asyncio
+    async def test_knowledge_store_triples_storage(self, mock_cassandra_store):
+        """Test knowledge store triples storage integration"""
+        # Arrange
+        processor = MagicMock()
+        processor.table_store = mock_cassandra_store
+        processor.on_triples = KnowledgeStoreProcessor.on_triples.__get__(processor, KnowledgeStoreProcessor)
+        
+        sample_triples = Triples(
+            metadata=Metadata(
+                id="test-doc",
+                user="test_user",
+                collection="test_collection",
+                metadata=[]
+            ),
+            triples=[
+                Triple(
+                    s=Value(value="http://trustgraph.ai/e/machine-learning", is_uri=True),
+                    p=Value(value=DEFINITION, is_uri=True),
+                    o=Value(value="A subset of AI", is_uri=False)
+                )
+            ]
+        )
+        
+        mock_msg = MagicMock()
+        mock_msg.value.return_value = sample_triples
+
+        # Act
+        await processor.on_triples(mock_msg, None, None)
+
+        # Assert
+        mock_cassandra_store.add_triples.assert_called_once_with(sample_triples)
+
+    @pytest.mark.asyncio
+    async def test_knowledge_store_graph_embeddings_storage(self, mock_cassandra_store):
+        """Test knowledge store graph embeddings storage integration"""
+        # Arrange
+        processor = MagicMock()
+        processor.table_store = mock_cassandra_store
+        processor.on_graph_embeddings = KnowledgeStoreProcessor.on_graph_embeddings.__get__(processor, KnowledgeStoreProcessor)
+        
+        sample_embeddings = GraphEmbeddings(
+            metadata=Metadata(
+                id="test-doc",
+                user="test_user",
+                collection="test_collection",
+                metadata=[]
+            ),
+            entities=[]
+        )
+        
+        mock_msg = MagicMock()
+        mock_msg.value.return_value = sample_embeddings
+
+        # Act
+        await processor.on_graph_embeddings(mock_msg, None, None)
+
+        # Assert
+        mock_cassandra_store.add_graph_embeddings.assert_called_once_with(sample_embeddings)
+
+    @pytest.mark.asyncio
+    async def test_end_to_end_pipeline_coordination(self, definitions_processor, relationships_processor, 
+                                                   mock_flow_context, sample_chunk):
+        """Test end-to-end pipeline coordination from chunk to storage"""
+        # Arrange
+        mock_msg = MagicMock()
+        mock_msg.value.return_value = sample_chunk
+        mock_consumer = MagicMock()
+
+        # Act - Process through definitions extractor
+        await definitions_processor.on_message(mock_msg, mock_consumer, mock_flow_context)
+        
+        # Act - Process through relationships extractor
+        await relationships_processor.on_message(mock_msg, mock_consumer, mock_flow_context)
+
+        # Assert
+        # Verify both extractors called prompt service
+        prompt_client = mock_flow_context("prompt-request")
+        prompt_client.extract_definitions.assert_called_once()
+        prompt_client.extract_relationships.assert_called_once()
+        
+        # Verify triples were produced from both extractors
+        triples_producer = mock_flow_context("triples")
+        assert triples_producer.send.call_count == 2  # One from each extractor
+        
+        # Verify entity contexts were produced from definitions extractor
+        entity_contexts_producer = mock_flow_context("entity-contexts")
+        entity_contexts_producer.send.assert_called_once()
+
+    @pytest.mark.asyncio
+    async def test_error_handling_in_definitions_extraction(self, definitions_processor, mock_flow_context, sample_chunk):
+        """Test error handling in definitions extraction"""
+        # Arrange
+        mock_flow_context("prompt-request").extract_definitions.side_effect = Exception("Prompt service unavailable")
+        
+        mock_msg = MagicMock()
+        mock_msg.value.return_value = sample_chunk
+        mock_consumer = MagicMock()
+
+        # Act & Assert
+        # Should not raise exception, but should handle it gracefully
+        await definitions_processor.on_message(mock_msg, mock_consumer, mock_flow_context)
+        
+        # Verify prompt was attempted
+        prompt_client = mock_flow_context("prompt-request")
+        prompt_client.extract_definitions.assert_called_once()
+
+    @pytest.mark.asyncio
+    async def test_error_handling_in_relationships_extraction(self, relationships_processor, mock_flow_context, sample_chunk):
+        """Test error handling in relationships extraction"""
+        # Arrange
+        mock_flow_context("prompt-request").extract_relationships.side_effect = Exception("Prompt service unavailable")
+        
+        mock_msg = MagicMock()
+        mock_msg.value.return_value = sample_chunk
+        mock_consumer = MagicMock()
+
+        # Act & Assert
+        # Should not raise exception, but should handle it gracefully
+        await relationships_processor.on_message(mock_msg, mock_consumer, mock_flow_context)
+        
+        # Verify prompt was attempted
+        prompt_client = mock_flow_context("prompt-request")
+        prompt_client.extract_relationships.assert_called_once()
+
+    @pytest.mark.asyncio
+    async def test_empty_extraction_results_handling(self, definitions_processor, mock_flow_context, sample_chunk):
+        """Test handling of empty extraction results"""
+        # Arrange
+        mock_flow_context("prompt-request").extract_definitions.return_value = []
+        
+        mock_msg = MagicMock()
+        mock_msg.value.return_value = sample_chunk
+        mock_consumer = MagicMock()
+
+        # Act
+        await definitions_processor.on_message(mock_msg, mock_consumer, mock_flow_context)
+
+        # Assert
+        # Should still call producers but with empty results
+        triples_producer = mock_flow_context("triples")
+        entity_contexts_producer = mock_flow_context("entity-contexts")
+        
+        triples_producer.send.assert_called_once()
+        entity_contexts_producer.send.assert_called_once()
+
+    @pytest.mark.asyncio
+    async def test_invalid_extraction_format_handling(self, definitions_processor, mock_flow_context, sample_chunk):
+        """Test handling of invalid extraction response format"""
+        # Arrange
+        mock_flow_context("prompt-request").extract_definitions.return_value = "invalid format"  # Should be list
+        
+        mock_msg = MagicMock()
+        mock_msg.value.return_value = sample_chunk
+        mock_consumer = MagicMock()
+
+        # Act & Assert
+        # Should handle invalid format gracefully
+        await definitions_processor.on_message(mock_msg, mock_consumer, mock_flow_context)
+        
+        # Verify prompt was attempted
+        prompt_client = mock_flow_context("prompt-request")
+        prompt_client.extract_definitions.assert_called_once()
+
+    @pytest.mark.asyncio
+    async def test_entity_filtering_and_validation(self, definitions_processor, mock_flow_context):
+        """Test entity filtering and validation in extraction"""
+        # Arrange
+        mock_flow_context("prompt-request").extract_definitions.return_value = [
+            {"entity": "Valid Entity", "definition": "Valid definition"},
+            {"entity": "", "definition": "Empty entity"},  # Should be filtered
+            {"entity": "Valid Entity 2", "definition": ""},  # Should be filtered
+            {"entity": None, "definition": "None entity"},  # Should be filtered
+            {"entity": "Valid Entity 3", "definition": None},  # Should be filtered
+        ]
+        
+        sample_chunk = Chunk(
+            metadata=Metadata(id="test", user="user", collection="collection", metadata=[]),
+            chunk=b"Test chunk"
+        )
+        
+        mock_msg = MagicMock()
+        mock_msg.value.return_value = sample_chunk
+        mock_consumer = MagicMock()
+
+        # Act
+        await definitions_processor.on_message(mock_msg, mock_consumer, mock_flow_context)
+
+        # Assert
+        # Should only process valid entities
+        triples_producer = mock_flow_context("triples")
+        entity_contexts_producer = mock_flow_context("entity-contexts")
+        
+        triples_producer.send.assert_called_once()
+        entity_contexts_producer.send.assert_called_once()
+
+    @pytest.mark.asyncio
+    @pytest.mark.slow
+    async def test_large_batch_processing_performance(self, definitions_processor, relationships_processor, 
+                                                     mock_flow_context):
+        """Test performance with large batch of chunks"""
+        # Arrange
+        large_chunk_batch = [
+            Chunk(
+                metadata=Metadata(id=f"doc-{i}", user="user", collection="collection", metadata=[]),
+                chunk=f"Document {i} contains machine learning and AI content.".encode("utf-8")
+            )
+            for i in range(100)  # Large batch
+        ]
+        
+        mock_consumer = MagicMock()
+
+        # Act
+        import time
+        start_time = time.time()
+        
+        for chunk in large_chunk_batch:
+            mock_msg = MagicMock()
+            mock_msg.value.return_value = chunk
+            
+            # Process through both extractors
+            await definitions_processor.on_message(mock_msg, mock_consumer, mock_flow_context)
+            await relationships_processor.on_message(mock_msg, mock_consumer, mock_flow_context)
+        
+        end_time = time.time()
+        execution_time = end_time - start_time
+
+        # Assert
+        assert execution_time < 30.0  # Should complete within reasonable time
+        
+        # Verify all chunks were processed
+        prompt_client = mock_flow_context("prompt-request")
+        assert prompt_client.extract_definitions.call_count == 100
+        assert prompt_client.extract_relationships.call_count == 100
+
+    @pytest.mark.asyncio
+    async def test_metadata_propagation_through_pipeline(self, definitions_processor, mock_flow_context):
+        """Test metadata propagation through the pipeline"""
+        # Arrange
+        original_metadata = Metadata(
+            id="test-doc-123",
+            user="test_user",
+            collection="test_collection",
+            metadata=[
+                Triple(
+                    s=Value(value="doc:test", is_uri=True),
+                    p=Value(value="dc:title", is_uri=True),
+                    o=Value(value="Test Document", is_uri=False)
+                )
+            ]
+        )
+        
+        sample_chunk = Chunk(
+            metadata=original_metadata,
+            chunk=b"Test content for metadata propagation"
+        )
+        
+        mock_msg = MagicMock()
+        mock_msg.value.return_value = sample_chunk
+        mock_consumer = MagicMock()
+
+        # Act
+        await definitions_processor.on_message(mock_msg, mock_consumer, mock_flow_context)
+
+        # Assert
+        # Verify metadata was propagated to output
+        triples_producer = mock_flow_context("triples")
+        entity_contexts_producer = mock_flow_context("entity-contexts")
+        
+        triples_producer.send.assert_called_once()
+        entity_contexts_producer.send.assert_called_once()
+        
+        # Check that metadata was included in the calls
+        triples_call = triples_producer.send.call_args[0][0]
+        entity_contexts_call = entity_contexts_producer.send.call_args[0][0]
+        
+        assert triples_call.metadata.id == "test-doc-123"
+        assert triples_call.metadata.user == "test_user"
+        assert triples_call.metadata.collection == "test_collection"
+        
+        assert entity_contexts_call.metadata.id == "test-doc-123"
+        assert entity_contexts_call.metadata.user == "test_user"
+        assert entity_contexts_call.metadata.collection == "test_collection"

tests/integration/test_object_extraction_integration.py

@@ -0,0 +1,540 @@
+"""
+Integration tests for Object Extraction Service
+
+These tests verify the end-to-end functionality of the object extraction service,
+testing configuration management, text-to-object transformation, and service coordination.
+Following the TEST_STRATEGY.md approach for integration testing.
+"""
+
+import pytest
+import json
+import asyncio
+from unittest.mock import AsyncMock, MagicMock, patch
+
+from trustgraph.extract.kg.objects.processor import Processor
+from trustgraph.schema import (
+    Chunk, ExtractedObject, Metadata, RowSchema, Field,
+    PromptRequest, PromptResponse
+)
+
+
+@pytest.mark.integration
+class TestObjectExtractionServiceIntegration:
+    """Integration tests for Object Extraction Service"""
+
+    @pytest.fixture
+    def integration_config(self):
+        """Integration test configuration with multiple schemas"""
+        customer_schema = {
+            "name": "customer_records",
+            "description": "Customer information schema",
+            "fields": [
+                {
+                    "name": "customer_id",
+                    "type": "string", 
+                    "primary_key": True,
+                    "required": True,
+                    "indexed": True,
+                    "description": "Unique customer identifier"
+                },
+                {
+                    "name": "name",
+                    "type": "string",
+                    "required": True,
+                    "description": "Customer full name"
+                },
+                {
+                    "name": "email",
+                    "type": "string",
+                    "required": True,
+                    "indexed": True,
+                    "description": "Customer email address"
+                },
+                {
+                    "name": "phone",
+                    "type": "string", 
+                    "required": False,
+                    "description": "Customer phone number"
+                }
+            ]
+        }
+        
+        product_schema = {
+            "name": "product_catalog",
+            "description": "Product catalog schema",
+            "fields": [
+                {
+                    "name": "product_id",
+                    "type": "string",
+                    "primary_key": True,
+                    "required": True,
+                    "indexed": True,
+                    "description": "Unique product identifier"
+                },
+                {
+                    "name": "name",
+                    "type": "string",
+                    "required": True,
+                    "description": "Product name"
+                },
+                {
+                    "name": "price",
+                    "type": "double",
+                    "required": True,
+                    "description": "Product price"
+                },
+                {
+                    "name": "category",
+                    "type": "string",
+                    "required": False,
+                    "enum": ["electronics", "clothing", "books", "home"],
+                    "description": "Product category"
+                }
+            ]
+        }
+        
+        return {
+            "schema": {
+                "customer_records": json.dumps(customer_schema),
+                "product_catalog": json.dumps(product_schema)
+            }
+        }
+
+    @pytest.fixture
+    def mock_integrated_flow(self):
+        """Mock integrated flow context with realistic prompt responses"""
+        context = MagicMock()
+        
+        # Mock prompt client with realistic responses
+        prompt_client = AsyncMock()
+        
+        def mock_extract_objects(schema, text):
+            """Mock extract_objects with schema-aware responses"""
+            # Schema is now a dict (converted by row_schema_translator)
+            schema_name = schema.get("name") if isinstance(schema, dict) else schema.name
+            if schema_name == "customer_records":
+                if "john" in text.lower():
+                    return [
+                        {
+                            "customer_id": "CUST001",
+                            "name": "John Smith", 
+                            "email": "john.smith@email.com",
+                            "phone": "555-0123"
+                        }
+                    ]
+                elif "jane" in text.lower():
+                    return [
+                        {
+                            "customer_id": "CUST002",
+                            "name": "Jane Doe",
+                            "email": "jane.doe@email.com",
+                            "phone": ""
+                        }
+                    ]
+                else:
+                    return []
+            
+            elif schema_name == "product_catalog":
+                if "laptop" in text.lower():
+                    return [
+                        {
+                            "product_id": "PROD001",
+                            "name": "Gaming Laptop",
+                            "price": "1299.99",
+                            "category": "electronics"
+                        }
+                    ]
+                elif "book" in text.lower():
+                    return [
+                        {
+                            "product_id": "PROD002", 
+                            "name": "Python Programming Guide",
+                            "price": "49.99",
+                            "category": "books"
+                        }
+                    ]
+                else:
+                    return []
+            
+            return []
+        
+        prompt_client.extract_objects.side_effect = mock_extract_objects
+        
+        # Mock output producer
+        output_producer = AsyncMock()
+        
+        def context_router(service_name):
+            if service_name == "prompt-request":
+                return prompt_client
+            elif service_name == "output":
+                return output_producer
+            else:
+                return AsyncMock()
+        
+        context.side_effect = context_router
+        return context
+
+    @pytest.mark.asyncio
+    async def test_multi_schema_configuration_integration(self, integration_config):
+        """Test integration with multiple schema configurations"""
+        # Arrange - Create mock processor with actual methods
+        processor = MagicMock()
+        processor.schemas = {}
+        processor.config_key = "schema"
+        processor.on_schema_config = Processor.on_schema_config.__get__(processor, Processor)
+        
+        # Act
+        await processor.on_schema_config(integration_config, version=1)
+        
+        # Assert
+        assert len(processor.schemas) == 2
+        assert "customer_records" in processor.schemas
+        assert "product_catalog" in processor.schemas
+        
+        # Verify customer schema
+        customer_schema = processor.schemas["customer_records"]
+        assert customer_schema.name == "customer_records"
+        assert len(customer_schema.fields) == 4
+        
+        # Verify product schema
+        product_schema = processor.schemas["product_catalog"]
+        assert product_schema.name == "product_catalog"
+        assert len(product_schema.fields) == 4
+        
+        # Check enum field in product schema
+        category_field = next((f for f in product_schema.fields if f.name == "category"), None)
+        assert category_field is not None
+        assert len(category_field.enum_values) == 4
+        assert "electronics" in category_field.enum_values
+
+    @pytest.mark.asyncio
+    async def test_full_service_integration_customer_extraction(self, integration_config, mock_integrated_flow):
+        """Test full service integration for customer data extraction"""
+        # Arrange - Create mock processor with actual methods
+        processor = MagicMock()
+        processor.schemas = {}
+        processor.config_key = "schema"
+        processor.flow = mock_integrated_flow
+        processor.on_schema_config = Processor.on_schema_config.__get__(processor, Processor)
+        processor.on_chunk = Processor.on_chunk.__get__(processor, Processor)
+        processor.extract_objects_for_schema = Processor.extract_objects_for_schema.__get__(processor, Processor)
+        
+        # Import and bind the convert_values_to_strings function
+        from trustgraph.extract.kg.objects.processor import convert_values_to_strings
+        processor.convert_values_to_strings = convert_values_to_strings
+        
+        # Load configuration
+        await processor.on_schema_config(integration_config, version=1)
+        
+        # Create realistic customer data chunk
+        metadata = Metadata(
+            id="customer-doc-001",
+            user="integration_test",
+            collection="test_documents",
+            metadata=[]
+        )
+        
+        chunk_text = """
+        Customer Registration Form
+        
+        Name: John Smith
+        Email: john.smith@email.com
+        Phone: 555-0123
+        Customer ID: CUST001
+        
+        Registration completed successfully.
+        """
+        
+        chunk = Chunk(metadata=metadata, chunk=chunk_text.encode('utf-8'))
+        
+        # Mock message
+        mock_msg = MagicMock()
+        mock_msg.value.return_value = chunk
+        
+        # Act
+        await processor.on_chunk(mock_msg, None, mock_integrated_flow)
+        
+        # Assert
+        output_producer = mock_integrated_flow("output")
+        
+        # Should have calls for both schemas (even if one returns empty)
+        assert output_producer.send.call_count >= 1
+        
+        # Find customer extraction
+        customer_calls = []
+        for call in output_producer.send.call_args_list:
+            extracted_obj = call[0][0]
+            if extracted_obj.schema_name == "customer_records":
+                customer_calls.append(extracted_obj)
+        
+        assert len(customer_calls) == 1
+        customer_obj = customer_calls[0]
+        
+        assert customer_obj.values["customer_id"] == "CUST001"
+        assert customer_obj.values["name"] == "John Smith"
+        assert customer_obj.values["email"] == "john.smith@email.com"
+        assert customer_obj.confidence > 0.5
+
+    @pytest.mark.asyncio
+    async def test_full_service_integration_product_extraction(self, integration_config, mock_integrated_flow):
+        """Test full service integration for product data extraction"""
+        # Arrange - Create mock processor with actual methods
+        processor = MagicMock()
+        processor.schemas = {}
+        processor.config_key = "schema"
+        processor.flow = mock_integrated_flow
+        processor.on_schema_config = Processor.on_schema_config.__get__(processor, Processor)
+        processor.on_chunk = Processor.on_chunk.__get__(processor, Processor)
+        processor.extract_objects_for_schema = Processor.extract_objects_for_schema.__get__(processor, Processor)
+        
+        # Import and bind the convert_values_to_strings function
+        from trustgraph.extract.kg.objects.processor import convert_values_to_strings
+        processor.convert_values_to_strings = convert_values_to_strings
+        
+        # Load configuration
+        await processor.on_schema_config(integration_config, version=1)
+        
+        # Create realistic product data chunk
+        metadata = Metadata(
+            id="product-doc-001",
+            user="integration_test",
+            collection="test_documents",
+            metadata=[]
+        )
+        
+        chunk_text = """
+        Product Specification Sheet
+        
+        Product Name: Gaming Laptop
+        Product ID: PROD001
+        Price: $1,299.99
+        Category: Electronics
+        
+        High-performance gaming laptop with latest specifications.
+        """
+        
+        chunk = Chunk(metadata=metadata, chunk=chunk_text.encode('utf-8'))
+        
+        # Mock message
+        mock_msg = MagicMock()
+        mock_msg.value.return_value = chunk
+        
+        # Act
+        await processor.on_chunk(mock_msg, None, mock_integrated_flow)
+        
+        # Assert
+        output_producer = mock_integrated_flow("output")
+        
+        # Find product extraction
+        product_calls = []
+        for call in output_producer.send.call_args_list:
+            extracted_obj = call[0][0]
+            if extracted_obj.schema_name == "product_catalog":
+                product_calls.append(extracted_obj)
+        
+        assert len(product_calls) == 1
+        product_obj = product_calls[0]
+        
+        assert product_obj.values["product_id"] == "PROD001"
+        assert product_obj.values["name"] == "Gaming Laptop"
+        assert product_obj.values["price"] == "1299.99"
+        assert product_obj.values["category"] == "electronics"
+
+    @pytest.mark.asyncio
+    async def test_concurrent_extraction_integration(self, integration_config, mock_integrated_flow):
+        """Test concurrent processing of multiple chunks"""
+        # Arrange - Create mock processor with actual methods
+        processor = MagicMock()
+        processor.schemas = {}
+        processor.config_key = "schema"
+        processor.flow = mock_integrated_flow
+        processor.on_schema_config = Processor.on_schema_config.__get__(processor, Processor)
+        processor.on_chunk = Processor.on_chunk.__get__(processor, Processor)
+        processor.extract_objects_for_schema = Processor.extract_objects_for_schema.__get__(processor, Processor)
+        
+        # Import and bind the convert_values_to_strings function
+        from trustgraph.extract.kg.objects.processor import convert_values_to_strings
+        processor.convert_values_to_strings = convert_values_to_strings
+        
+        # Load configuration
+        await processor.on_schema_config(integration_config, version=1)
+        
+        # Create multiple test chunks
+        chunks_data = [
+            ("customer-chunk-1", "Customer: John Smith, email: john.smith@email.com, ID: CUST001"),
+            ("customer-chunk-2", "Customer: Jane Doe, email: jane.doe@email.com, ID: CUST002"),
+            ("product-chunk-1", "Product: Gaming Laptop, ID: PROD001, Price: $1299.99, Category: electronics"),
+            ("product-chunk-2", "Product: Python Programming Guide, ID: PROD002, Price: $49.99, Category: books")
+        ]
+        
+        chunks = []
+        for chunk_id, text in chunks_data:
+            metadata = Metadata(
+                id=chunk_id,
+                user="concurrent_test",
+                collection="test_collection",
+                metadata=[]
+            )
+            chunk = Chunk(metadata=metadata, chunk=text.encode('utf-8'))
+            chunks.append(chunk)
+        
+        # Act - Process chunks concurrently
+        tasks = []
+        for chunk in chunks:
+            mock_msg = MagicMock()
+            mock_msg.value.return_value = chunk
+            task = processor.on_chunk(mock_msg, None, mock_integrated_flow)
+            tasks.append(task)
+        
+        await asyncio.gather(*tasks)
+        
+        # Assert
+        output_producer = mock_integrated_flow("output")
+        
+        # Should have processed all chunks (some may produce objects, some may not)
+        assert output_producer.send.call_count >= 2  # At least customer and product extractions
+        
+        # Verify we got both types of objects
+        extracted_objects = []
+        for call in output_producer.send.call_args_list:
+            extracted_objects.append(call[0][0])
+        
+        customer_objects = [obj for obj in extracted_objects if obj.schema_name == "customer_records"]
+        product_objects = [obj for obj in extracted_objects if obj.schema_name == "product_catalog"]
+        
+        assert len(customer_objects) >= 1
+        assert len(product_objects) >= 1
+
+    @pytest.mark.asyncio
+    async def test_configuration_reload_integration(self, integration_config, mock_integrated_flow):
+        """Test configuration reload during service operation"""
+        # Arrange - Create mock processor with actual methods
+        processor = MagicMock()
+        processor.schemas = {}
+        processor.config_key = "schema"
+        processor.flow = mock_integrated_flow
+        processor.on_schema_config = Processor.on_schema_config.__get__(processor, Processor)
+        
+        # Load initial configuration (only customer schema)
+        initial_config = {
+            "schema": {
+                "customer_records": integration_config["schema"]["customer_records"]
+            }
+        }
+        await processor.on_schema_config(initial_config, version=1)
+        
+        assert len(processor.schemas) == 1
+        assert "customer_records" in processor.schemas
+        assert "product_catalog" not in processor.schemas
+        
+        # Act - Reload with full configuration
+        await processor.on_schema_config(integration_config, version=2)
+        
+        # Assert
+        assert len(processor.schemas) == 2
+        assert "customer_records" in processor.schemas
+        assert "product_catalog" in processor.schemas
+
+    @pytest.mark.asyncio
+    async def test_error_resilience_integration(self, integration_config):
+        """Test service resilience to various error conditions"""
+        # Arrange - Create mock processor with actual methods
+        processor = MagicMock()
+        processor.schemas = {}
+        processor.config_key = "schema"
+        processor.on_schema_config = Processor.on_schema_config.__get__(processor, Processor)
+        processor.on_chunk = Processor.on_chunk.__get__(processor, Processor)
+        processor.extract_objects_for_schema = Processor.extract_objects_for_schema.__get__(processor, Processor)
+        
+        # Import and bind the convert_values_to_strings function
+        from trustgraph.extract.kg.objects.processor import convert_values_to_strings
+        processor.convert_values_to_strings = convert_values_to_strings
+        
+        # Mock flow with failing prompt service
+        failing_flow = MagicMock()
+        failing_prompt = AsyncMock()
+        failing_prompt.extract_rows.side_effect = Exception("Prompt service unavailable")
+        
+        def failing_context_router(service_name):
+            if service_name == "prompt-request":
+                return failing_prompt
+            elif service_name == "output":
+                return AsyncMock()
+            else:
+                return AsyncMock()
+        
+        failing_flow.side_effect = failing_context_router
+        processor.flow = failing_flow
+        
+        # Load configuration
+        await processor.on_schema_config(integration_config, version=1)
+        
+        # Create test chunk
+        metadata = Metadata(id="error-test", user="test", collection="test", metadata=[])
+        chunk = Chunk(metadata=metadata, chunk=b"Some text that will fail to process")
+        
+        mock_msg = MagicMock()
+        mock_msg.value.return_value = chunk
+        
+        # Act & Assert - Should not raise exception
+        try:
+            await processor.on_chunk(mock_msg, None, failing_flow)
+            # Should complete without throwing exception
+        except Exception as e:
+            pytest.fail(f"Service should handle errors gracefully, but raised: {e}")
+
+    @pytest.mark.asyncio
+    async def test_metadata_propagation_integration(self, integration_config, mock_integrated_flow):
+        """Test proper metadata propagation through extraction pipeline"""
+        # Arrange - Create mock processor with actual methods
+        processor = MagicMock()
+        processor.schemas = {}
+        processor.config_key = "schema"
+        processor.flow = mock_integrated_flow
+        processor.on_schema_config = Processor.on_schema_config.__get__(processor, Processor)
+        processor.on_chunk = Processor.on_chunk.__get__(processor, Processor)
+        processor.extract_objects_for_schema = Processor.extract_objects_for_schema.__get__(processor, Processor)
+        
+        # Import and bind the convert_values_to_strings function
+        from trustgraph.extract.kg.objects.processor import convert_values_to_strings
+        processor.convert_values_to_strings = convert_values_to_strings
+        
+        # Load configuration
+        await processor.on_schema_config(integration_config, version=1)
+        
+        # Create chunk with rich metadata
+        original_metadata = Metadata(
+            id="metadata-test-chunk",
+            user="test_user",
+            collection="test_collection",
+            metadata=[]  # Could include source document metadata
+        )
+        
+        chunk = Chunk(
+            metadata=original_metadata,
+            chunk=b"Customer: John Smith, ID: CUST001, email: john.smith@email.com"
+        )
+        
+        mock_msg = MagicMock()
+        mock_msg.value.return_value = chunk
+        
+        # Act
+        await processor.on_chunk(mock_msg, None, mock_integrated_flow)
+        
+        # Assert
+        output_producer = mock_integrated_flow("output")
+        
+        # Find extracted object
+        extracted_obj = None
+        for call in output_producer.send.call_args_list:
+            obj = call[0][0]
+            if obj.schema_name == "customer_records":
+                extracted_obj = obj
+                break
+        
+        assert extracted_obj is not None
+        
+        # Verify metadata propagation
+        assert extracted_obj.metadata.user == "test_user"
+        assert extracted_obj.metadata.collection == "test_collection"
+        assert "metadata-test-chunk" in extracted_obj.metadata.id  # Should include source reference

tests/integration/test_objects_cassandra_integration.py

@@ -0,0 +1,384 @@
+"""
+Integration tests for Cassandra Object Storage
+
+These tests verify the end-to-end functionality of storing ExtractedObjects
+in Cassandra, including table creation, data insertion, and error handling.
+"""
+
+import pytest
+from unittest.mock import MagicMock, AsyncMock, patch
+import json
+import uuid
+
+from trustgraph.storage.objects.cassandra.write import Processor
+from trustgraph.schema import ExtractedObject, Metadata, RowSchema, Field
+
+
+@pytest.mark.integration
+class TestObjectsCassandraIntegration:
+    """Integration tests for Cassandra object storage"""
+
+    @pytest.fixture
+    def mock_cassandra_session(self):
+        """Mock Cassandra session for integration tests"""
+        session = MagicMock()
+        session.execute = MagicMock()
+        return session
+
+    @pytest.fixture
+    def mock_cassandra_cluster(self, mock_cassandra_session):
+        """Mock Cassandra cluster"""
+        cluster = MagicMock()
+        cluster.connect.return_value = mock_cassandra_session
+        cluster.shutdown = MagicMock()
+        return cluster
+
+    @pytest.fixture
+    def processor_with_mocks(self, mock_cassandra_cluster, mock_cassandra_session):
+        """Create processor with mocked Cassandra dependencies"""
+        processor = MagicMock()
+        processor.graph_host = "localhost"
+        processor.graph_username = None
+        processor.graph_password = None
+        processor.config_key = "schema"
+        processor.schemas = {}
+        processor.known_keyspaces = set()
+        processor.known_tables = {}
+        processor.cluster = None
+        processor.session = None
+        
+        # Bind actual methods
+        processor.connect_cassandra = Processor.connect_cassandra.__get__(processor, Processor)
+        processor.ensure_keyspace = Processor.ensure_keyspace.__get__(processor, Processor)
+        processor.ensure_table = Processor.ensure_table.__get__(processor, Processor)
+        processor.sanitize_name = Processor.sanitize_name.__get__(processor, Processor)
+        processor.sanitize_table = Processor.sanitize_table.__get__(processor, Processor)
+        processor.get_cassandra_type = Processor.get_cassandra_type.__get__(processor, Processor)
+        processor.convert_value = Processor.convert_value.__get__(processor, Processor)
+        processor.on_schema_config = Processor.on_schema_config.__get__(processor, Processor)
+        processor.on_object = Processor.on_object.__get__(processor, Processor)
+        
+        return processor, mock_cassandra_cluster, mock_cassandra_session
+
+    @pytest.mark.asyncio
+    async def test_end_to_end_object_storage(self, processor_with_mocks):
+        """Test complete flow from schema config to object storage"""
+        processor, mock_cluster, mock_session = processor_with_mocks
+        
+        # Mock Cluster creation
+        with patch('trustgraph.storage.objects.cassandra.write.Cluster', return_value=mock_cluster):
+            # Step 1: Configure schema
+            config = {
+                "schema": {
+                    "customer_records": json.dumps({
+                        "name": "customer_records",
+                        "description": "Customer information",
+                        "fields": [
+                            {"name": "customer_id", "type": "string", "primary_key": True},
+                            {"name": "name", "type": "string", "required": True},
+                            {"name": "email", "type": "string", "indexed": True},
+                            {"name": "age", "type": "integer"}
+                        ]
+                    })
+                }
+            }
+            
+            await processor.on_schema_config(config, version=1)
+            assert "customer_records" in processor.schemas
+            
+            # Step 2: Process an ExtractedObject
+            test_obj = ExtractedObject(
+                metadata=Metadata(
+                    id="doc-001",
+                    user="test_user",
+                    collection="import_2024",
+                    metadata=[]
+                ),
+                schema_name="customer_records",
+                values={
+                    "customer_id": "CUST001",
+                    "name": "John Doe",
+                    "email": "john@example.com",
+                    "age": "30"
+                },
+                confidence=0.95,
+                source_span="Customer: John Doe..."
+            )
+            
+            msg = MagicMock()
+            msg.value.return_value = test_obj
+            
+            await processor.on_object(msg, None, None)
+            
+            # Verify Cassandra interactions
+            assert mock_cluster.connect.called
+            
+            # Verify keyspace creation
+            keyspace_calls = [call for call in mock_session.execute.call_args_list 
+                            if "CREATE KEYSPACE" in str(call)]
+            assert len(keyspace_calls) == 1
+            assert "test_user" in str(keyspace_calls[0])
+            
+            # Verify table creation
+            table_calls = [call for call in mock_session.execute.call_args_list 
+                         if "CREATE TABLE" in str(call)]
+            assert len(table_calls) == 1
+            assert "o_customer_records" in str(table_calls[0])  # Table gets o_ prefix
+            assert "collection text" in str(table_calls[0])
+            assert "PRIMARY KEY ((collection, customer_id))" in str(table_calls[0])
+            
+            # Verify index creation
+            index_calls = [call for call in mock_session.execute.call_args_list 
+                         if "CREATE INDEX" in str(call)]
+            assert len(index_calls) == 1
+            assert "email" in str(index_calls[0])
+            
+            # Verify data insertion
+            insert_calls = [call for call in mock_session.execute.call_args_list 
+                          if "INSERT INTO" in str(call)]
+            assert len(insert_calls) == 1
+            insert_call = insert_calls[0]
+            assert "test_user.o_customer_records" in str(insert_call)  # Table gets o_ prefix
+            
+            # Check inserted values
+            values = insert_call[0][1]
+            assert "import_2024" in values  # collection
+            assert "CUST001" in values      # customer_id
+            assert "John Doe" in values     # name
+            assert "john@example.com" in values  # email
+            assert 30 in values             # age (converted to int)
+
+    @pytest.mark.asyncio
+    async def test_multi_schema_handling(self, processor_with_mocks):
+        """Test handling multiple schemas and objects"""
+        processor, mock_cluster, mock_session = processor_with_mocks
+        
+        with patch('trustgraph.storage.objects.cassandra.write.Cluster', return_value=mock_cluster):
+            # Configure multiple schemas
+            config = {
+                "schema": {
+                    "products": json.dumps({
+                        "name": "products",
+                        "fields": [
+                            {"name": "product_id", "type": "string", "primary_key": True},
+                            {"name": "name", "type": "string"},
+                            {"name": "price", "type": "float"}
+                        ]
+                    }),
+                    "orders": json.dumps({
+                        "name": "orders",
+                        "fields": [
+                            {"name": "order_id", "type": "string", "primary_key": True},
+                            {"name": "customer_id", "type": "string"},
+                            {"name": "total", "type": "float"}
+                        ]
+                    })
+                }
+            }
+            
+            await processor.on_schema_config(config, version=1)
+            assert len(processor.schemas) == 2
+            
+            # Process objects for different schemas
+            product_obj = ExtractedObject(
+                metadata=Metadata(id="p1", user="shop", collection="catalog", metadata=[]),
+                schema_name="products",
+                values={"product_id": "P001", "name": "Widget", "price": "19.99"},
+                confidence=0.9,
+                source_span="Product..."
+            )
+            
+            order_obj = ExtractedObject(
+                metadata=Metadata(id="o1", user="shop", collection="sales", metadata=[]),
+                schema_name="orders",
+                values={"order_id": "O001", "customer_id": "C001", "total": "59.97"},
+                confidence=0.85,
+                source_span="Order..."
+            )
+            
+            # Process both objects
+            for obj in [product_obj, order_obj]:
+                msg = MagicMock()
+                msg.value.return_value = obj
+                await processor.on_object(msg, None, None)
+            
+            # Verify separate tables were created
+            table_calls = [call for call in mock_session.execute.call_args_list 
+                         if "CREATE TABLE" in str(call)]
+            assert len(table_calls) == 2
+            assert any("o_products" in str(call) for call in table_calls)  # Tables get o_ prefix
+            assert any("o_orders" in str(call) for call in table_calls)    # Tables get o_ prefix
+
+    @pytest.mark.asyncio
+    async def test_missing_required_fields(self, processor_with_mocks):
+        """Test handling of objects with missing required fields"""
+        processor, mock_cluster, mock_session = processor_with_mocks
+        
+        with patch('trustgraph.storage.objects.cassandra.write.Cluster', return_value=mock_cluster):
+            # Configure schema with required field
+            processor.schemas["test_schema"] = RowSchema(
+                name="test_schema",
+                description="Test",
+                fields=[
+                    Field(name="id", type="string", size=50, primary=True, required=True),
+                    Field(name="required_field", type="string", size=100, required=True)
+                ]
+            )
+            
+            # Create object missing required field
+            test_obj = ExtractedObject(
+                metadata=Metadata(id="t1", user="test", collection="test", metadata=[]),
+                schema_name="test_schema",
+                values={"id": "123"},  # missing required_field
+                confidence=0.8,
+                source_span="Test"
+            )
+            
+            msg = MagicMock()
+            msg.value.return_value = test_obj
+            
+            # Should still process (Cassandra doesn't enforce NOT NULL)
+            await processor.on_object(msg, None, None)
+            
+            # Verify insert was attempted
+            insert_calls = [call for call in mock_session.execute.call_args_list 
+                          if "INSERT INTO" in str(call)]
+            assert len(insert_calls) == 1
+
+    @pytest.mark.asyncio
+    async def test_schema_without_primary_key(self, processor_with_mocks):
+        """Test handling schemas without defined primary keys"""
+        processor, mock_cluster, mock_session = processor_with_mocks
+        
+        with patch('trustgraph.storage.objects.cassandra.write.Cluster', return_value=mock_cluster):
+            # Configure schema without primary key
+            processor.schemas["events"] = RowSchema(
+                name="events",
+                description="Event log",
+                fields=[
+                    Field(name="event_type", type="string", size=50),
+                    Field(name="timestamp", type="timestamp", size=0)
+                ]
+            )
+            
+            # Process object
+            test_obj = ExtractedObject(
+                metadata=Metadata(id="e1", user="logger", collection="app_events", metadata=[]),
+                schema_name="events",
+                values={"event_type": "login", "timestamp": "2024-01-01T10:00:00Z"},
+                confidence=1.0,
+                source_span="Event"
+            )
+            
+            msg = MagicMock()
+            msg.value.return_value = test_obj
+            
+            await processor.on_object(msg, None, None)
+            
+            # Verify synthetic_id was added
+            table_calls = [call for call in mock_session.execute.call_args_list 
+                         if "CREATE TABLE" in str(call)]
+            assert len(table_calls) == 1
+            assert "synthetic_id uuid" in str(table_calls[0])
+            
+            # Verify insert includes UUID
+            insert_calls = [call for call in mock_session.execute.call_args_list 
+                          if "INSERT INTO" in str(call)]
+            assert len(insert_calls) == 1
+            values = insert_calls[0][0][1]
+            # Check that a UUID was generated (will be in values list)
+            uuid_found = any(isinstance(v, uuid.UUID) for v in values)
+            assert uuid_found
+
+    @pytest.mark.asyncio
+    async def test_authentication_handling(self, processor_with_mocks):
+        """Test Cassandra authentication"""
+        processor, mock_cluster, mock_session = processor_with_mocks
+        processor.graph_username = "cassandra_user"
+        processor.graph_password = "cassandra_pass"
+        
+        with patch('trustgraph.storage.objects.cassandra.write.Cluster') as mock_cluster_class:
+            with patch('trustgraph.storage.objects.cassandra.write.PlainTextAuthProvider') as mock_auth:
+                mock_cluster_class.return_value = mock_cluster
+                
+                # Trigger connection
+                processor.connect_cassandra()
+                
+                # Verify authentication was configured
+                mock_auth.assert_called_once_with(
+                    username="cassandra_user",
+                    password="cassandra_pass"
+                )
+                mock_cluster_class.assert_called_once()
+                call_kwargs = mock_cluster_class.call_args[1]
+                assert 'auth_provider' in call_kwargs
+
+    @pytest.mark.asyncio 
+    async def test_error_handling_during_insert(self, processor_with_mocks):
+        """Test error handling when insertion fails"""
+        processor, mock_cluster, mock_session = processor_with_mocks
+        
+        with patch('trustgraph.storage.objects.cassandra.write.Cluster', return_value=mock_cluster):
+            processor.schemas["test"] = RowSchema(
+                name="test",
+                fields=[Field(name="id", type="string", size=50, primary=True)]
+            )
+            
+            # Make insert fail
+            mock_session.execute.side_effect = [
+                None,  # keyspace creation succeeds
+                None,  # table creation succeeds
+                Exception("Connection timeout")  # insert fails
+            ]
+            
+            test_obj = ExtractedObject(
+                metadata=Metadata(id="t1", user="test", collection="test", metadata=[]),
+                schema_name="test",
+                values={"id": "123"},
+                confidence=0.9,
+                source_span="Test"
+            )
+            
+            msg = MagicMock()
+            msg.value.return_value = test_obj
+            
+            # Should raise the exception
+            with pytest.raises(Exception, match="Connection timeout"):
+                await processor.on_object(msg, None, None)
+
+    @pytest.mark.asyncio
+    async def test_collection_partitioning(self, processor_with_mocks):
+        """Test that objects are properly partitioned by collection"""
+        processor, mock_cluster, mock_session = processor_with_mocks
+        
+        with patch('trustgraph.storage.objects.cassandra.write.Cluster', return_value=mock_cluster):
+            processor.schemas["data"] = RowSchema(
+                name="data",
+                fields=[Field(name="id", type="string", size=50, primary=True)]
+            )
+            
+            # Process objects from different collections
+            collections = ["import_jan", "import_feb", "import_mar"]
+            
+            for coll in collections:
+                obj = ExtractedObject(
+                    metadata=Metadata(id=f"{coll}-1", user="analytics", collection=coll, metadata=[]),
+                    schema_name="data",
+                    values={"id": f"ID-{coll}"},
+                    confidence=0.9,
+                    source_span="Data"
+                )
+                
+                msg = MagicMock()
+                msg.value.return_value = obj
+                await processor.on_object(msg, None, None)
+            
+            # Verify all inserts include collection in values
+            insert_calls = [call for call in mock_session.execute.call_args_list 
+                          if "INSERT INTO" in str(call)]
+            assert len(insert_calls) == 3
+            
+            # Check each insert has the correct collection
+            for i, call in enumerate(insert_calls):
+                values = call[0][1]
+                assert collections[i] in values

tests/integration/test_template_service_integration.py

@@ -0,0 +1,205 @@
+"""
+Simplified integration tests for Template Service
+
+These tests verify the basic functionality of the template service
+without the full message queue infrastructure.
+"""
+
+import pytest
+import json
+from unittest.mock import AsyncMock, MagicMock
+
+from trustgraph.schema import PromptRequest, PromptResponse
+from trustgraph.template.prompt_manager import PromptManager
+
+
+@pytest.mark.integration 
+class TestTemplateServiceSimple:
+    """Simplified integration tests for Template Service components"""
+
+    @pytest.fixture
+    def sample_config(self):
+        """Sample configuration for testing"""
+        return {
+            "system": json.dumps("You are a helpful assistant."),
+            "template-index": json.dumps(["greeting", "json_test"]),
+            "template.greeting": json.dumps({
+                "prompt": "Hello {{ name }}, welcome to {{ system_name }}!",
+                "response-type": "text"
+            }),
+            "template.json_test": json.dumps({
+                "prompt": "Generate profile for {{ username }}",
+                "response-type": "json",
+                "schema": {
+                    "type": "object",
+                    "properties": {
+                        "name": {"type": "string"},
+                        "role": {"type": "string"}
+                    },
+                    "required": ["name", "role"]
+                }
+            })
+        }
+
+    @pytest.fixture
+    def prompt_manager(self, sample_config):
+        """Create a configured PromptManager"""
+        pm = PromptManager()
+        pm.load_config(sample_config)
+        pm.terms["system_name"] = "TrustGraph"
+        return pm
+
+    @pytest.mark.asyncio
+    async def test_prompt_manager_text_invocation(self, prompt_manager):
+        """Test PromptManager text response invocation"""
+        # Mock LLM function
+        async def mock_llm(system, prompt):
+            assert system == "You are a helpful assistant."
+            assert "Hello Alice, welcome to TrustGraph!" in prompt
+            return "Welcome message processed!"
+
+        result = await prompt_manager.invoke("greeting", {"name": "Alice"}, mock_llm)
+        
+        assert result == "Welcome message processed!"
+
+    @pytest.mark.asyncio
+    async def test_prompt_manager_json_invocation(self, prompt_manager):
+        """Test PromptManager JSON response invocation"""
+        # Mock LLM function
+        async def mock_llm(system, prompt):
+            assert "Generate profile for johndoe" in prompt
+            return '{"name": "John Doe", "role": "user"}'
+
+        result = await prompt_manager.invoke("json_test", {"username": "johndoe"}, mock_llm)
+        
+        assert isinstance(result, dict)
+        assert result["name"] == "John Doe"
+        assert result["role"] == "user"
+
+    @pytest.mark.asyncio
+    async def test_prompt_manager_json_validation_error(self, prompt_manager):
+        """Test JSON schema validation failure"""
+        # Mock LLM function that returns invalid JSON
+        async def mock_llm(system, prompt):
+            return '{"name": "John Doe"}'  # Missing required "role"
+
+        with pytest.raises(RuntimeError) as exc_info:
+            await prompt_manager.invoke("json_test", {"username": "johndoe"}, mock_llm)
+        
+        assert "Schema validation fail" in str(exc_info.value)
+
+    @pytest.mark.asyncio
+    async def test_prompt_manager_json_parse_error(self, prompt_manager):
+        """Test JSON parsing failure"""
+        # Mock LLM function that returns non-JSON
+        async def mock_llm(system, prompt):
+            return "This is not JSON at all"
+
+        with pytest.raises(RuntimeError) as exc_info:
+            await prompt_manager.invoke("json_test", {"username": "johndoe"}, mock_llm)
+        
+        assert "JSON parse fail" in str(exc_info.value)
+
+    @pytest.mark.asyncio
+    async def test_prompt_manager_unknown_prompt(self, prompt_manager):
+        """Test unknown prompt ID handling"""
+        async def mock_llm(system, prompt):
+            return "Response"
+
+        with pytest.raises(KeyError):
+            await prompt_manager.invoke("unknown_prompt", {}, mock_llm)
+
+    @pytest.mark.asyncio
+    async def test_prompt_manager_term_merging(self, prompt_manager):
+        """Test proper term merging (global + prompt + input)"""
+        # Add prompt-specific terms
+        prompt_manager.prompts["greeting"].terms = {"greeting_prefix": "Hi"}
+        
+        async def mock_llm(system, prompt):
+            # Should have global term (system_name), input term (name), and any prompt terms
+            assert "TrustGraph" in prompt  # Global term
+            assert "Bob" in prompt  # Input term
+            return "Merged correctly"
+
+        result = await prompt_manager.invoke("greeting", {"name": "Bob"}, mock_llm)
+        assert result == "Merged correctly"
+
+    def test_prompt_manager_template_rendering(self, prompt_manager):
+        """Test direct template rendering"""
+        result = prompt_manager.render("greeting", {"name": "Charlie"})
+        
+        assert "Hello Charlie, welcome to TrustGraph!" == result.strip()
+
+    def test_prompt_manager_configuration_loading(self):
+        """Test configuration loading with various formats"""
+        pm = PromptManager()
+        
+        # Test empty configuration
+        pm.load_config({})
+        assert pm.config.system_template == "Be helpful."
+        assert len(pm.prompts) == 0
+        
+        # Test configuration with single prompt
+        config = {
+            "system": json.dumps("Test system"),
+            "template-index": json.dumps(["test"]),
+            "template.test": json.dumps({
+                "prompt": "Test {{ value }}",
+                "response-type": "text"
+            })
+        }
+        pm.load_config(config)
+        
+        assert pm.config.system_template == "Test system"
+        assert "test" in pm.prompts
+        assert pm.prompts["test"].response_type == "text"
+
+    @pytest.mark.asyncio
+    async def test_prompt_manager_json_with_markdown(self, prompt_manager):
+        """Test JSON extraction from markdown code blocks"""
+        async def mock_llm(system, prompt):
+            return '''
+            Here's the profile:
+            ```json
+            {"name": "Jane Smith", "role": "admin"}
+            ```
+            '''
+
+        result = await prompt_manager.invoke("json_test", {"username": "jane"}, mock_llm)
+        
+        assert isinstance(result, dict)
+        assert result["name"] == "Jane Smith"
+        assert result["role"] == "admin"
+
+    def test_prompt_manager_error_handling_in_templates(self, prompt_manager):
+        """Test error handling in template rendering"""
+        # Test with missing variable - ibis might handle this differently than Jinja2
+        try:
+            result = prompt_manager.render("greeting", {})  # Missing 'name'
+            # If no exception, check that result is still a string
+            assert isinstance(result, str)
+        except Exception as e:
+            # If exception is raised, that's also acceptable
+            assert "name" in str(e) or "undefined" in str(e).lower() or "variable" in str(e).lower()
+
+    @pytest.mark.asyncio
+    async def test_concurrent_prompt_invocations(self, prompt_manager):
+        """Test concurrent invocations"""
+        async def mock_llm(system, prompt):
+            # Extract name from prompt for response
+            if "Alice" in prompt:
+                return "Alice response"
+            elif "Bob" in prompt:
+                return "Bob response"
+            else:
+                return "Default response"
+
+        # Run concurrent invocations
+        import asyncio
+        results = await asyncio.gather(
+            prompt_manager.invoke("greeting", {"name": "Alice"}, mock_llm),
+            prompt_manager.invoke("greeting", {"name": "Bob"}, mock_llm),
+        )
+
+        assert "Alice response" in results
+        assert "Bob response" in results

tests/integration/test_text_completion_integration.py

@@ -0,0 +1,429 @@
+"""
+Integration tests for Text Completion Service (OpenAI)
+
+These tests verify the end-to-end functionality of the OpenAI text completion service,
+testing API connectivity, authentication, rate limiting, error handling, and token tracking.
+Following the TEST_STRATEGY.md approach for integration testing.
+"""
+
+import pytest
+import os
+from unittest.mock import AsyncMock, MagicMock, patch
+from openai import OpenAI, RateLimitError
+from openai.types.chat import ChatCompletion, ChatCompletionMessage
+from openai.types.chat.chat_completion import Choice
+from openai.types.completion_usage import CompletionUsage
+
+from trustgraph.model.text_completion.openai.llm import Processor
+from trustgraph.exceptions import TooManyRequests
+from trustgraph.base import LlmResult
+from trustgraph.schema import TextCompletionRequest, TextCompletionResponse, Error
+
+
+@pytest.mark.integration
+class TestTextCompletionIntegration:
+    """Integration tests for OpenAI text completion service coordination"""
+
+    @pytest.fixture
+    def mock_openai_client(self):
+        """Mock OpenAI client that returns realistic responses"""
+        client = MagicMock(spec=OpenAI)
+        
+        # Mock chat completion response
+        usage = CompletionUsage(prompt_tokens=50, completion_tokens=100, total_tokens=150)
+        message = ChatCompletionMessage(role="assistant", content="This is a test response from the AI model.")
+        choice = Choice(index=0, message=message, finish_reason="stop")
+        
+        completion = ChatCompletion(
+            id="chatcmpl-test123",
+            choices=[choice],
+            created=1234567890,
+            model="gpt-3.5-turbo",
+            object="chat.completion",
+            usage=usage
+        )
+        
+        client.chat.completions.create.return_value = completion
+        return client
+
+    @pytest.fixture
+    def processor_config(self):
+        """Configuration for processor testing"""
+        return {
+            "model": "gpt-3.5-turbo",
+            "temperature": 0.7,
+            "max_output": 1024,
+        }
+
+    @pytest.fixture
+    def text_completion_processor(self, processor_config):
+        """Create text completion processor with test configuration"""
+        # Create a minimal processor instance for testing generate_content
+        processor = MagicMock()
+        processor.model = processor_config["model"]
+        processor.temperature = processor_config["temperature"]
+        processor.max_output = processor_config["max_output"]
+        
+        # Add the actual generate_content method from Processor class
+        processor.generate_content = Processor.generate_content.__get__(processor, Processor)
+        
+        return processor
+
+    @pytest.mark.asyncio
+    async def test_text_completion_successful_generation(self, text_completion_processor, mock_openai_client):
+        """Test successful text completion generation"""
+        # Arrange
+        text_completion_processor.openai = mock_openai_client
+        system_prompt = "You are a helpful assistant."
+        user_prompt = "What is machine learning?"
+
+        # Act
+        result = await text_completion_processor.generate_content(system_prompt, user_prompt)
+
+        # Assert
+        assert isinstance(result, LlmResult)
+        assert result.text == "This is a test response from the AI model."
+        assert result.in_token == 50
+        assert result.out_token == 100
+        assert result.model == "gpt-3.5-turbo"
+
+        # Verify OpenAI API was called correctly
+        mock_openai_client.chat.completions.create.assert_called_once()
+        call_args = mock_openai_client.chat.completions.create.call_args
+        
+        assert call_args.kwargs['model'] == "gpt-3.5-turbo"
+        assert call_args.kwargs['temperature'] == 0.7
+        assert call_args.kwargs['max_tokens'] == 1024
+        assert len(call_args.kwargs['messages']) == 1
+        assert call_args.kwargs['messages'][0]['role'] == "user"
+        assert "You are a helpful assistant." in call_args.kwargs['messages'][0]['content'][0]['text']
+        assert "What is machine learning?" in call_args.kwargs['messages'][0]['content'][0]['text']
+
+    @pytest.mark.asyncio
+    async def test_text_completion_with_different_configurations(self, mock_openai_client):
+        """Test text completion with various configuration parameters"""
+        # Test different configurations
+        test_configs = [
+            {"model": "gpt-4", "temperature": 0.0, "max_output": 512},
+            {"model": "gpt-3.5-turbo", "temperature": 1.0, "max_output": 2048},
+            {"model": "gpt-4-turbo", "temperature": 0.5, "max_output": 4096}
+        ]
+
+        for config in test_configs:
+            # Arrange - Create minimal processor mock
+            processor = MagicMock()
+            processor.model = config['model']
+            processor.temperature = config['temperature']
+            processor.max_output = config['max_output']
+            processor.openai = mock_openai_client
+            
+            # Add the actual generate_content method
+            processor.generate_content = Processor.generate_content.__get__(processor, Processor)
+
+            # Act
+            result = await processor.generate_content("System prompt", "User prompt")
+
+            # Assert
+            assert isinstance(result, LlmResult)
+            assert result.text == "This is a test response from the AI model."
+            assert result.in_token == 50
+            assert result.out_token == 100
+            # Note: result.model comes from mock response, not processor config
+            
+            # Verify configuration was applied
+            call_args = mock_openai_client.chat.completions.create.call_args
+            assert call_args.kwargs['model'] == config['model']
+            assert call_args.kwargs['temperature'] == config['temperature']
+            assert call_args.kwargs['max_tokens'] == config['max_output']
+            
+            # Reset mock for next iteration
+            mock_openai_client.reset_mock()
+
+    @pytest.mark.asyncio
+    async def test_text_completion_rate_limit_handling(self, text_completion_processor, mock_openai_client):
+        """Test proper rate limit error handling"""
+        # Arrange
+        mock_openai_client.chat.completions.create.side_effect = RateLimitError(
+            "Rate limit exceeded", 
+            response=MagicMock(status_code=429),
+            body={}
+        )
+        text_completion_processor.openai = mock_openai_client
+
+        # Act & Assert
+        with pytest.raises(TooManyRequests):
+            await text_completion_processor.generate_content("System prompt", "User prompt")
+
+        # Verify OpenAI API was called
+        mock_openai_client.chat.completions.create.assert_called_once()
+
+    @pytest.mark.asyncio
+    async def test_text_completion_api_error_handling(self, text_completion_processor, mock_openai_client):
+        """Test handling of general API errors"""
+        # Arrange
+        mock_openai_client.chat.completions.create.side_effect = Exception("API connection failed")
+        text_completion_processor.openai = mock_openai_client
+
+        # Act & Assert
+        with pytest.raises(Exception) as exc_info:
+            await text_completion_processor.generate_content("System prompt", "User prompt")
+
+        assert "API connection failed" in str(exc_info.value)
+        mock_openai_client.chat.completions.create.assert_called_once()
+
+    @pytest.mark.asyncio
+    async def test_text_completion_token_tracking(self, text_completion_processor, mock_openai_client):
+        """Test accurate token counting and tracking"""
+        # Arrange - Different token counts for multiple requests
+        test_cases = [
+            (25, 75),   # Small request
+            (100, 200), # Medium request
+            (500, 1000) # Large request
+        ]
+
+        for input_tokens, output_tokens in test_cases:
+            # Update mock response with different token counts
+            usage = CompletionUsage(
+                prompt_tokens=input_tokens, 
+                completion_tokens=output_tokens, 
+                total_tokens=input_tokens + output_tokens
+            )
+            message = ChatCompletionMessage(role="assistant", content="Test response")
+            choice = Choice(index=0, message=message, finish_reason="stop")
+            
+            completion = ChatCompletion(
+                id="chatcmpl-test123",
+                choices=[choice],
+                created=1234567890,
+                model="gpt-3.5-turbo",
+                object="chat.completion",
+                usage=usage
+            )
+            
+            mock_openai_client.chat.completions.create.return_value = completion
+            text_completion_processor.openai = mock_openai_client
+
+            # Act
+            result = await text_completion_processor.generate_content("System", "Prompt")
+
+            # Assert
+            assert result.in_token == input_tokens
+            assert result.out_token == output_tokens
+            assert result.model == "gpt-3.5-turbo"
+
+            # Reset mock for next iteration
+            mock_openai_client.reset_mock()
+
+    @pytest.mark.asyncio
+    async def test_text_completion_prompt_construction(self, text_completion_processor, mock_openai_client):
+        """Test proper prompt construction with system and user prompts"""
+        # Arrange
+        text_completion_processor.openai = mock_openai_client
+        system_prompt = "You are an expert in artificial intelligence."
+        user_prompt = "Explain neural networks in simple terms."
+
+        # Act
+        result = await text_completion_processor.generate_content(system_prompt, user_prompt)
+
+        # Assert
+        call_args = mock_openai_client.chat.completions.create.call_args
+        sent_message = call_args.kwargs['messages'][0]['content'][0]['text']
+        
+        # Verify system and user prompts are combined correctly
+        assert system_prompt in sent_message
+        assert user_prompt in sent_message
+        assert sent_message.startswith(system_prompt)
+        assert user_prompt in sent_message
+
+    @pytest.mark.asyncio
+    async def test_text_completion_concurrent_requests(self, processor_config, mock_openai_client):
+        """Test handling of concurrent requests"""
+        # Arrange
+        processors = []
+        for i in range(5):
+            processor = MagicMock()
+            processor.model = processor_config["model"]
+            processor.temperature = processor_config["temperature"]
+            processor.max_output = processor_config["max_output"]
+            processor.openai = mock_openai_client
+            processor.generate_content = Processor.generate_content.__get__(processor, Processor)
+            processors.append(processor)
+
+        # Simulate multiple concurrent requests
+        tasks = []
+        for i, processor in enumerate(processors):
+            task = processor.generate_content(f"System {i}", f"Prompt {i}")
+            tasks.append(task)
+
+        # Act
+        import asyncio
+        results = await asyncio.gather(*tasks)
+
+        # Assert
+        assert len(results) == 5
+        for result in results:
+            assert isinstance(result, LlmResult)
+            assert result.text == "This is a test response from the AI model."
+            assert result.in_token == 50
+            assert result.out_token == 100
+
+        # Verify all requests were processed
+        assert mock_openai_client.chat.completions.create.call_count == 5
+
+    @pytest.mark.asyncio
+    async def test_text_completion_response_format_validation(self, text_completion_processor, mock_openai_client):
+        """Test response format and structure validation"""
+        # Arrange
+        text_completion_processor.openai = mock_openai_client
+
+        # Act
+        result = await text_completion_processor.generate_content("System", "Prompt")
+
+        # Assert
+        # Verify OpenAI API call parameters
+        call_args = mock_openai_client.chat.completions.create.call_args
+        assert call_args.kwargs['response_format'] == {"type": "text"}
+        assert call_args.kwargs['top_p'] == 1
+        assert call_args.kwargs['frequency_penalty'] == 0
+        assert call_args.kwargs['presence_penalty'] == 0
+
+        # Verify result structure
+        assert hasattr(result, 'text')
+        assert hasattr(result, 'in_token')
+        assert hasattr(result, 'out_token')
+        assert hasattr(result, 'model')
+
+    @pytest.mark.asyncio
+    async def test_text_completion_authentication_patterns(self):
+        """Test different authentication configurations"""
+        # Test missing API key first (this should fail early)
+        with pytest.raises(RuntimeError) as exc_info:
+            Processor(id="test-no-key", api_key=None)
+        assert "OpenAI API key not specified" in str(exc_info.value)
+
+        # Test authentication pattern by examining the initialization logic
+        # Since we can't fully instantiate due to taskgroup requirements,
+        # we'll test the authentication logic directly
+        from trustgraph.model.text_completion.openai.llm import default_api_key, default_base_url
+        
+        # Test default values
+        assert default_base_url == "https://api.openai.com/v1"
+        
+        # Test configuration parameters
+        test_configs = [
+            {"api_key": "test-key-1", "url": "https://api.openai.com/v1"},
+            {"api_key": "test-key-2", "url": "https://custom.openai.com/v1"},
+        ]
+        
+        for config in test_configs:
+            # We can't fully test instantiation due to taskgroup,
+            # but we can verify the authentication logic would work
+            assert config["api_key"] is not None
+            assert config["url"] is not None
+
+    @pytest.mark.asyncio
+    async def test_text_completion_error_propagation(self, text_completion_processor, mock_openai_client):
+        """Test error propagation through the service"""
+        # Test different error types
+        error_cases = [
+            (RateLimitError("Rate limit", response=MagicMock(status_code=429), body={}), TooManyRequests),
+            (Exception("Connection timeout"), Exception),
+            (ValueError("Invalid request"), ValueError),
+        ]
+
+        for error_input, expected_error in error_cases:
+            # Arrange
+            mock_openai_client.chat.completions.create.side_effect = error_input
+            text_completion_processor.openai = mock_openai_client
+
+            # Act & Assert
+            with pytest.raises(expected_error):
+                await text_completion_processor.generate_content("System", "Prompt")
+
+            # Reset mock for next iteration
+            mock_openai_client.reset_mock()
+
+    @pytest.mark.asyncio
+    async def test_text_completion_model_parameter_validation(self, mock_openai_client):
+        """Test that model parameters are correctly passed to OpenAI API"""
+        # Arrange
+        processor = MagicMock()
+        processor.model = "gpt-4"
+        processor.temperature = 0.8
+        processor.max_output = 2048
+        processor.openai = mock_openai_client
+        processor.generate_content = Processor.generate_content.__get__(processor, Processor)
+
+        # Act
+        await processor.generate_content("System prompt", "User prompt")
+
+        # Assert
+        call_args = mock_openai_client.chat.completions.create.call_args
+        assert call_args.kwargs['model'] == "gpt-4"
+        assert call_args.kwargs['temperature'] == 0.8
+        assert call_args.kwargs['max_tokens'] == 2048
+        assert call_args.kwargs['top_p'] == 1
+        assert call_args.kwargs['frequency_penalty'] == 0
+        assert call_args.kwargs['presence_penalty'] == 0
+
+    @pytest.mark.asyncio
+    @pytest.mark.slow
+    async def test_text_completion_performance_timing(self, text_completion_processor, mock_openai_client):
+        """Test performance timing for text completion"""
+        # Arrange
+        text_completion_processor.openai = mock_openai_client
+        
+        # Act
+        import time
+        start_time = time.time()
+        
+        result = await text_completion_processor.generate_content("System", "Prompt")
+        
+        end_time = time.time()
+        execution_time = end_time - start_time
+
+        # Assert
+        assert isinstance(result, LlmResult)
+        assert execution_time < 1.0  # Should complete quickly with mocked API
+        mock_openai_client.chat.completions.create.assert_called_once()
+
+    @pytest.mark.asyncio
+    async def test_text_completion_response_content_extraction(self, text_completion_processor, mock_openai_client):
+        """Test proper extraction of response content from OpenAI API"""
+        # Arrange
+        test_responses = [
+            "This is a simple response.",
+            "This is a multi-line response.\nWith multiple lines.\nAnd more content.",
+            "Response with special characters: @#$%^&*()_+-=[]{}|;':\",./<>?",
+            ""  # Empty response
+        ]
+
+        for test_content in test_responses:
+            # Update mock response
+            usage = CompletionUsage(prompt_tokens=10, completion_tokens=20, total_tokens=30)
+            message = ChatCompletionMessage(role="assistant", content=test_content)
+            choice = Choice(index=0, message=message, finish_reason="stop")
+            
+            completion = ChatCompletion(
+                id="chatcmpl-test123",
+                choices=[choice],
+                created=1234567890,
+                model="gpt-3.5-turbo",
+                object="chat.completion",
+                usage=usage
+            )
+            
+            mock_openai_client.chat.completions.create.return_value = completion
+            text_completion_processor.openai = mock_openai_client
+
+            # Act
+            result = await text_completion_processor.generate_content("System", "Prompt")
+
+            # Assert
+            assert result.text == test_content
+            assert result.in_token == 10
+            assert result.out_token == 20
+            assert result.model == "gpt-3.5-turbo"
+
+            # Reset mock for next iteration
+            mock_openai_client.reset_mock()

tests/pytest.ini

@@ -0,0 +1,22 @@
+[pytest]
+testpaths = tests
+python_paths = .
+python_files = test_*.py
+python_classes = Test*
+python_functions = test_*
+addopts = 
+    -v
+    --tb=short
+    --strict-markers
+    --disable-warnings
+    --cov=trustgraph
+    --cov-report=html
+    --cov-report=term-missing
+#    --cov-fail-under=80
+asyncio_mode = auto
+markers =
+    slow: marks tests as slow (deselect with '-m "not slow"')
+    integration: marks tests as integration tests
+    unit: marks tests as unit tests
+    contract: marks tests as contract tests (service interface validation)
+    vertexai: marks tests as vertex ai specific tests

tests/requirements.txt

@@ -0,0 +1,9 @@
+pytest>=7.0.0
+pytest-asyncio>=0.21.0
+pytest-mock>=3.10.0
+pytest-cov>=4.0.0
+google-cloud-aiplatform>=1.25.0
+google-auth>=2.17.0
+google-api-core>=2.11.0
+pulsar-client>=3.0.0
+prometheus-client>=0.16.0

tests/unit/__init__.py

@@ -0,0 +1,3 @@
+"""
+Unit tests for TrustGraph services
+"""

tests/unit/test_agent/__init__.py

@@ -0,0 +1,10 @@
+"""
+Unit tests for agent processing and ReAct pattern logic
+
+Testing Strategy:
+- Mock external LLM calls and tool executions
+- Test core ReAct reasoning cycle logic (Think-Act-Observe)
+- Test tool selection and coordination algorithms
+- Test conversation state management and multi-turn reasoning
+- Test response synthesis and answer generation
+"""

tests/unit/test_agent/conftest.py

@@ -0,0 +1,209 @@
+"""
+Shared fixtures for agent unit tests
+"""
+
+import pytest
+from unittest.mock import Mock, AsyncMock
+
+
+# Mock agent schema classes for testing
+class AgentRequest:
+    def __init__(self, question, conversation_id=None):
+        self.question = question
+        self.conversation_id = conversation_id
+
+
+class AgentResponse:
+    def __init__(self, answer, conversation_id=None, steps=None):
+        self.answer = answer
+        self.conversation_id = conversation_id
+        self.steps = steps or []
+
+
+class AgentStep:
+    def __init__(self, step_type, content, tool_name=None, tool_result=None):
+        self.step_type = step_type  # "think", "act", "observe"
+        self.content = content
+        self.tool_name = tool_name
+        self.tool_result = tool_result
+
+
+@pytest.fixture
+def sample_agent_request():
+    """Sample agent request for testing"""
+    return AgentRequest(
+        question="What is the capital of France?",
+        conversation_id="conv-123"
+    )
+
+
+@pytest.fixture
+def sample_agent_response():
+    """Sample agent response for testing"""
+    steps = [
+        AgentStep("think", "I need to find information about France's capital"),
+        AgentStep("act", "search", tool_name="knowledge_search", tool_result="Paris is the capital of France"),
+        AgentStep("observe", "I found that Paris is the capital of France"),
+        AgentStep("think", "I can now provide a complete answer")
+    ]
+    
+    return AgentResponse(
+        answer="The capital of France is Paris.",
+        conversation_id="conv-123",
+        steps=steps
+    )
+
+
+@pytest.fixture
+def mock_llm_client():
+    """Mock LLM client for agent reasoning"""
+    mock = AsyncMock()
+    mock.generate.return_value = "I need to search for information about the capital of France."
+    return mock
+
+
+@pytest.fixture
+def mock_knowledge_search_tool():
+    """Mock knowledge search tool"""
+    def search_tool(query):
+        if "capital" in query.lower() and "france" in query.lower():
+            return "Paris is the capital and largest city of France."
+        return "No relevant information found."
+    
+    return search_tool
+
+
+@pytest.fixture
+def mock_graph_rag_tool():
+    """Mock graph RAG tool"""
+    def graph_rag_tool(query):
+        return {
+            "entities": ["France", "Paris"],
+            "relationships": [("Paris", "capital_of", "France")],
+            "context": "Paris is the capital city of France, located in northern France."
+        }
+    
+    return graph_rag_tool
+
+
+@pytest.fixture
+def mock_calculator_tool():
+    """Mock calculator tool"""
+    def calculator_tool(expression):
+        # Simple mock calculator
+        try:
+            # Very basic expression evaluation for testing
+            if "+" in expression:
+                parts = expression.split("+")
+                return str(sum(int(p.strip()) for p in parts))
+            elif "*" in expression:
+                parts = expression.split("*")
+                result = 1
+                for p in parts:
+                    result *= int(p.strip())
+                return str(result)
+            return str(eval(expression))  # Simplified for testing
+        except:
+            return "Error: Invalid expression"
+    
+    return calculator_tool
+
+
+@pytest.fixture
+def available_tools(mock_knowledge_search_tool, mock_graph_rag_tool, mock_calculator_tool):
+    """Available tools for agent testing"""
+    return {
+        "knowledge_search": {
+            "function": mock_knowledge_search_tool,
+            "description": "Search knowledge base for information",
+            "parameters": ["query"]
+        },
+        "graph_rag": {
+            "function": mock_graph_rag_tool,
+            "description": "Query knowledge graph with RAG",
+            "parameters": ["query"]
+        },
+        "calculator": {
+            "function": mock_calculator_tool,
+            "description": "Perform mathematical calculations",
+            "parameters": ["expression"]
+        }
+    }
+
+
+@pytest.fixture
+def sample_conversation_history():
+    """Sample conversation history for multi-turn testing"""
+    return [
+        {
+            "role": "user",
+            "content": "What is 2 + 2?",
+            "timestamp": "2024-01-01T10:00:00Z"
+        },
+        {
+            "role": "assistant", 
+            "content": "2 + 2 = 4",
+            "steps": [
+                {"step_type": "think", "content": "This is a simple arithmetic question"},
+                {"step_type": "act", "content": "calculator", "tool_name": "calculator", "tool_result": "4"},
+                {"step_type": "observe", "content": "The calculator returned 4"},
+                {"step_type": "think", "content": "I can provide the answer"}
+            ],
+            "timestamp": "2024-01-01T10:00:05Z"
+        },
+        {
+            "role": "user",
+            "content": "What about 3 + 3?",
+            "timestamp": "2024-01-01T10:01:00Z"
+        }
+    ]
+
+
+@pytest.fixture
+def react_prompts():
+    """ReAct prompting templates for testing"""
+    return {
+        "system_prompt": """You are a helpful AI assistant that uses the ReAct (Reasoning and Acting) pattern.
+
+For each question, follow this cycle:
+1. Think: Analyze the question and plan your approach
+2. Act: Use available tools to gather information 
+3. Observe: Review the tool results
+4. Repeat if needed, then provide final answer
+
+Available tools: {tools}
+
+Format your response as:
+Think: [your reasoning]
+Act: [tool_name: parameters]
+Observe: [analysis of results]
+Answer: [final response]""",
+
+        "think_prompt": "Think step by step about this question: {question}\nPrevious context: {context}",
+        
+        "act_prompt": "Based on your thinking, what tool should you use? Available tools: {tools}",
+        
+        "observe_prompt": "You used {tool_name} and got result: {tool_result}\nHow does this help answer the question?",
+        
+        "synthesize_prompt": "Based on all your steps, provide a complete answer to: {question}"
+    }
+
+
+@pytest.fixture
+def mock_agent_processor():
+    """Mock agent processor for testing"""
+    class MockAgentProcessor:
+        def __init__(self, llm_client=None, tools=None):
+            self.llm_client = llm_client
+            self.tools = tools or {}
+            self.conversation_history = {}
+        
+        async def process_request(self, request):
+            # Mock processing logic
+            return AgentResponse(
+                answer="Mock response",
+                conversation_id=request.conversation_id,
+                steps=[]
+            )
+    
+    return MockAgentProcessor

tests/unit/test_agent/test_conversation_state.py

@@ -0,0 +1,596 @@
+"""
+Unit tests for conversation state management
+
+Tests the core business logic for managing conversation state,
+including history tracking, context preservation, and multi-turn
+reasoning support.
+"""
+
+import pytest
+from unittest.mock import Mock
+from datetime import datetime, timedelta
+import json
+
+
+class TestConversationStateLogic:
+    """Test cases for conversation state management business logic"""
+
+    def test_conversation_initialization(self):
+        """Test initialization of new conversation state"""
+        # Arrange
+        class ConversationState:
+            def __init__(self, conversation_id=None, user_id=None):
+                self.conversation_id = conversation_id or f"conv_{datetime.now().strftime('%Y%m%d_%H%M%S')}"
+                self.user_id = user_id
+                self.created_at = datetime.now()
+                self.updated_at = datetime.now()
+                self.turns = []
+                self.context = {}
+                self.metadata = {}
+                self.is_active = True
+            
+            def to_dict(self):
+                return {
+                    "conversation_id": self.conversation_id,
+                    "user_id": self.user_id,
+                    "created_at": self.created_at.isoformat(),
+                    "updated_at": self.updated_at.isoformat(),
+                    "turns": self.turns,
+                    "context": self.context,
+                    "metadata": self.metadata,
+                    "is_active": self.is_active
+                }
+        
+        # Act
+        conv1 = ConversationState(user_id="user123")
+        conv2 = ConversationState(conversation_id="custom_conv_id", user_id="user456")
+        
+        # Assert
+        assert conv1.conversation_id.startswith("conv_")
+        assert conv1.user_id == "user123"
+        assert conv1.is_active is True
+        assert len(conv1.turns) == 0
+        assert isinstance(conv1.created_at, datetime)
+        
+        assert conv2.conversation_id == "custom_conv_id"
+        assert conv2.user_id == "user456"
+        
+        # Test serialization
+        conv_dict = conv1.to_dict()
+        assert "conversation_id" in conv_dict
+        assert "created_at" in conv_dict
+        assert isinstance(conv_dict["turns"], list)
+
+    def test_turn_management(self):
+        """Test adding and managing conversation turns"""
+        # Arrange
+        class ConversationState:
+            def __init__(self, conversation_id=None, user_id=None):
+                self.conversation_id = conversation_id or f"conv_{datetime.now().strftime('%Y%m%d_%H%M%S')}"
+                self.user_id = user_id
+                self.created_at = datetime.now()
+                self.updated_at = datetime.now()
+                self.turns = []
+                self.context = {}
+                self.metadata = {}
+                self.is_active = True
+            
+            def to_dict(self):
+                return {
+                    "conversation_id": self.conversation_id,
+                    "user_id": self.user_id,
+                    "created_at": self.created_at.isoformat(),
+                    "updated_at": self.updated_at.isoformat(),
+                    "turns": self.turns,
+                    "context": self.context,
+                    "metadata": self.metadata,
+                    "is_active": self.is_active
+                }
+        
+        class ConversationTurn:
+            def __init__(self, role, content, timestamp=None, metadata=None):
+                self.role = role  # "user" or "assistant"
+                self.content = content
+                self.timestamp = timestamp or datetime.now()
+                self.metadata = metadata or {}
+            
+            def to_dict(self):
+                return {
+                    "role": self.role,
+                    "content": self.content,
+                    "timestamp": self.timestamp.isoformat(),
+                    "metadata": self.metadata
+                }
+        
+        class ConversationManager:
+            def __init__(self):
+                self.conversations = {}
+            
+            def add_turn(self, conversation_id, role, content, metadata=None):
+                if conversation_id not in self.conversations:
+                    return False, "Conversation not found"
+                
+                turn = ConversationTurn(role, content, metadata=metadata)
+                self.conversations[conversation_id].turns.append(turn)
+                self.conversations[conversation_id].updated_at = datetime.now()
+                
+                return True, turn
+            
+            def get_recent_turns(self, conversation_id, limit=10):
+                if conversation_id not in self.conversations:
+                    return []
+                
+                turns = self.conversations[conversation_id].turns
+                return turns[-limit:] if len(turns) > limit else turns
+            
+            def get_turn_count(self, conversation_id):
+                if conversation_id not in self.conversations:
+                    return 0
+                return len(self.conversations[conversation_id].turns)
+        
+        # Act
+        manager = ConversationManager()
+        conv_id = "test_conv"
+        
+        # Create conversation - use the local ConversationState class
+        conv_state = ConversationState(conv_id)
+        manager.conversations[conv_id] = conv_state
+        
+        # Add turns
+        success1, turn1 = manager.add_turn(conv_id, "user", "Hello, what is 2+2?")
+        success2, turn2 = manager.add_turn(conv_id, "assistant", "2+2 equals 4.")
+        success3, turn3 = manager.add_turn(conv_id, "user", "What about 3+3?")
+        
+        # Assert
+        assert success1 is True
+        assert turn1.role == "user"
+        assert turn1.content == "Hello, what is 2+2?"
+        
+        assert manager.get_turn_count(conv_id) == 3
+        
+        recent_turns = manager.get_recent_turns(conv_id, limit=2)
+        assert len(recent_turns) == 2
+        assert recent_turns[0].role == "assistant"
+        assert recent_turns[1].role == "user"
+
+    def test_context_preservation(self):
+        """Test preservation and retrieval of conversation context"""
+        # Arrange
+        class ContextManager:
+            def __init__(self):
+                self.contexts = {}
+            
+            def set_context(self, conversation_id, key, value, ttl_minutes=None):
+                """Set context value with optional TTL"""
+                if conversation_id not in self.contexts:
+                    self.contexts[conversation_id] = {}
+                
+                context_entry = {
+                    "value": value,
+                    "created_at": datetime.now(),
+                    "ttl_minutes": ttl_minutes
+                }
+                
+                self.contexts[conversation_id][key] = context_entry
+            
+            def get_context(self, conversation_id, key, default=None):
+                """Get context value, respecting TTL"""
+                if conversation_id not in self.contexts:
+                    return default
+                
+                if key not in self.contexts[conversation_id]:
+                    return default
+                
+                entry = self.contexts[conversation_id][key]
+                
+                # Check TTL
+                if entry["ttl_minutes"]:
+                    age = datetime.now() - entry["created_at"]
+                    if age > timedelta(minutes=entry["ttl_minutes"]):
+                        # Expired
+                        del self.contexts[conversation_id][key]
+                        return default
+                
+                return entry["value"]
+            
+            def update_context(self, conversation_id, updates):
+                """Update multiple context values"""
+                for key, value in updates.items():
+                    self.set_context(conversation_id, key, value)
+            
+            def clear_context(self, conversation_id, keys=None):
+                """Clear specific keys or entire context"""
+                if conversation_id not in self.contexts:
+                    return
+                
+                if keys is None:
+                    # Clear all context
+                    self.contexts[conversation_id] = {}
+                else:
+                    # Clear specific keys
+                    for key in keys:
+                        self.contexts[conversation_id].pop(key, None)
+            
+            def get_all_context(self, conversation_id):
+                """Get all context for conversation"""
+                if conversation_id not in self.contexts:
+                    return {}
+                
+                # Filter out expired entries
+                valid_context = {}
+                for key, entry in self.contexts[conversation_id].items():
+                    if entry["ttl_minutes"]:
+                        age = datetime.now() - entry["created_at"]
+                        if age <= timedelta(minutes=entry["ttl_minutes"]):
+                            valid_context[key] = entry["value"]
+                    else:
+                        valid_context[key] = entry["value"]
+                
+                return valid_context
+        
+        # Act
+        context_manager = ContextManager()
+        conv_id = "test_conv"
+        
+        # Set various context values
+        context_manager.set_context(conv_id, "user_name", "Alice")
+        context_manager.set_context(conv_id, "topic", "mathematics")
+        context_manager.set_context(conv_id, "temp_calculation", "2+2=4", ttl_minutes=1)
+        
+        # Assert
+        assert context_manager.get_context(conv_id, "user_name") == "Alice"
+        assert context_manager.get_context(conv_id, "topic") == "mathematics"
+        assert context_manager.get_context(conv_id, "temp_calculation") == "2+2=4"
+        assert context_manager.get_context(conv_id, "nonexistent", "default") == "default"
+        
+        # Test bulk updates
+        context_manager.update_context(conv_id, {
+            "calculation_count": 1,
+            "last_operation": "addition"
+        })
+        
+        all_context = context_manager.get_all_context(conv_id)
+        assert "calculation_count" in all_context
+        assert "last_operation" in all_context
+        assert len(all_context) == 5
+        
+        # Test clearing specific keys
+        context_manager.clear_context(conv_id, ["temp_calculation"])
+        assert context_manager.get_context(conv_id, "temp_calculation") is None
+        assert context_manager.get_context(conv_id, "user_name") == "Alice"
+
+    def test_multi_turn_reasoning_state(self):
+        """Test state management for multi-turn reasoning"""
+        # Arrange
+        class ReasoningStateManager:
+            def __init__(self):
+                self.reasoning_states = {}
+            
+            def start_reasoning_session(self, conversation_id, question, reasoning_type="sequential"):
+                """Start a new reasoning session"""
+                session_id = f"{conversation_id}_reasoning_{datetime.now().strftime('%H%M%S')}"
+                
+                self.reasoning_states[session_id] = {
+                    "conversation_id": conversation_id,
+                    "original_question": question,
+                    "reasoning_type": reasoning_type,
+                    "status": "active",
+                    "steps": [],
+                    "intermediate_results": {},
+                    "final_answer": None,
+                    "created_at": datetime.now(),
+                    "updated_at": datetime.now()
+                }
+                
+                return session_id
+            
+            def add_reasoning_step(self, session_id, step_type, content, tool_result=None):
+                """Add a step to reasoning session"""
+                if session_id not in self.reasoning_states:
+                    return False
+                
+                step = {
+                    "step_number": len(self.reasoning_states[session_id]["steps"]) + 1,
+                    "step_type": step_type,  # "think", "act", "observe"
+                    "content": content,
+                    "tool_result": tool_result,
+                    "timestamp": datetime.now()
+                }
+                
+                self.reasoning_states[session_id]["steps"].append(step)
+                self.reasoning_states[session_id]["updated_at"] = datetime.now()
+                
+                return True
+            
+            def set_intermediate_result(self, session_id, key, value):
+                """Store intermediate result for later use"""
+                if session_id not in self.reasoning_states:
+                    return False
+                
+                self.reasoning_states[session_id]["intermediate_results"][key] = value
+                return True
+            
+            def get_intermediate_result(self, session_id, key):
+                """Retrieve intermediate result"""
+                if session_id not in self.reasoning_states:
+                    return None
+                
+                return self.reasoning_states[session_id]["intermediate_results"].get(key)
+            
+            def complete_reasoning_session(self, session_id, final_answer):
+                """Mark reasoning session as complete"""
+                if session_id not in self.reasoning_states:
+                    return False
+                
+                self.reasoning_states[session_id]["final_answer"] = final_answer
+                self.reasoning_states[session_id]["status"] = "completed"
+                self.reasoning_states[session_id]["updated_at"] = datetime.now()
+                
+                return True
+            
+            def get_reasoning_summary(self, session_id):
+                """Get summary of reasoning session"""
+                if session_id not in self.reasoning_states:
+                    return None
+                
+                state = self.reasoning_states[session_id]
+                return {
+                    "original_question": state["original_question"],
+                    "step_count": len(state["steps"]),
+                    "status": state["status"],
+                    "final_answer": state["final_answer"],
+                    "reasoning_chain": [step["content"] for step in state["steps"] if step["step_type"] == "think"]
+                }
+        
+        # Act
+        reasoning_manager = ReasoningStateManager()
+        conv_id = "test_conv"
+        
+        # Start reasoning session
+        session_id = reasoning_manager.start_reasoning_session(
+            conv_id, 
+            "What is the population of the capital of France?"
+        )
+        
+        # Add reasoning steps
+        reasoning_manager.add_reasoning_step(session_id, "think", "I need to find the capital first")
+        reasoning_manager.add_reasoning_step(session_id, "act", "search for capital of France", "Paris")
+        reasoning_manager.set_intermediate_result(session_id, "capital", "Paris")
+        
+        reasoning_manager.add_reasoning_step(session_id, "observe", "Found that Paris is the capital")
+        reasoning_manager.add_reasoning_step(session_id, "think", "Now I need to find Paris population")
+        reasoning_manager.add_reasoning_step(session_id, "act", "search for Paris population", "2.1 million")
+        
+        reasoning_manager.complete_reasoning_session(session_id, "The population of Paris is approximately 2.1 million")
+        
+        # Assert
+        assert session_id.startswith(f"{conv_id}_reasoning_")
+        
+        capital = reasoning_manager.get_intermediate_result(session_id, "capital")
+        assert capital == "Paris"
+        
+        summary = reasoning_manager.get_reasoning_summary(session_id)
+        assert summary["original_question"] == "What is the population of the capital of France?"
+        assert summary["step_count"] == 5
+        assert summary["status"] == "completed"
+        assert "2.1 million" in summary["final_answer"]
+        assert len(summary["reasoning_chain"]) == 2  # Two "think" steps
+
+    def test_conversation_memory_management(self):
+        """Test memory management for long conversations"""
+        # Arrange
+        class ConversationMemoryManager:
+            def __init__(self, max_turns=100, max_context_age_hours=24):
+                self.max_turns = max_turns
+                self.max_context_age_hours = max_context_age_hours
+                self.conversations = {}
+            
+            def add_conversation_turn(self, conversation_id, role, content, metadata=None):
+                """Add turn with automatic memory management"""
+                if conversation_id not in self.conversations:
+                    self.conversations[conversation_id] = {
+                        "turns": [],
+                        "context": {},
+                        "created_at": datetime.now()
+                    }
+                
+                turn = {
+                    "role": role,
+                    "content": content,
+                    "timestamp": datetime.now(),
+                    "metadata": metadata or {}
+                }
+                
+                self.conversations[conversation_id]["turns"].append(turn)
+                
+                # Apply memory management
+                self._manage_memory(conversation_id)
+            
+            def _manage_memory(self, conversation_id):
+                """Apply memory management policies"""
+                conv = self.conversations[conversation_id]
+                
+                # Limit turn count
+                if len(conv["turns"]) > self.max_turns:
+                    # Keep recent turns and important summary turns
+                    turns_to_keep = self.max_turns // 2
+                    important_turns = self._identify_important_turns(conv["turns"])
+                    recent_turns = conv["turns"][-turns_to_keep:]
+                    
+                    # Combine important and recent turns, avoiding duplicates
+                    kept_turns = []
+                    seen_indices = set()
+                    
+                    # Add important turns first
+                    for turn_index, turn in important_turns:
+                        if turn_index not in seen_indices:
+                            kept_turns.append(turn)
+                            seen_indices.add(turn_index)
+                    
+                    # Add recent turns
+                    for i, turn in enumerate(recent_turns):
+                        original_index = len(conv["turns"]) - len(recent_turns) + i
+                        if original_index not in seen_indices:
+                            kept_turns.append(turn)
+                    
+                    conv["turns"] = kept_turns[-self.max_turns:]  # Final limit
+                
+                # Clean old context
+                self._clean_old_context(conversation_id)
+            
+            def _identify_important_turns(self, turns):
+                """Identify important turns to preserve"""
+                important = []
+                
+                for i, turn in enumerate(turns):
+                    # Keep turns with high information content
+                    if (len(turn["content"]) > 100 or 
+                        any(keyword in turn["content"].lower() for keyword in ["calculate", "result", "answer", "conclusion"])):
+                        important.append((i, turn))
+                
+                return important[:10]  # Limit important turns
+            
+            def _clean_old_context(self, conversation_id):
+                """Remove old context entries"""
+                if conversation_id not in self.conversations:
+                    return
+                
+                cutoff_time = datetime.now() - timedelta(hours=self.max_context_age_hours)
+                context = self.conversations[conversation_id]["context"]
+                
+                keys_to_remove = []
+                for key, entry in context.items():
+                    if isinstance(entry, dict) and "timestamp" in entry:
+                        if entry["timestamp"] < cutoff_time:
+                            keys_to_remove.append(key)
+                
+                for key in keys_to_remove:
+                    del context[key]
+            
+            def get_conversation_summary(self, conversation_id):
+                """Get summary of conversation state"""
+                if conversation_id not in self.conversations:
+                    return None
+                
+                conv = self.conversations[conversation_id]
+                return {
+                    "turn_count": len(conv["turns"]),
+                    "context_keys": list(conv["context"].keys()),
+                    "age_hours": (datetime.now() - conv["created_at"]).total_seconds() / 3600,
+                    "last_activity": conv["turns"][-1]["timestamp"] if conv["turns"] else None
+                }
+        
+        # Act
+        memory_manager = ConversationMemoryManager(max_turns=5, max_context_age_hours=1)
+        conv_id = "test_memory_conv"
+        
+        # Add many turns to test memory management
+        for i in range(10):
+            memory_manager.add_conversation_turn(
+                conv_id, 
+                "user" if i % 2 == 0 else "assistant",
+                f"Turn {i}: {'Important calculation result' if i == 5 else 'Regular content'}"
+            )
+        
+        # Assert
+        summary = memory_manager.get_conversation_summary(conv_id)
+        assert summary["turn_count"] <= 5  # Should be limited
+        
+        # Check that important turns are preserved
+        turns = memory_manager.conversations[conv_id]["turns"]
+        important_preserved = any("Important calculation" in turn["content"] for turn in turns)
+        assert important_preserved, "Important turns should be preserved"
+
+    def test_conversation_state_persistence(self):
+        """Test serialization and deserialization of conversation state"""
+        # Arrange
+        class ConversationStatePersistence:
+            def __init__(self):
+                pass
+            
+            def serialize_conversation(self, conversation_state):
+                """Serialize conversation state to JSON-compatible format"""
+                def datetime_serializer(obj):
+                    if isinstance(obj, datetime):
+                        return obj.isoformat()
+                    raise TypeError(f"Object of type {type(obj)} is not JSON serializable")
+                
+                return json.dumps(conversation_state, default=datetime_serializer, indent=2)
+            
+            def deserialize_conversation(self, serialized_data):
+                """Deserialize conversation state from JSON"""
+                def datetime_deserializer(data):
+                    """Convert ISO datetime strings back to datetime objects"""
+                    if isinstance(data, dict):
+                        for key, value in data.items():
+                            if isinstance(value, str) and self._is_iso_datetime(value):
+                                data[key] = datetime.fromisoformat(value)
+                            elif isinstance(value, (dict, list)):
+                                data[key] = datetime_deserializer(value)
+                    elif isinstance(data, list):
+                        for i, item in enumerate(data):
+                            data[i] = datetime_deserializer(item)
+                    
+                    return data
+                
+                parsed_data = json.loads(serialized_data)
+                return datetime_deserializer(parsed_data)
+            
+            def _is_iso_datetime(self, value):
+                """Check if string is ISO datetime format"""
+                try:
+                    datetime.fromisoformat(value.replace('Z', '+00:00'))
+                    return True
+                except (ValueError, AttributeError):
+                    return False
+        
+        # Create sample conversation state
+        conversation_state = {
+            "conversation_id": "test_conv_123",
+            "user_id": "user456",
+            "created_at": datetime.now(),
+            "updated_at": datetime.now(),
+            "turns": [
+                {
+                    "role": "user",
+                    "content": "Hello",
+                    "timestamp": datetime.now(),
+                    "metadata": {}
+                },
+                {
+                    "role": "assistant", 
+                    "content": "Hi there!",
+                    "timestamp": datetime.now(),
+                    "metadata": {"confidence": 0.9}
+                }
+            ],
+            "context": {
+                "user_preference": "detailed_answers",
+                "topic": "general"
+            },
+            "metadata": {
+                "platform": "web",
+                "session_start": datetime.now()
+            }
+        }
+        
+        # Act
+        persistence = ConversationStatePersistence()
+        
+        # Serialize
+        serialized = persistence.serialize_conversation(conversation_state)
+        assert isinstance(serialized, str)
+        assert "test_conv_123" in serialized
+        
+        # Deserialize
+        deserialized = persistence.deserialize_conversation(serialized)
+        
+        # Assert
+        assert deserialized["conversation_id"] == "test_conv_123"
+        assert deserialized["user_id"] == "user456"
+        assert isinstance(deserialized["created_at"], datetime)
+        assert len(deserialized["turns"]) == 2
+        assert deserialized["turns"][0]["role"] == "user"
+        assert isinstance(deserialized["turns"][0]["timestamp"], datetime)
+        assert deserialized["context"]["topic"] == "general"
+        assert deserialized["metadata"]["platform"] == "web"

tests/unit/test_agent/test_react_processor.py

@@ -0,0 +1,477 @@
+"""
+Unit tests for ReAct processor logic
+
+Tests the core business logic for the ReAct (Reasoning and Acting) pattern
+without relying on external LLM services, focusing on the Think-Act-Observe
+cycle and tool coordination.
+"""
+
+import pytest
+from unittest.mock import Mock, AsyncMock, patch
+import re
+
+
+class TestReActProcessorLogic:
+    """Test cases for ReAct processor business logic"""
+
+    def test_react_cycle_parsing(self):
+        """Test parsing of ReAct cycle components from LLM output"""
+        # Arrange
+        llm_output = """Think: I need to find information about the capital of France.
+Act: knowledge_search: capital of France
+Observe: The search returned that Paris is the capital of France.
+Think: I now have enough information to answer.
+Answer: The capital of France is Paris."""
+        
+        def parse_react_output(text):
+            """Parse ReAct format output into structured steps"""
+            steps = []
+            lines = text.strip().split('\n')
+            
+            for line in lines:
+                line = line.strip()
+                if line.startswith('Think:'):
+                    steps.append({
+                        'type': 'think',
+                        'content': line[6:].strip()
+                    })
+                elif line.startswith('Act:'):
+                    act_content = line[4:].strip()
+                    # Parse "tool_name: parameters" format
+                    if ':' in act_content:
+                        tool_name, params = act_content.split(':', 1)
+                        steps.append({
+                            'type': 'act',
+                            'tool_name': tool_name.strip(),
+                            'parameters': params.strip()
+                        })
+                    else:
+                        steps.append({
+                            'type': 'act',
+                            'content': act_content
+                        })
+                elif line.startswith('Observe:'):
+                    steps.append({
+                        'type': 'observe',
+                        'content': line[8:].strip()
+                    })
+                elif line.startswith('Answer:'):
+                    steps.append({
+                        'type': 'answer',
+                        'content': line[7:].strip()
+                    })
+            
+            return steps
+        
+        # Act
+        steps = parse_react_output(llm_output)
+        
+        # Assert
+        assert len(steps) == 5
+        assert steps[0]['type'] == 'think'
+        assert steps[1]['type'] == 'act'
+        assert steps[1]['tool_name'] == 'knowledge_search'
+        assert steps[1]['parameters'] == 'capital of France'
+        assert steps[2]['type'] == 'observe'
+        assert steps[3]['type'] == 'think'
+        assert steps[4]['type'] == 'answer'
+
+    def test_tool_selection_logic(self):
+        """Test tool selection based on question type and context"""
+        # Arrange
+        test_cases = [
+            ("What is 2 + 2?", "calculator"),
+            ("Who is the president of France?", "knowledge_search"),
+            ("Tell me about the relationship between Paris and France", "graph_rag"),
+            ("What time is it?", "knowledge_search")  # Default to general search
+        ]
+        
+        available_tools = {
+            "calculator": {"description": "Perform mathematical calculations"},
+            "knowledge_search": {"description": "Search knowledge base for facts"},
+            "graph_rag": {"description": "Query knowledge graph for relationships"}
+        }
+        
+        def select_tool(question, tools):
+            """Select appropriate tool based on question content"""
+            question_lower = question.lower()
+            
+            # Math keywords
+            if any(word in question_lower for word in ['+', '-', '*', '/', 'calculate', 'math']):
+                return "calculator"
+            
+            # Relationship/graph keywords
+            if any(word in question_lower for word in ['relationship', 'between', 'connected', 'related']):
+                return "graph_rag"
+            
+            # General knowledge keywords or default case
+            if any(word in question_lower for word in ['who', 'what', 'where', 'when', 'why', 'how', 'time']):
+                return "knowledge_search"
+            
+            return None
+        
+        # Act & Assert
+        for question, expected_tool in test_cases:
+            selected_tool = select_tool(question, available_tools)
+            assert selected_tool == expected_tool, f"Question '{question}' should select {expected_tool}"
+
+    def test_tool_execution_logic(self):
+        """Test tool execution and result processing"""
+        # Arrange
+        def mock_knowledge_search(query):
+            if "capital" in query.lower() and "france" in query.lower():
+                return "Paris is the capital of France."
+            return "Information not found."
+        
+        def mock_calculator(expression):
+            try:
+                # Simple expression evaluation
+                if '+' in expression:
+                    parts = expression.split('+')
+                    return str(sum(int(p.strip()) for p in parts))
+                return str(eval(expression))
+            except:
+                return "Error: Invalid expression"
+        
+        tools = {
+            "knowledge_search": mock_knowledge_search,
+            "calculator": mock_calculator
+        }
+        
+        def execute_tool(tool_name, parameters, available_tools):
+            """Execute tool with given parameters"""
+            if tool_name not in available_tools:
+                return {"error": f"Tool {tool_name} not available"}
+            
+            try:
+                tool_function = available_tools[tool_name]
+                result = tool_function(parameters)
+                return {"success": True, "result": result}
+            except Exception as e:
+                return {"error": str(e)}
+        
+        # Act & Assert
+        test_cases = [
+            ("knowledge_search", "capital of France", "Paris is the capital of France."),
+            ("calculator", "2 + 2", "4"),
+            ("calculator", "invalid expression", "Error: Invalid expression"),
+            ("nonexistent_tool", "anything", None)  # Error case
+        ]
+        
+        for tool_name, params, expected in test_cases:
+            result = execute_tool(tool_name, params, tools)
+            
+            if expected is None:
+                assert "error" in result
+            else:
+                assert result.get("result") == expected
+
+    def test_conversation_context_integration(self):
+        """Test integration of conversation history into ReAct reasoning"""
+        # Arrange
+        conversation_history = [
+            {"role": "user", "content": "What is 2 + 2?"},
+            {"role": "assistant", "content": "2 + 2 = 4"},
+            {"role": "user", "content": "What about 3 + 3?"}
+        ]
+        
+        def build_context_prompt(question, history, max_turns=3):
+            """Build context prompt from conversation history"""
+            context_parts = []
+            
+            # Include recent conversation turns
+            recent_history = history[-(max_turns*2):] if history else []
+            
+            for turn in recent_history:
+                role = turn["role"]
+                content = turn["content"]
+                context_parts.append(f"{role}: {content}")
+            
+            current_question = f"user: {question}"
+            context_parts.append(current_question)
+            
+            return "\n".join(context_parts)
+        
+        # Act
+        context_prompt = build_context_prompt("What about 3 + 3?", conversation_history)
+        
+        # Assert
+        assert "2 + 2" in context_prompt
+        assert "2 + 2 = 4" in context_prompt
+        assert "3 + 3" in context_prompt
+        assert context_prompt.count("user:") == 3
+        assert context_prompt.count("assistant:") == 1
+
+    def test_react_cycle_validation(self):
+        """Test validation of complete ReAct cycles"""
+        # Arrange
+        complete_cycle = [
+            {"type": "think", "content": "I need to solve this math problem"},
+            {"type": "act", "tool_name": "calculator", "parameters": "2 + 2"},
+            {"type": "observe", "content": "The calculator returned 4"},
+            {"type": "think", "content": "I can now provide the answer"},
+            {"type": "answer", "content": "2 + 2 = 4"}
+        ]
+        
+        incomplete_cycle = [
+            {"type": "think", "content": "I need to solve this"},
+            {"type": "act", "tool_name": "calculator", "parameters": "2 + 2"}
+            # Missing observe and answer steps
+        ]
+        
+        def validate_react_cycle(steps):
+            """Validate that ReAct cycle is complete"""
+            step_types = [step.get("type") for step in steps]
+            
+            # Must have at least one think, act, observe, and answer
+            required_types = ["think", "act", "observe", "answer"]
+            
+            validation_results = {
+                "is_complete": all(req_type in step_types for req_type in required_types),
+                "has_reasoning": "think" in step_types,
+                "has_action": "act" in step_types,
+                "has_observation": "observe" in step_types,
+                "has_answer": "answer" in step_types,
+                "step_count": len(steps)
+            }
+            
+            return validation_results
+        
+        # Act & Assert
+        complete_validation = validate_react_cycle(complete_cycle)
+        assert complete_validation["is_complete"] is True
+        assert complete_validation["has_reasoning"] is True
+        assert complete_validation["has_action"] is True
+        assert complete_validation["has_observation"] is True
+        assert complete_validation["has_answer"] is True
+        
+        incomplete_validation = validate_react_cycle(incomplete_cycle)
+        assert incomplete_validation["is_complete"] is False
+        assert incomplete_validation["has_reasoning"] is True
+        assert incomplete_validation["has_action"] is True
+        assert incomplete_validation["has_observation"] is False
+        assert incomplete_validation["has_answer"] is False
+
+    def test_multi_step_reasoning_logic(self):
+        """Test multi-step reasoning chains"""
+        # Arrange
+        complex_question = "What is the population of the capital of France?"
+        
+        def plan_reasoning_steps(question):
+            """Plan the reasoning steps needed for complex questions"""
+            steps = []
+            
+            question_lower = question.lower()
+            
+            # Check if question requires multiple pieces of information
+            if "capital of" in question_lower and ("population" in question_lower or "how many" in question_lower):
+                steps.append({
+                    "step": 1,
+                    "action": "find_capital",
+                    "description": "First find the capital city"
+                })
+                steps.append({
+                    "step": 2,
+                    "action": "find_population",
+                    "description": "Then find the population of that city"
+                })
+            elif "capital of" in question_lower:
+                steps.append({
+                    "step": 1,
+                    "action": "find_capital",
+                    "description": "Find the capital city"
+                })
+            elif "population" in question_lower:
+                steps.append({
+                    "step": 1,
+                    "action": "find_population",
+                    "description": "Find the population"
+                })
+            else:
+                steps.append({
+                    "step": 1,
+                    "action": "general_search",
+                    "description": "Search for relevant information"
+                })
+            
+            return steps
+        
+        # Act
+        reasoning_plan = plan_reasoning_steps(complex_question)
+        
+        # Assert
+        assert len(reasoning_plan) == 2
+        assert reasoning_plan[0]["action"] == "find_capital"
+        assert reasoning_plan[1]["action"] == "find_population"
+        assert all("step" in step for step in reasoning_plan)
+
+    def test_error_handling_in_react_cycle(self):
+        """Test error handling during ReAct execution"""
+        # Arrange
+        def execute_react_step_with_errors(step_type, content, tools=None):
+            """Execute ReAct step with potential error handling"""
+            try:
+                if step_type == "think":
+                    # Thinking step - validate reasoning
+                    if not content or len(content.strip()) < 5:
+                        return {"error": "Reasoning too brief"}
+                    return {"success": True, "content": content}
+                
+                elif step_type == "act":
+                    # Action step - validate tool exists and execute
+                    if not tools or not content:
+                        return {"error": "No tools available or no action specified"}
+                    
+                    # Parse tool and parameters
+                    if ":" in content:
+                        tool_name, params = content.split(":", 1)
+                        tool_name = tool_name.strip()
+                        params = params.strip()
+                        
+                        if tool_name not in tools:
+                            return {"error": f"Tool {tool_name} not available"}
+                        
+                        # Execute tool
+                        result = tools[tool_name](params)
+                        return {"success": True, "tool_result": result}
+                    else:
+                        return {"error": "Invalid action format"}
+                
+                elif step_type == "observe":
+                    # Observation step - validate observation
+                    if not content:
+                        return {"error": "No observation provided"}
+                    return {"success": True, "content": content}
+                
+                else:
+                    return {"error": f"Unknown step type: {step_type}"}
+                    
+            except Exception as e:
+                return {"error": f"Execution error: {str(e)}"}
+        
+        # Test cases
+        mock_tools = {
+            "calculator": lambda x: str(eval(x)) if x.replace('+', '').replace('-', '').replace('*', '').replace('/', '').replace(' ', '').isdigit() else "Error"
+        }
+        
+        test_cases = [
+            ("think", "I need to calculate", {"success": True}),
+            ("think", "", {"error": True}),  # Empty reasoning
+            ("act", "calculator: 2 + 2", {"success": True}),
+            ("act", "nonexistent: something", {"error": True}),  # Tool doesn't exist
+            ("act", "invalid format", {"error": True}),  # Invalid format
+            ("observe", "The result is 4", {"success": True}),
+            ("observe", "", {"error": True}),  # Empty observation
+            ("invalid_step", "content", {"error": True})  # Invalid step type
+        ]
+        
+        # Act & Assert
+        for step_type, content, expected in test_cases:
+            result = execute_react_step_with_errors(step_type, content, mock_tools)
+            
+            if expected.get("error"):
+                assert "error" in result, f"Expected error for step {step_type}: {content}"
+            else:
+                assert "success" in result, f"Expected success for step {step_type}: {content}"
+
+    def test_response_synthesis_logic(self):
+        """Test synthesis of final response from ReAct steps"""
+        # Arrange
+        react_steps = [
+            {"type": "think", "content": "I need to find the capital of France"},
+            {"type": "act", "tool_name": "knowledge_search", "tool_result": "Paris is the capital of France"},
+            {"type": "observe", "content": "The search confirmed Paris is the capital"},
+            {"type": "think", "content": "I have the information needed to answer"}
+        ]
+        
+        def synthesize_response(steps, original_question):
+            """Synthesize final response from ReAct steps"""
+            # Extract key information from steps
+            tool_results = []
+            observations = []
+            reasoning = []
+            
+            for step in steps:
+                if step["type"] == "think":
+                    reasoning.append(step["content"])
+                elif step["type"] == "act" and "tool_result" in step:
+                    tool_results.append(step["tool_result"])
+                elif step["type"] == "observe":
+                    observations.append(step["content"])
+            
+            # Build response based on available information
+            if tool_results:
+                # Use tool results as primary information source
+                primary_info = tool_results[0]
+                
+                # Extract specific answer from tool result
+                if "capital" in original_question.lower() and "Paris" in primary_info:
+                    return "The capital of France is Paris."
+                elif "+" in original_question and any(char.isdigit() for char in primary_info):
+                    return f"The answer is {primary_info}."
+                else:
+                    return primary_info
+            else:
+                # Fallback to reasoning if no tool results
+                return "I need more information to answer this question."
+        
+        # Act
+        response = synthesize_response(react_steps, "What is the capital of France?")
+        
+        # Assert
+        assert "Paris" in response
+        assert "capital of france" in response.lower()
+        assert len(response) > 10  # Should be a complete sentence
+
+    def test_tool_parameter_extraction(self):
+        """Test extraction and validation of tool parameters"""
+        # Arrange
+        def extract_tool_parameters(action_content, tool_schema):
+            """Extract and validate parameters for tool execution"""
+            # Parse action content for tool name and parameters
+            if ":" not in action_content:
+                return {"error": "Invalid action format - missing tool parameters"}
+            
+            tool_name, params_str = action_content.split(":", 1)
+            tool_name = tool_name.strip()
+            params_str = params_str.strip()
+            
+            if tool_name not in tool_schema:
+                return {"error": f"Unknown tool: {tool_name}"}
+            
+            schema = tool_schema[tool_name]
+            required_params = schema.get("required_parameters", [])
+            
+            # Simple parameter extraction (for more complex tools, this would be more sophisticated)
+            if len(required_params) == 1 and required_params[0] == "query":
+                # Single query parameter
+                return {"tool_name": tool_name, "parameters": {"query": params_str}}
+            elif len(required_params) == 1 and required_params[0] == "expression":
+                # Single expression parameter  
+                return {"tool_name": tool_name, "parameters": {"expression": params_str}}
+            else:
+                # Multiple parameters would need more complex parsing
+                return {"tool_name": tool_name, "parameters": {"input": params_str}}
+        
+        tool_schema = {
+            "knowledge_search": {"required_parameters": ["query"]},
+            "calculator": {"required_parameters": ["expression"]},
+            "graph_rag": {"required_parameters": ["query"]}
+        }
+        
+        test_cases = [
+            ("knowledge_search: capital of France", "knowledge_search", {"query": "capital of France"}),
+            ("calculator: 2 + 2", "calculator", {"expression": "2 + 2"}),
+            ("invalid format", None, None),  # No colon
+            ("unknown_tool: something", None, None)  # Unknown tool
+        ]
+        
+        # Act & Assert
+        for action_content, expected_tool, expected_params in test_cases:
+            result = extract_tool_parameters(action_content, tool_schema)
+            
+            if expected_tool is None:
+                assert "error" in result
+            else:
+                assert result["tool_name"] == expected_tool
+                assert result["parameters"] == expected_params