apunkt/trustgraph
1
0
Fork 0
mirror of https://github.com/trustgraph-ai/trustgraph.git synced 2026-07-24 12:41:02 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Structured data tech spec emerging

Browse source
This commit is contained in:
Cyber MacGeddon 2025-08-04 17:25:57 +01:00
parent 9640020dc3
commit ac15882673
1 changed files with 23 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

28
docs/tech-specs/STRUCTURED_DATA.md
View file

@ -46,6 +46,8 @@ The structured data integration requires the following technical components:
- Converts natural language questions into structured queries - Converts natural language questions into structured queries
- Supports multiple query language targets (initially SQL-like syntax) - Supports multiple query language targets (initially SQL-like syntax)
- Integrates with existing TrustGraph NLP capabilities - Integrates with existing TrustGraph NLP capabilities
Module: trustgraph-flow/trustgraph/query/nlp_query/cassandra
2. **Configuration Schema Support** 2. **Configuration Schema Support**
- Extended configuration system to store structured data schemas - Extended configuration system to store structured data schemas
@ -58,18 +60,24 @@ The structured data integration requires the following technical components:
- Identifies and extracts structured objects from unstructured sources - Identifies and extracts structured objects from unstructured sources
- Maintains provenance and confidence scores - Maintains provenance and confidence scores
Module: trustgraph-flow/trustgraph/extract/object/row/
4. **Structured Store Writer Module** 4. **Structured Store Writer Module**
- Handles object persistence to structured data stores - Handles object persistence to structured data stores
- Initial implementation targeting Apache Cassandra - Initial implementation targeting Apache Cassandra
- Supports batch and streaming write operations - Supports batch and streaming write operations
- Manages schema mapping and data transformation - Manages schema mapping and data transformation
Module: trustgraph-flow/trustgraph/storage/objects/cassandra
5. **Structured Query Service** 5. **Structured Query Service**
- Accepts structured queries in defined formats - Accepts structured queries in defined formats
- Executes queries against the structured store - Executes queries against the structured store
- Returns objects matching query criteria - Returns objects matching query criteria
- Supports pagination and result filtering - Supports pagination and result filtering
Module: trustgraph-flow/trustgraph/query/objects/cassandra
6. **Agent Tool Integration** 6. **Agent Tool Integration**
- New tool class for agent frameworks - New tool class for agent frameworks
- Enables agents to query structured data stores - Enables agents to query structured data stores
@ -83,6 +91,16 @@ The structured data integration requires the following technical components:
- Emits objects to appropriate message queues for processing - Emits objects to appropriate message queues for processing
- Supports bulk uploads and streaming ingestion - Supports bulk uploads and streaming ingestion
Module: trustgraph-flow/trustgraph/decoding/structured
8. **Object Embedding Service**
- Generates vector embeddings for structured objects
- Enables semantic search across structured data
- Supports hybrid search combining structured queries with semantic similarity
- Integrates with existing vector stores
Module: trustgraph-flow/trustgraph/embeddings/object_embeddings/qdrant
### Data Models ### Data Models
#### Schema Storage Mechanism #### Schema Storage Mechanism
@ -139,11 +157,11 @@ This approach allows:
### APIs ### APIs
New APIs: New APIs:
- `POST /api/v1/nlp-to-query` - Convert natural language to structured query - Pulsar schemas for above types
- `POST /api/v1/structured-data/ingest` - Ingest structured data (CSV, JSON, XML) - Pulsar interfaces in new flows
- `POST /api/v1/structured-data/query` - Execute structured queries - Need a means to specify schema types in flows so that flows know which
- `GET /api/v1/schemas` - List available schemas schema types to load
- `GET /api/v1/schemas/{name}` - Get specific schema definition - APIs added to gateway and rev-gateway
Modified APIs: Modified APIs:
- Knowledge extraction endpoints - Add structured object output option - Knowledge extraction endpoints - Add structured object output option