apunkt/trustgraph
1
0
Fork 0
mirror of https://github.com/trustgraph-ai/trustgraph.git synced 2026-07-24 04:31:02 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

Tweak to tech spec

Browse source
This commit is contained in:
Cyber MacGeddon 2025-09-30 12:14:25 +01:00
parent d73e55ade2
commit e0ae76321e
1 changed files with 25 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

30
docs/tech-specs/collection-management.md
View file

@ -102,19 +102,31 @@ This approach allows:
#### Collection Lifecycle #### Collection Lifecycle
Collections follow an explicit creation pattern requiring user action before data can be stored: Collections are explicitly created in the librarian before data operations can proceed:
1. **Explicit Creation**: Collections must be created via `tg-set-collection` command before any data operations: 1. **Collection Creation** (Two Paths):
**Path A: User-Initiated Creation** via `tg-set-collection`:
- User provides collection ID, name, description, and tags - User provides collection ID, name, description, and tags
- Librarian creates metadata record in `collections` table - Librarian creates metadata record in `collections` table
- Librarian broadcasts "create-collection" to all storage backends - Librarian broadcasts "create-collection" to all storage backends
- All storage processors create collection and confirm success - All storage processors create collection and confirm success
- Only after all confirmations does collection become available - Collection is now ready for data operations
**Path B: Automatic Creation on Document Submission**:
- User submits document specifying a collection ID
- Librarian checks if collection exists in metadata table
- If not exists: Librarian creates metadata with defaults (name=collection_id, empty description/tags)
- Librarian broadcasts "create-collection" to all storage backends
- All storage processors create collection and confirm success
- Document processing proceeds with collection now established
Both paths ensure collection exists in librarian metadata AND all storage backends before data operations.
2. **Storage Validation**: Write operations validate collection exists: 2. **Storage Validation**: Write operations validate collection exists:
- Storage processors check collection state before accepting writes - Storage processors check collection state before accepting writes
- Writes to non-existent collections return error - Writes to non-existent collections return error
- Error message directs user to create collection first - This prevents direct writes bypassing the librarian's collection creation logic
3. **Query Behavior**: Query operations handle non-existent collections gracefully: 3. **Query Behavior**: Query operations handle non-existent collections gracefully:
- Queries to non-existent collections return empty results - Queries to non-existent collections return empty results
@ -132,8 +144,10 @@ Collections follow an explicit creation pattern requiring user action before dat
- Deletes librarian metadata record only after storage cleanup complete - Deletes librarian metadata record only after storage cleanup complete
- Ensures no orphaned data remains in storage - Ensures no orphaned data remains in storage
**Key Principle**: The librarian is the single point of control for collection creation. Whether initiated by user command or document submission, the librarian ensures proper metadata tracking and storage backend synchronization before allowing data operations.
Operations required: Operations required:
- **Create Collection**: User operation via `tg-set-collection` to create new collection - **Create Collection**: User operation via `tg-set-collection` OR automatic on document submission
- **Update Collection Metadata**: User operation to modify name, description, and tags - **Update Collection Metadata**: User operation to modify name, description, and tags
- **Delete Collection**: User operation to remove collection and its data across all stores - **Delete Collection**: User operation to remove collection and its data across all stores
- **List Collections**: User operation to view collections with filtering by tags - **List Collections**: User operation to view collections with filtering by tags
@ -358,6 +372,12 @@ Comprehensive testing will cover:
- Wait for confirmations from all storage processors - Wait for confirmations from all storage processors
- Handle creation failures appropriately - Handle creation failures appropriately
2. **Document Submission Handler** (`trustgraph-flow/trustgraph/librarian/service.py` or similar)
- Check if collection exists when document submitted
- If not exists: Create collection with defaults before processing document
- Trigger same "create-collection" broadcast as `tg-set-collection`
- Ensure collection established before document flows to storage processors
### ❌ Pending Components ### ❌ Pending Components
1. **Collection State Tracking** - Need to implement in each storage backend: 1. **Collection State Tracking** - Need to implement in each storage backend: