Commit graph

  • 2670c5f80e Fixing tests Cyber MacGeddon 2026-03-09 10:46:03 +00:00
  • d8a4a8c57f Fixing tests Cyber MacGeddon 2026-03-09 10:36:52 +00:00
  • 9782421b1f Fixing tests Cyber MacGeddon 2026-03-09 10:33:33 +00:00
  • 2cb29380fa Fixing tests Cyber MacGeddon 2026-03-09 10:33:24 +00:00
  • f261604991 Fixing tests Cyber MacGeddon 2026-03-09 10:30:46 +00:00
  • b7894c7088 Fixing tests Cyber MacGeddon 2026-03-09 10:27:14 +00:00
  • 8d25f4d4a8 Fixing tests Cyber MacGeddon 2026-03-09 10:24:53 +00:00
  • f89e002614 Fixing tests Cyber MacGeddon 2026-03-09 10:22:08 +00:00
  • a3ae54a11d Fixing tests Cyber MacGeddon 2026-03-09 10:18:53 +00:00
  • 7bcced37f2 Fixing tests Cyber MacGeddon 2026-03-09 10:17:48 +00:00
  • df12467510 Fixing tests Cyber MacGeddon 2026-03-09 10:14:01 +00:00
  • 61a100dec9 Fixing tests Cyber MacGeddon 2026-03-09 10:11:05 +00:00
  • fe82b844c6 Fixing tests Cyber MacGeddon 2026-03-09 10:09:50 +00:00
  • 9a4bfafb25 Fixing tests Cyber MacGeddon 2026-03-09 10:08:44 +00:00
  • 4e3db11323 Fixing tests Cyber MacGeddon 2026-03-09 10:05:39 +00:00
  • dcee1b8de2 Update tests Cyber MacGeddon 2026-03-09 10:01:55 +00:00
  • 356d7f75ac Implementation Cyber MacGeddon 2026-03-09 09:59:57 +00:00
  • 4fa7cc7d7c
    Fix/embeddings integration 2 (#670) v2.1.5 cybermaggedon 2026-03-08 19:42:26 +00:00
  • 919b760c05
    Update embeddings integration for new batch embeddings interfaces (#669) cybermaggedon 2026-03-08 19:41:52 +00:00
  • a4e0854d11 Fix embeddings integration Cyber MacGeddon 2026-03-08 19:40:47 +00:00
  • 22ba73e680 Fix embeddings integration Cyber MacGeddon 2026-03-08 19:38:08 +00:00
  • ea95effbf2 Fix embeddings integration Cyber MacGeddon 2026-03-08 19:35:38 +00:00
  • a7ae5a8823 Fix embeddings integration Cyber MacGeddon 2026-03-08 19:30:39 +00:00
  • aca5ee5653 Fix vector extraction Cyber MacGeddon 2026-03-08 18:52:07 +00:00
  • 0a2ce47a88
    Batch embeddings (#668) cybermaggedon 2026-03-08 18:36:54 +00:00
  • 3d4f859aa2 Implementation Cyber MacGeddon 2026-03-08 18:34:22 +00:00
  • 87ef89ef7b Base Service (trustgraph-base/trustgraph/base/embeddings_service.py): - Changed on_request to use request.texts Cyber MacGeddon 2026-03-08 18:10:20 +00:00
  • 3bf8a65409
    Fix tests (#666) cybermaggedon 2026-03-07 23:38:09 +00:00
  • 35dac42c73 Fixing tests Cyber MacGeddon 2026-03-07 23:36:02 +00:00
  • 2d47102d70 Fixing tests Cyber MacGeddon 2026-03-07 23:32:25 +00:00
  • b40076ffe1 Fixing tests Cyber MacGeddon 2026-03-07 23:26:27 +00:00
  • 6ebddb85d0 Fixing tests Cyber MacGeddon 2026-03-07 23:26:00 +00:00
  • 2503791b9a Fixing tests Cyber MacGeddon 2026-03-07 23:24:00 +00:00
  • 24bbe94136
    Document chunks not stored in vector store (#665) v2.1.4 cybermaggedon 2026-03-07 23:10:45 +00:00
  • da37738478 Make sure nul output is specific Cyber MacGeddon 2026-03-07 21:09:08 +00:00
  • 04e03c0707 - Schema - ChunkEmbeddings now uses chunk_id: str instead of chunk: bytes - Schema - DocumentEmbeddingsResponse now returns chunk_ids: list[str] instead of chunks - Translators - Updated to serialize/deserialize chunk_id - Clients - DocumentEmbeddingsClient.query() returns chunk_ids - SDK/API - flow.py, socket_client.py, bulk_client.py updated - Document embeddings service - Stores chunk_id (document ID) instead of chunk text - Storage writers - Qdrant, Milvus, Pinecone store chunk_id in payload - Query services - Return chunk_id from vector store searches - Gateway dispatchers - Serialize chunk_id in API responses - Document RAG - Added librarian client to fetch chunk content from Garage using chunk_ids - CLI tools - Updated all three tools: - invoke_document_embeddings.py - displays chunk_ids, removed max_chunk_length - save_doc_embeds.py - exports chunk_id - load_doc_embeds.py - imports chunk_id Cyber MacGeddon 2026-03-07 20:03:23 +00:00
  • 0c62f0e0fa CLI and API changes Cyber MacGeddon 2026-03-07 15:02:36 +00:00
  • 9f90075367 Doc RAG updates Cyber MacGeddon 2026-03-07 15:01:52 +00:00
  • 01eef99218 Tech specc Cyber MacGeddon 2026-03-07 14:59:24 +00:00
  • be358efe67
    Fix tests (#663) cybermaggedon 2026-03-06 12:40:02 +00:00
  • 50d330efe9 Fix tests Cyber MacGeddon 2026-03-06 12:38:54 +00:00
  • 2b9232917c
    Fix/extraction prov (#662) v2.1.3 cybermaggedon 2026-03-06 12:23:58 +00:00
  • d3ae8f7d6f Fixed quotaed triples Cyber MacGeddon 2026-03-06 00:01:01 +00:00
  • fe36f7db81 Fixed quotaed triples Cyber MacGeddon 2026-03-05 23:43:27 +00:00
  • 86ca3edbfc Fixed quotaed triples Cyber MacGeddon 2026-03-05 21:38:12 +00:00
  • abff65431a Fixed quotaed triples Cyber MacGeddon 2026-03-05 21:29:20 +00:00
  • 2a7b57091f Reified triples support Cyber MacGeddon 2026-03-05 21:29:08 +00:00
  • 78dba08ddc Reified triples support Cyber MacGeddon 2026-03-05 21:24:04 +00:00
  • d6d74fb2ef Reified triples support Cyber MacGeddon 2026-03-05 21:17:07 +00:00
  • e2b481072d Fix triple serialiseation Cyber MacGeddon 2026-03-05 21:05:20 +00:00
  • 8a109fcf43 1. Updated triple_provenance_triples() in triples.py: - Now accepts a Triple object directly - Creates the reification triple using TRIPLE term type: stmt_uri tg:reifies <<extracted_triple>> - Includes it in the returned provenance triples Cyber MacGeddon 2026-03-05 19:55:09 +00:00
  • 9d1d466014 Fix missing labels Cyber MacGeddon 2026-03-05 19:16:01 +00:00
  • c3471df6ce Fix embeddings to chunk linkage Cyber MacGeddon 2026-03-05 19:15:33 +00:00
  • 7527bd787b Fix doc IRIs Cyber MacGeddon 2026-03-05 18:37:40 +00:00
  • cd5580be59
    Extract-time provenance (#661) cybermaggedon 2026-03-05 18:36:10 +00:00
  • 8b2455906d Fixing some tests Cyber MacGeddon 2026-03-05 17:49:04 +00:00
  • 8da0229304 Fixing some tests Cyber MacGeddon 2026-03-05 17:47:43 +00:00
  • 877f17f886 Fixing some tests Cyber MacGeddon 2026-03-05 17:43:57 +00:00
  • 4e5cd91eb1 1. Shared Provenance Module - URI generators, namespace constants, triple builders, vocabulary bootstrap 2. Librarian - Emits document metadata to graph on processing initiation (vocabulary bootstrap + PROV-O triples) 3. PDF Extractor - Saves pages as child documents, emits parent-child provenance edges, forwards page IDs 4. Chunker - Saves chunks as child documents, emits provenance edges, forwards chunk ID + content 5. Knowledge Extractors (both definitions and relationships): - Link entities to chunks via SUBJECT_OF (not top-level document) - Removed duplicate metadata emission (now handled by librarian) - Get chunk_doc_id and chunk_uri from incoming Chunk message 6. Embedding Provenance: - EntityContext schema has chunk_id field - EntityEmbeddings schema has chunk_id field - Definitions extractor sets chunk_id when creating EntityContext - Graph embeddings processor passes chunk_id through to EntityEmbeddings Cyber MacGeddon 2026-03-05 17:17:21 +00:00
  • d8f0a576af
    Document API updates (#660) v2.1.2 cybermaggedon 2026-03-05 15:20:45 +00:00
  • 150190c8d8 Add CLI args Cyber MacGeddon 2026-03-05 15:16:43 +00:00
  • 285d98aa22 Fix chunk minimum confusion Cyber MacGeddon 2026-03-05 15:15:12 +00:00
  • 5a64aebc59 Doc streaming from librarian Cyber MacGeddon 2026-03-05 14:33:39 +00:00
  • a630e143ef
    Incremental / large document loading (#659) v2.1.1 cybermaggedon 2026-03-04 16:57:58 +00:00
  • 48d7b2a0ea Fixing tests Cyber MacGeddon 2026-03-04 16:53:45 +00:00
  • 39681e3003 Fixing tests Cyber MacGeddon 2026-03-04 16:46:43 +00:00
  • 81812ee262 Move chunker/librarian comms to base class Cyber MacGeddon 2026-03-04 16:37:19 +00:00
  • 9df348489b Move chunker/librarian comms to base class Cyber MacGeddon 2026-03-04 16:36:00 +00:00
  • 64447f6d60 Move chunker/librarian comms to base class Cyber MacGeddon 2026-03-04 16:35:53 +00:00
  • 279e4ef0b9 Fixed text loader to use librarian as a source Cyber MacGeddon 2026-03-04 16:33:47 +00:00
  • 57862fba8e Fixed text loader to use librarian as a source Cyber MacGeddon 2026-03-04 16:32:13 +00:00
  • 410e44bec7 Use Pulsar comms, not gateway Cyber MacGeddon 2026-03-04 16:29:13 +00:00
  • 04e17f8c36 Fixing librarian API errors Cyber MacGeddon 2026-03-04 16:23:02 +00:00
  • 7edc3f72fb Minor fixes Cyber MacGeddon 2026-03-04 16:09:03 +00:00
  • 80708c6a9c Fix schema mapping Cyber MacGeddon 2026-03-04 15:59:26 +00:00
  • ff5e6d7df1 Librarian fixed Cyber MacGeddon 2026-03-04 15:53:20 +00:00
  • fa0c3d9a59 Remove load_pdf and load_text Cyber MacGeddon 2026-03-04 15:35:18 +00:00
  • 1b61ad1581 - Document table: Added parent_id and document_type columns with index - Document schema (knowledge/document.py): Added document_id field for streaming retrieval - Librarian operations: - add-child-document for extracted PDF pages - list-children to get child documents - stream-document for chunked content retrieval - Cascade delete removes children when parent is deleted - list-documents filters children by default - PDF decoder (decoding/pdf/pdf_decoder.py): Updated to stream large documents from librarian API to temp file - Librarian service (librarian/service.py): Sends document_id instead of content for large PDFs (>2MB) - Deprecated tools (load_pdf.py, load_text.py): Added deprecation warnings directing users to tg-add-library-document + tg-start-library-processing Cyber MacGeddon 2026-03-04 15:33:05 +00:00
  • 6495c9b78d Lower chunk transition point Cyber MacGeddon 2026-03-04 15:21:32 +00:00
  • fbedbb3eab - Schema extended with UploadSession, UploadProgress, and new request/response fields - Librarian methods: begin_upload, upload_chunk, complete_upload, abort_upload, get_upload_status, list_uploads - Service routing for all new operations - Python SDK with transparent chunked upload: - add_document() auto-switches to chunked for files > 10MB - Progress callback support (on_progress) - get_pending_uploads(), get_upload_status(), abort_upload(), resume_upload() Cyber MacGeddon 2026-03-04 15:16:39 +00:00
  • d858616319 BlobStore (trustgraph-flow/trustgraph/librarian/blob_store.py): - get_stream() - yields document content in chunks for streaming retrieval - create_multipart_upload() - initializes S3 multipart upload, returns upload_id - upload_part() - uploads a single part, returns etag - complete_multipart_upload() - finalizes upload with part etags - abort_multipart_upload() - cancels and cleans up Cyber MacGeddon 2026-03-04 15:03:46 +00:00
  • b0fb887433 Tech spec Cyber MacGeddon 2026-03-04 12:19:42 +00:00
  • a38ca9474f
    Tool services - dynamically pluggable tool implementations for agent frameworks (#658) v2.1.0 cybermaggedon 2026-03-04 14:51:32 +00:00
  • 221b9b81dd Update unit tests for tool services Cyber MacGeddon 2026-03-04 14:50:44 +00:00
  • cb6a6cbf98 Updated tech spec Cyber MacGeddon 2026-03-04 14:45:29 +00:00
  • 672e62fdd0 llama special case in joke demo Cyber MacGeddon 2026-03-04 14:27:02 +00:00
  • 73b7917d31 llama special case in joke demo Cyber MacGeddon 2026-03-04 14:25:54 +00:00
  • 5f53cf1848 Specify request/response queues Cyber MacGeddon 2026-03-04 14:25:06 +00:00
  • ade19c7d4b Fix invocation Cyber MacGeddon 2026-03-04 14:20:33 +00:00
  • 306ffd3dfa Fix invocation Cyber MacGeddon 2026-03-04 14:18:31 +00:00
  • a625efb94b Fixing service wiring Cyber MacGeddon 2026-03-04 14:12:11 +00:00
  • 3d61ecd060 Fixing service wiring Cyber MacGeddon 2026-03-04 14:09:02 +00:00
  • 942f7854f7 Base service Cyber MacGeddon 2026-03-04 14:08:49 +00:00
  • 02c81d4740 Fixed tool service import Cyber MacGeddon 2026-03-04 14:00:01 +00:00
  • db8d19e7e9 Joke service, for testing Cyber MacGeddon 2026-03-04 13:59:48 +00:00
  • 98ea1fd68a Base class Cyber MacGeddon 2026-03-04 13:59:24 +00:00
  • 311c4de970 Tool service implementation Cyber MacGeddon 2026-03-04 13:40:36 +00:00
  • fe3977b5d1 New schema Cyber MacGeddon 2026-03-04 13:34:53 +00:00
  • 0b83c08ae4
    Use model in Azure LLM integration (#657) cybermaggedon 2026-03-04 12:06:06 +00:00
  • ae27eccbed Use model in Azure LLM integration Cyber MacGeddon 2026-03-04 12:05:40 +00:00