mirror of
https://github.com/trustgraph-ai/trustgraph.git
synced 2026-04-25 00:16:23 +02:00
5c6fe90fe2
Add universal document decoder with multi-format support using 'unstructured'. New universal decoder service powered by the unstructured library, handling DOCX, XLSX, PPTX, HTML, Markdown, CSV, RTF, ODT, EPUB and more through a single service. Tables are preserved as HTML markup for better downstream extraction. Images are stored in the librarian but excluded from the text pipeline. Configurable section grouping strategies (whole-document, heading, element-type, count, size) for non-page formats. Page-based formats (PDF, PPTX, XLSX) are automatically grouped by page. All four decoders (PDF, Mistral OCR, Tesseract OCR, universal) now share the "document-decoder" ident so they are interchangeable. PDF-only decoders fetch document metadata to check MIME type and gracefully skip unsupported formats. Librarian changes: removed MIME type whitelist validation so any document format can be ingested. Simplified routing so text/plain goes to text-load and everything else goes to document-load. Removed dual inline/streaming data paths — documents always use document_id for content retrieval. New provenance entity types (tg:Section, tg:Image) and metadata predicates (tg:elementTypes, tg:tableCount, tg:imageCount) for richer explainability. Universal decoder is in its own package (trustgraph-unstructured) and container image (trustgraph-unstructured).
18 lines
No EOL
572 B
Text
18 lines
No EOL
572 B
Text
*~
|
|
__pycache__/
|
|
env/
|
|
*/build/
|
|
*.egg-info/
|
|
*.parquet
|
|
templates/values/version.jsonnet
|
|
trustgraph-base/trustgraph/base_version.py
|
|
trustgraph-cli/trustgraph/cli_version.py
|
|
trustgraph-bedrock/trustgraph/bedrock_version.py
|
|
trustgraph-embeddings-hf/trustgraph/embeddings_hf_version.py
|
|
trustgraph-flow/trustgraph/flow_version.py
|
|
trustgraph-ocr/trustgraph/ocr_version.py
|
|
trustgraph-parquet/trustgraph/parquet_version.py
|
|
trustgraph-vertexai/trustgraph/vertexai_version.py
|
|
trustgraph-unstructured/trustgraph/unstructured_version.py
|
|
trustgraph-mcp/trustgraph/mcp_version.py
|
|
vertexai/ |