--- layout: default title: "Universal Document Decoder" parent: "Tech Specs" --- # Universal Document Decoder ## Headline Universal document decoder powered by `unstructured` — ingest any common document format through a single service with full provenance and librarian integration, recording source positions as knowledge graph metadata for end-to-end traceability. ## Problem TrustGraph currently has a PDF-specific decoder. Supporting additional formats (DOCX, XLSX, HTML, Markdown, plain text, PPTX, etc.) requires either writing a new decoder per format or adopting a universal extraction library. Each format has different structure — some are page-based, some aren't — and the provenance chain must record where in the original document each piece of extracted text originated. ## Approach ### Library: `unstructured` Use `unstructured.partition.auto.partition()` which auto-detects format from mime type or file extension and extracts structured elements (Title, NarrativeText, Table, ListItem, etc.). Each element carries metadata including: - `page_number` (for page-based formats like PDF, PPTX) - `element_id` (unique per element) - `coordinates` (bounding box for PDFs) - `text` (the extracted text content) - `category` (element type: Title, NarrativeText, Table, etc.) ### Element Types `unstructured` extracts typed elements from documents. Each element has a category and associated metadata: **Text elements:** - `Title` — section headings - `NarrativeText` — body paragraphs - `ListItem` — bullet/numbered list items - `Header`, `Footer` — page headers/footers - `FigureCaption` — captions for figures/images - `Formula` — mathematical expressions - `Address`, `EmailAddress` — contact information - `CodeSnippet` — code blocks (from markdown) **Tables:** - `Table` — structured tabular data. `unstructured` provides both `element.text` (plain text) and `element.metadata.text_as_html` (full HTML `` with rows, columns, and headers preserved). For formats with explicit table structure (DOCX, XLSX, HTML), the extraction is highly reliable. For PDFs, table detection depends on the `hi_res` strategy with layout analysis. **Images:** - `Image` — embedded images detected via layout analysis (requires `hi_res` strategy). With `extract_image_block_to_payload=True`, returns the image data as base64 in `element.metadata.image_base64`. OCR text from the image is available in `element.text`. ### Table Handling Tables are a first-class output. When the decoder encounters a `Table` element, it preserves the HTML structure rather than flattening to plain text. This gives the downstream LLM extractor much better input for pulling structured knowledge out of tabular data. The page/section text is assembled as follows: - Text elements: plain text, joined with newlines - Table elements: HTML table markup from `text_as_html`, wrapped in a `
` marker so the LLM can distinguish tables from narrative For example, a page with a heading, paragraph, and table produces: ``` Financial Overview Revenue grew 15% year-over-year driven by enterprise adoption.
QuarterRevenueGrowth
Q1$12M12%
Q2$14M17%
``` This preserves table structure through chunking and into the extraction pipeline, where the LLM can extract relationships directly from structured cells rather than guessing at column alignment from whitespace. ### Image Handling Images are extracted and stored in the librarian as child documents with `document_type="image"` and a `urn:image:{uuid}` ID. They get provenance triples with type `tg:Image`, linked to their parent page/section via `prov:wasDerivedFrom`. Image metadata (coordinates, dimensions, element_id) is recorded in provenance. **Crucially, images are NOT emitted as TextDocument outputs.** They are stored only — not sent downstream to the chunker or any text processing pipeline. This is intentional: 1. There is no image processing pipeline yet (vision model integration is future work) 2. Feeding base64 image data or OCR fragments into the text extraction pipeline would produce garbage KG triples Images are also excluded from the assembled page text — any `Image` elements are silently skipped when concatenating element texts for a page/section. The provenance chain records that images exist and where they appeared in the document, so they can be picked up by a future image processing pipeline without re-ingesting the document. #### Future work - Route `tg:Image` entities to a vision model for description, diagram interpretation, or chart data extraction - Store image descriptions as text child documents that feed into the standard chunking/extraction pipeline - Link extracted knowledge back to source images via provenance ### Section Strategies For page-based formats (PDF, PPTX, XLSX), elements are always grouped by page/slide/sheet first. For non-page formats (DOCX, HTML, Markdown, etc.), the decoder needs a strategy for splitting the document into sections. This is runtime-configurable via `--section-strategy`. Each strategy is a grouping function over the list of `unstructured` elements. The output is a list of element groups; the rest of the pipeline (text assembly, librarian storage, provenance, TextDocument emission) is identical regardless of strategy. #### `whole-document` (default) Emit the entire document as a single section. Let the downstream chunker handle all splitting. - Simplest approach, good baseline - May produce very large TextDocument for big files, but the chunker handles that - Best when you want maximum context per section #### `heading` Split at heading elements (`Title`). Each section is a heading plus all content until the next heading of equal or higher level. Nested headings create nested sections. - Produces topically coherent units - Works well for structured documents (reports, manuals, specs) - Gives the extraction LLM heading context alongside content - Falls back to `whole-document` if no headings are found #### `element-type` Split when the element type changes significantly — specifically, start a new section at transitions between narrative text and tables. Consecutive elements of the same broad category (text, text, text or table, table) stay grouped. - Keeps tables as standalone sections - Good for documents with mixed content (reports with data tables) - Tables get dedicated extraction attention #### `count` Group a fixed number of elements per section. Configurable via `--section-element-count` (default: 20). - Simple and predictable - Doesn't respect document structure - Useful as a fallback or for experimentation #### `size` Accumulate elements until a character limit is reached, then start a new section. Respects element boundaries — never splits mid-element. Configurable via `--section-max-size` (default: 4000 characters). - Produces roughly uniform section sizes - Respects element boundaries (unlike the downstream chunker) - Good compromise between structure and size control - If a single element exceeds the limit, it becomes its own section #### Page-based format interaction For page-based formats, the page grouping always takes priority. Section strategies can optionally apply *within* a page if it's very large (e.g. a PDF page with an enormous table), controlled by `--section-within-pages` (default: false). When false, each page is always one section regardless of size. ### Format Detection The decoder needs to know the document's mime type to pass to `unstructured`'s `partition()`. Two paths: - **Librarian path** (`document_id` set): fetch document metadata from the librarian first — this gives us the `kind` (mime type) that was recorded at upload time. Then fetch document content. Two librarian calls, but the metadata fetch is lightweight. - **Inline path** (backward compat, `data` set): no metadata available on the message. Use `python-magic` to detect format from content bytes as a fallback. No changes to the `Document` schema are needed — the librarian already stores the mime type. ### Architecture A single `universal-decoder` service that: 1. Receives a `Document` message (inline or via librarian reference) 2. If librarian path: fetch document metadata (get mime type), then fetch content. If inline path: detect format from content bytes. 3. Calls `partition()` to extract elements 4. Groups elements: by page for page-based formats, by configured section strategy for non-page formats 5. For each page/section: - Generates a `urn:page:{uuid}` or `urn:section:{uuid}` ID - Assembles page text: narrative as plain text, tables as HTML, images skipped - Computes character offsets for each element within the page text - Saves to librarian as child document - Emits provenance triples with positional metadata - Sends `TextDocument` downstream for chunking 6. For each image element: - Generates a `urn:image:{uuid}` ID - Saves image data to librarian as child document - Emits provenance triples (stored only, not sent downstream) ### Format Handling | Format | Mime Type | Page-based | Notes | |----------|------------------------------------|------------|--------------------------------| | PDF | application/pdf | Yes | Per-page grouping | | DOCX | application/vnd.openxmlformats... | No | Uses section strategy | | PPTX | application/vnd.openxmlformats... | Yes | Per-slide grouping | | XLSX/XLS | application/vnd.openxmlformats... | Yes | Per-sheet grouping | | HTML | text/html | No | Uses section strategy | | Markdown | text/markdown | No | Uses section strategy | | Plain | text/plain | No | Uses section strategy | | CSV | text/csv | No | Uses section strategy | | RST | text/x-rst | No | Uses section strategy | | RTF | application/rtf | No | Uses section strategy | | ODT | application/vnd.oasis... | No | Uses section strategy | | TSV | text/tab-separated-values | No | Uses section strategy | ### Provenance Metadata Each page/section entity records positional metadata as provenance triples in `GRAPH_SOURCE`, enabling full traceability from KG triples back to source document positions. #### Existing fields (already in `derived_entity_triples`) - `page_number` — page/sheet/slide number (1-indexed, page-based only) - `char_offset` — character offset of this page/section within the full document text - `char_length` — character length of this page/section's text #### New fields (extend `derived_entity_triples`) - `mime_type` — original document format (e.g. `application/pdf`) - `element_types` — comma-separated list of `unstructured` element categories found in this page/section (e.g. "Title,NarrativeText,Table") - `table_count` — number of tables in this page/section - `image_count` — number of images in this page/section These require new TG namespace predicates: ``` TG_SECTION_TYPE = "https://trustgraph.ai/ns/Section" TG_IMAGE_TYPE = "https://trustgraph.ai/ns/Image" TG_ELEMENT_TYPES = "https://trustgraph.ai/ns/elementTypes" TG_TABLE_COUNT = "https://trustgraph.ai/ns/tableCount" TG_IMAGE_COUNT = "https://trustgraph.ai/ns/imageCount" ``` Image URN scheme: `urn:image:{uuid}` (`TG_MIME_TYPE` already exists.) #### New entity type For non-page formats (DOCX, HTML, Markdown, etc.) where the decoder emits the whole document as a single unit rather than splitting by page, the entity gets a new type: ``` TG_SECTION_TYPE = "https://trustgraph.ai/ns/Section" ``` This distinguishes sections from pages when querying provenance: | Entity | Type | When used | |----------|-----------------------------|----------------------------------------| | Document | `tg:Document` | Original uploaded file | | Page | `tg:Page` | Page-based formats (PDF, PPTX, XLSX) | | Section | `tg:Section` | Non-page formats (DOCX, HTML, MD, etc) | | Image | `tg:Image` | Embedded images (stored, not processed)| | Chunk | `tg:Chunk` | Output of chunker | | Subgraph | `tg:Subgraph` | KG extraction output | The type is set by the decoder based on whether it's grouping by page or emitting a whole-document section. `derived_entity_triples` gains an optional `section` boolean parameter — when true, the entity is typed as `tg:Section` instead of `tg:Page`. #### Full provenance chain ``` KG triple → subgraph (extraction provenance) → chunk (char_offset, char_length within page) → page/section (page_number, char_offset, char_length within doc, mime_type, element_types) → document (original file in librarian) ``` Every link is a set of triples in the `GRAPH_SOURCE` named graph. ### Service Configuration Command-line arguments: ``` --strategy Partitioning strategy: auto, hi_res, fast (default: auto) --languages Comma-separated OCR language codes (default: eng) --section-strategy Section grouping: whole-document, heading, element-type, count, size (default: whole-document) --section-element-count Elements per section for 'count' strategy (default: 20) --section-max-size Max chars per section for 'size' strategy (default: 4000) --section-within-pages Apply section strategy within pages too (default: false) ``` Plus the standard `FlowProcessor` and librarian queue arguments. ### Flow Integration The universal decoder occupies the same position in the processing flow as the current PDF decoder: ``` Document → [universal-decoder] → TextDocument → [chunker] → Chunk → ... ``` It registers: - `input` consumer (Document schema) - `output` producer (TextDocument schema) - `triples` producer (Triples schema) - Librarian request/response (for fetch and child document storage) ### Deployment - New container: `trustgraph-flow-universal-decoder` - Dependency: `unstructured[all-docs]` (includes PDF, DOCX, PPTX, etc.) - Can run alongside or replace the existing PDF decoder depending on flow configuration - The existing PDF decoder remains available for environments where `unstructured` dependencies are too heavy ### What Changes | Component | Change | |------------------------------|-------------------------------------------------| | `provenance/namespaces.py` | Add `TG_SECTION_TYPE`, `TG_IMAGE_TYPE`, `TG_ELEMENT_TYPES`, `TG_TABLE_COUNT`, `TG_IMAGE_COUNT` | | `provenance/triples.py` | Add `mime_type`, `element_types`, `table_count`, `image_count` kwargs | | `provenance/__init__.py` | Export new constants | | New: `decoding/universal/` | New decoder service module | | `setup.cfg` / `pyproject` | Add `unstructured[all-docs]` dependency | | Docker | New container image | | Flow definitions | Wire universal-decoder as document input | ### What Doesn't Change - Chunker (receives TextDocument, works as before) - Downstream extractors (receive Chunk, unchanged) - Librarian (stores child documents, unchanged) - Schema (Document, TextDocument, Chunk unchanged) - Query-time provenance (unchanged) ## Risks - `unstructured[all-docs]` has heavy dependencies (poppler, tesseract, libreoffice for some formats). Container image will be larger. Mitigation: offer a `[light]` variant without OCR/office deps. - Some formats may produce poor text extraction (scanned PDFs without OCR, complex XLSX layouts). Mitigation: configurable `strategy` parameter, and the existing Mistral OCR decoder remains available for high-quality PDF OCR. - `unstructured` version updates may change element metadata. Mitigation: pin version, test extraction quality per format.