fix: large document handling and Cassandra query pagination (#969)

- Paginate heavy Cassandra reads (triples, graph/document embeddings)
  using synchronous session.execute() in run_in_executor with fetch_size
  paging, preventing materialization hang on large result sets
- Fix document stream endpoint to use workspace-scoped librarian queues
- Add decoder error handling for PDF/OCR/unstructured processors
- Add WebSocket mux guards for missing auth fields
- Add null check in librarian document streaming
- Rewrite get_document_content CLI to stream via librarian
- Add Poppler dependency to unstructured container
This commit is contained in:
cybermaggedon 2026-06-01 22:39:30 +01:00 committed by GitHub
parent 7e1fb76bc9
commit 6b1dd16f9f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
11 changed files with 166 additions and 74 deletions

View file

@ -107,7 +107,14 @@ class Processor(FlowProcessor):
# Get the source document ID
source_doc_id = v.document_id or v.metadata.id
pages = convert_from_bytes(blob)
try:
pages = convert_from_bytes(blob)
except Exception as e:
logger.error(
f"Failed to decode PDF {source_doc_id}: "
f"{type(e).__name__}: {e}"
)
return
for ix, page in enumerate(pages):