DhruvTilva bf805fd81e fix: use token-safe truncation for document embeddings ... The document saving logic used a hardcoded character slice ([:4000]) for the document summary content fed to the embedder. For UTF-8 documents (e.g., Arabic, Chinese, Japanese), characters above U+007F take multiple bytes but count as 1 character in a slice, potentially producing text that exceeds the token limit of the embedding model. Replaced the arbitrary slice with runcate_for_embedding(), which safely bounds the text using the embedding model's actual tokenizer.		2026-06-26 00:09:11 +05:30
..
celery_tasks	continue indexing when notification creation fails	2026-06-17 15:06:05 +02:00
chat	feat: enable streaming in LLM bundle construction	2026-06-18 23:39:55 +05:30
connector_indexers	fix(chunks): set position on remaining chunk insert paths	2026-06-12 18:53:08 +02:00
document_processors	fix: use token-safe truncation for document embeddings	2026-06-26 00:09:11 +05:30
__init__.py	feat: SurfSense v0.0.6 init	2025-03-14 18:53:14 -07:00
composio_indexer.py	chore: ran frontend and backend linting	2026-02-01 22:54:25 +05:30