DhruvTilva bf805fd81e fix: use token-safe truncation for document embeddings ... The document saving logic used a hardcoded character slice ([:4000]) for the document summary content fed to the embedder. For UTF-8 documents (e.g., Arabic, Chinese, Japanese), characters above U+007F take multiple bytes but count as 1 character in a slice, potentially producing text that exceeds the token limit of the embedding model. Replaced the arbitrary slice with runcate_for_embedding(), which safely bounds the text using the embedding model's actual tokenizer.		2026-06-26 00:09:11 +05:30
..
__init__.py	refactor: consolidate document processing logic and remove unused files and ETL strategies	2026-04-05 17:29:24 +05:30
_direct_converters.py	refactor: improve content extraction and encoding handling	2026-04-16 00:25:46 -07:00
_helpers.py	refactor: consolidate document processing logic and remove unused files and ETL strategies	2026-04-05 17:29:24 +05:30
_save.py	fix: use token-safe truncation for document embeddings	2026-06-26 00:09:11 +05:30
base.py	chore: ran linting	2026-03-17 04:40:46 +05:30
circleback_processor.py	feat(backend): Remove LLM summaries from document indexing	2026-06-04 00:50:19 +05:30
extension_processor.py	feat(backend): Remove LLM summaries from document indexing	2026-06-04 00:50:19 +05:30
file_processors.py	feat(etl-cache): route all file-based sources through the parse cache	2026-06-12 14:47:25 +02:00
markdown_processor.py	fix: use token-safe truncation for document embeddings	2026-06-26 00:09:11 +05:30
youtube_processor.py	feat(proxy): integrate Scrapling for enhanced web scraping capabilities	2026-06-09 00:15:10 -07:00