mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-04 22:02:16 +02:00
bf805fd81e
The document saving logic used a hardcoded character slice ([:4000]) for the document summary content fed to the embedder. For UTF-8 documents (e.g., Arabic, Chinese, Japanese), characters above U+007F take multiple bytes but count as 1 character in a slice, potentially producing text that exceeds the token limit of the embedding model. Replaced the arbitrary slice with runcate_for_embedding(), which safely bounds the text using the embedding model's actual tokenizer. |
||
|---|---|---|
| .. | ||
| alembic | ||
| app | ||
| scripts | ||
| tests | ||
| .dockerignore | ||
| .env.example | ||
| .gitignore | ||
| .python-version | ||
| alembic.ini | ||
| celery_worker.py | ||
| Dockerfile | ||
| main.py | ||
| pyproject.toml | ||
| uv.lock | ||