feat: enhance document formatting and context management for LLM tools

- Introduced dynamic character budget calculation for document formatting based on model's context window.
- Updated `format_documents_for_context` to respect character limits and improve output quality.
- Added `max_input_tokens` parameter to various functions to facilitate context-aware processing.
- Enhanced error handling for context overflow in LLM router service.

surfsense_backend/app/agents/new_chat/tools/registry.py

@@ -118,6 +118,7 @@ BUILTIN_TOOLS: list[ToolDefinition] = [
             # Optional: dynamically discovered connectors/document types
             available_connectors=deps.get("available_connectors"),
             available_document_types=deps.get("available_document_types"),
+            max_input_tokens=deps.get("max_input_tokens"),
         ),
         requires=["search_space_id", "db_session", "connector_service"],
         # Note: available_connectors and available_document_types are optional