SurfSense/INSTALLATION_LOCAL_LLM.md

Local LLM Installation Guide

Complete guide for deploying SurfSense with local Ollama models (Mistral NeMo + TildeOpen).

Prerequisites

Hardware Requirements

• RAM: 32GB+ recommended (minimum 24GB)
• Disk Space: 50GB+ free space
• CPU: Modern multi-core processor (GPU optional but not required)
• Network: Stable internet for initial model downloads

Software Requirements

• OS: Ubuntu 20.04+, Debian 11+, or similar Linux distribution
• Python: 3.10 or higher
• Node.js: 18+ (for frontend)
• Git: For repository management

Installation Steps

Step 1: Install Ollama

Ollama provides local LLM inference with automatic model management.

# Download and install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Enable Ollama service to start on boot
sudo systemctl enable ollama

# Start Ollama service
sudo systemctl start ollama

# Verify Ollama is running
sudo systemctl status ollama

# Test Ollama API
curl http://localhost:11434/api/tags

Expected output: JSON response with empty model list (we'll add models next).

Step 2: Download Required Models

Download Mistral NeMo (7.1GB) and TildeOpen (21GB).

# Download Mistral NeMo base model
ollama pull mistral-nemo

# This will take 5-10 minutes depending on your connection
# Progress will be shown in the terminal

# Download TildeOpen Latvian model
ollama pull tildeopen:30b-q5_k_m

# This will take 10-20 minutes (21GB download)

# Verify models are installed
ollama list

Expected output:

NAME                   ID              SIZE      MODIFIED
mistral-nemo:latest    e7e06d107c6c    7.1 GB    X minutes ago
tildeopen:30b-q5_k_m   7f0adb68ec7d    21 GB     X minutes ago

Step 3: Create Optimized Mistral NeMo Model

The default mistral-nemo has a 4K context window. We need to create a custom version with 128K context.

# Create Modelfile for 128K context
cat > /tmp/mistral-nemo-128k.modelfile << 'EOF'
FROM mistral-nemo:latest

# Set context window to 128K tokens (Mistral NeMo maximum)
PARAMETER num_ctx 131072

# Keep other parameters optimized for RAG
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER top_k 40
EOF

# Create the custom model
ollama create mistral-nemo:128k -f /tmp/mistral-nemo-128k.modelfile

# Verify the model was created
ollama list | grep "128k"

Expected output:

mistral-nemo:128k      7d56f30917ac    7.1 GB    X seconds ago

Step 4: Verify Model Configuration

Test that the 128K context is properly configured.

# Check the model configuration
ollama show mistral-nemo:128k --modelfile | grep num_ctx

Expected output:

PARAMETER num_ctx 131072

Step 5: Configure SurfSense Backend

Navigate to your SurfSense backend directory and configure the LLM settings.

# Navigate to backend config directory
cd /opt/SurfSense/surfsense_backend/app/config

# Copy the template to create your config
cp global_llm_config.yaml.template global_llm_config.yaml

# Edit the config file
nano global_llm_config.yaml

Replace the placeholder with your actual Gemini API key:

api_key: "${GEMINI_API_KEY}"  # Change this line

To:

api_key: "your_actual_gemini_api_key_here"

Save and exit (Ctrl+X, then Y, then Enter in nano).

Step 6: Update Backend Code

The backend needs two Python files patched to handle the correct context window.

Patch 1: /opt/SurfSense/surfsense_backend/app/agents/researcher/utils.py

Find the get_model_context_window() function and update it:

def get_model_context_window(model_name: str) -> int:
    """Get the total context window size for a model (input + output tokens)."""

    # Override for Ollama models with known incorrect LiteLLM values
    if "mistral-nemo" in model_name.lower():
        return 131072  # Mistral NeMo actual context window: 128K tokens

    try:
        model_info = get_model_info(model_name)
        context_window = model_info.get("max_input_tokens", 4096)
        return context_window
    except Exception as e:
        print(
            f"Warning: Could not get model info for {model_name}, using default 4096 tokens. Error: {e}"
        )
        return 4096

Patch 2: /opt/SurfSense/surfsense_backend/app/utils/document_converters.py

Apply the same change to the get_model_context_window() function in this file.

Or use this automated script:

# Automated patching script
cd /opt/SurfSense/surfsense_backend

# Backup original files
cp app/agents/researcher/utils.py app/agents/researcher/utils.py.backup
cp app/utils/document_converters.py app/utils/document_converters.py.backup

# Apply patches (download from repository or apply manually)
# See MIGRATION_LOCAL_LLM.md for detailed patch content

Step 7: Set Environment Variables

Ensure your environment has the necessary variables.

# Edit backend environment file
nano /opt/SurfSense/surfsense_backend/.env

Add or verify these lines:

GEMINI_API_KEY=your_gemini_api_key_here
OLLAMA_BASE_URL=http://localhost:11434

Step 8: Restart Services

Restart all SurfSense services to apply changes.

# Restart backend
sudo systemctl restart surfsense

# Restart frontend (if separate service)
sudo systemctl restart surfsense-frontend

# Restart Celery workers
sudo systemctl restart surfsense-celery

# Restart Celery beat
sudo systemctl restart surfsense-celery-beat

# Wait a few seconds for services to start
sleep 10

Step 9: Verify Services

Check that all services started successfully.

# Check Ollama
systemctl status ollama | head -10

# Check SurfSense backend
systemctl status surfsense | head -10

# Check frontend
systemctl status surfsense-frontend | head -10

# Check Celery
systemctl status surfsense-celery | head -10

# Test Ollama API
curl http://localhost:11434/api/tags

# Test backend health endpoint
curl http://localhost:8000/health

All services should show "active (running)".

Step 10: Test the System

Open your SurfSense instance in a browser and test with queries.

Test 1: English Query

1. Navigate to https://your-domain.com
2. Enter an English question about your documents
3. Expected: Response in ~5 seconds

Test 2: Latvian Query

1. Enter a Latvian question: "Kāds ir galvenais mērķis?"
2. Expected: Response in ~23 seconds (includes grammar check)

Test 3: Monitor Logs

# Watch backend logs in real-time
journalctl -u surfsense -f

# Watch Ollama logs
journalctl -u ollama -f

Look for:

• "Context window=131072" (not 1024000 or 4096)
• No truncation warnings
• Successful response generation
• No timeout errors

Troubleshooting

Issue: Ollama service won't start

# Check Ollama logs
journalctl -u ollama -n 50

# Try manual start
ollama serve

# Check port availability
sudo lsof -i :11434

Issue: Model not found

# List installed models
ollama list

# Re-pull if missing
ollama pull mistral-nemo
ollama create mistral-nemo:128k -f /tmp/mistral-nemo-128k.modelfile

Issue: Out of memory errors

# Check available RAM
free -h

# Check Ollama memory usage
ps aux | grep ollama

# Consider reducing concurrent models or using smaller quantizations

Issue: Context still truncating to 4K

# Verify model configuration
ollama show mistral-nemo:128k --modelfile | grep num_ctx

# Should show: PARAMETER num_ctx 131072

# If not, recreate the model:
ollama rm mistral-nemo:128k
ollama create mistral-nemo:128k -f /tmp/mistral-nemo-128k.modelfile

Issue: Backend not connecting to Ollama

# Test Ollama from backend server
curl http://localhost:11434/api/tags

# Check firewall
sudo ufw status

# Verify OLLAMA_BASE_URL in .env
grep OLLAMA /opt/SurfSense/surfsense_backend/.env

Issue: Queries still hitting Gemini API instead of local models

# Check backend configuration
cat /opt/SurfSense/surfsense_backend/app/config/global_llm_config.yaml

# Verify model_name is "mistral-nemo:128k"
grep "model_name.*mistral" /opt/SurfSense/surfsense_backend/app/config/global_llm_config.yaml

# Restart backend
sudo systemctl restart surfsense

Performance Tuning

For 24GB RAM Systems

If you have less than 32GB RAM, use smaller models:

# Use smaller TildeOpen quantization
ollama pull tildeopen:30b-q4_k_m  # Instead of q5_k_m

# Or skip grammar checking by disabling in config

For Faster Inference

# Use GPU acceleration (if available)
# Ollama automatically detects and uses CUDA/ROCm GPUs

# Check GPU usage
nvidia-smi  # For NVIDIA GPUs

For Production Deployment

# Set Ollama to use specific GPU
CUDA_VISIBLE_DEVICES=0 ollama serve

# Limit concurrent requests in SurfSense config
# Edit systemd service file to set worker limits

Maintenance

Updating Models

# Check for model updates
ollama list

# Update a specific model
ollama pull mistral-nemo:latest

# Recreate optimized version
ollama create mistral-nemo:128k -f /tmp/mistral-nemo-128k.modelfile

Cleaning Up Old Models

# Remove old model versions
ollama rm mistral-nemo:latest  # Keep only :128k version

# Free up disk space
ollama prune  # Removes unused layers

Monitoring

# Monitor RAM usage
watch -n 1 free -h

# Monitor Ollama
journalctl -u ollama -f

# Monitor backend
journalctl -u surfsense -f

Security Considerations

1. API Key Security: Never commit global_llm_config.yaml with real API keys to git
2. Firewall: Ensure Ollama port 11434 is not exposed to internet
3. Updates: Keep Ollama and models updated for security patches
4. Logs: Regularly rotate and clean logs to prevent disk filling

Backup Recommendations

# Backup Ollama models directory
tar -czf ollama-models-backup.tar.gz /usr/share/ollama/.ollama/models/

# Backup SurfSense configuration
tar -czf surfsense-config-backup.tar.gz /opt/SurfSense/surfsense_backend/app/config/

# Store backups securely offsite

Support

For issues or questions:

• GitHub Issues: https://github.com/okapteinis/SurfSense/issues
• Documentation: See MIGRATION_LOCAL_LLM.md
• Email: ojars@kapteinis.lv

License

This installation guide is part of the SurfSense project.

---

Installation Complete! You now have a fully local European AI stack with 95% cost reduction and improved performance.