SurfSense/GITHUB_CONNECTOR_FIX.md

GitHub Connector Indexing Fix

Problem Summary

The GitHub connector was experiencing issues where:

1. Documents were not appearing in "Manage Documents" after indexing
2. The "Last Indexed" timestamp would show initially but revert to "Never" after page refresh
3. Queries always returned 0 related documents despite successful indexing logs

Root Causes Identified

1. Missing last_indexed_at Update

The GitHub indexer (github_indexer.py) was missing the call to update_connector_last_indexed() that other indexers (like JIRA) properly included. This caused the "Last Indexed" timestamp to not be persisted.

2. Timezone-Aware Timestamp Issues

The database schema expects timezone-aware timestamps (TIMESTAMP(timezone=True)), but the update functions were using datetime.now() without timezone information, causing potential issues with timestamp storage.

3. Session/Transaction Management Issues

The GitHub indexing route was trying to update the last_indexed_at timestamp in a separate session from the one used for document indexing, which could cause transaction conflicts.

Fixes Applied

1. Added Missing Import and Function Call

File: surfsense_backend/app/tasks/connector_indexers/github_indexer.py

# Added import
from .base import (
    check_duplicate_document_by_hash,
    get_connector_by_id,
    logger,
    update_connector_last_indexed,  # <- Added this
)

# Added before commit (around line 305)
# Update the last_indexed_at timestamp for the connector only if requested
if update_last_indexed:
    await update_connector_last_indexed(session, connector, update_last_indexed)

# Commit all changes at the end
await session.commit()

2. Fixed Timezone-Aware Timestamps

File: surfsense_backend/app/tasks/connector_indexers/base.py

# Added UTC import
from datetime import UTC, datetime, timedelta

# Fixed timestamp creation
connector.last_indexed_at = datetime.now(UTC)  # <- Added UTC

File: surfsense_backend/app/routes/search_source_connectors_routes.py

# Added UTC import
from datetime import UTC, datetime, timedelta

# Fixed timestamp creation
connector.last_indexed_at = datetime.now(UTC)  # <- Added UTC

3. Fixed Session Management

File: surfsense_backend/app/routes/search_source_connectors_routes.py

# Let the indexer handle timestamp updates within its own transaction
indexed_count, error_message = await index_github_repos(
    session,
    connector_id,
    search_space_id,
    user_id,
    start_date,
    end_date,
    update_last_indexed=True,  # <- Changed from False to True
)

# Removed separate timestamp update logic since indexer handles it

Testing the Fix

1. Test Document Persistence

After applying the fix:

1. Configure a GitHub connector with a valid PAT
2. Select repositories to index
3. Trigger indexing
4. Check that documents appear in "Manage Documents"
5. Verify that queries return relevant results

2. Test Last Indexed Timestamp

1. After successful indexing, note the "Last Indexed" timestamp
2. Refresh the connectors page
3. Verify the timestamp persists and doesn't revert to "Never"

3. Use Debug Script

A debug script has been provided (debug_github_connector.py) to help diagnose issues:

cd /path/to/SurfSense
python debug_github_connector.py --connector-id <connector_id> --search_space_id <search_space_id>

This script will:

• Check if the connector exists
• Count GitHub documents in the search space
• Show document distribution by type
• Verify database state

Additional Improvements Made

Enhanced Error Handling

The fix maintains the existing error handling while ensuring that successful indexing properly updates timestamps.

Consistent with Other Connectors

The GitHub indexer now follows the same pattern as other working connectors (like JIRA) for timestamp management.

Better Transaction Integrity

By handling timestamp updates within the same transaction as document creation, we avoid potential consistency issues.

Rollback Instructions

If issues arise, you can rollback by:

1. Reverting the import change in github_indexer.py
2. Removing the update_connector_last_indexed call
3. Reverting the UTC timestamp changes
4. Reverting the session management changes in the routes

However, these changes are minimal and follow established patterns from other working indexers, so rollback should not be necessary.

Monitoring

After deploying the fix, monitor:

1. GitHub connector indexing logs for any new errors
2. Document creation in the database
3. Last indexed timestamp persistence
4. Query result accuracy

The fix addresses the core issues while maintaining compatibility with the existing codebase and following established patterns from other working connectors.