Merge remote-tracking branch 'upstream/dev' into fix/chat-ui

This commit is contained in:
Anish Sarkar 2026-01-20 16:35:49 +05:30
commit 08e00d0991
40 changed files with 1274 additions and 865 deletions

View file

@ -158,7 +158,7 @@ Check out our public roadmap and contribute your ideas or feedback:
**Linux/macOS:** **Linux/macOS:**
```bash ```bash
docker run -d -p 3000:3000 -p 8000:8000 \ docker run -d -p 3000:3000 -p 8000:8000 -p 5133:5133 \
-v surfsense-data:/data \ -v surfsense-data:/data \
--name surfsense \ --name surfsense \
--restart unless-stopped \ --restart unless-stopped \
@ -168,7 +168,7 @@ docker run -d -p 3000:3000 -p 8000:8000 \
**Windows (PowerShell):** **Windows (PowerShell):**
```powershell ```powershell
docker run -d -p 3000:3000 -p 8000:8000 ` docker run -d -p 3000:3000 -p 8000:8000 -p 5133:5133 `
-v surfsense-data:/data ` -v surfsense-data:/data `
--name surfsense ` --name surfsense `
--restart unless-stopped ` --restart unless-stopped `
@ -180,7 +180,7 @@ docker run -d -p 3000:3000 -p 8000:8000 `
You can pass any environment variable using `-e` flags: You can pass any environment variable using `-e` flags:
```bash ```bash
docker run -d -p 3000:3000 -p 8000:8000 \ docker run -d -p 3000:3000 -p 8000:8000 -p 5133:5133 \
-v surfsense-data:/data \ -v surfsense-data:/data \
-e EMBEDDING_MODEL=openai://text-embedding-ada-002 \ -e EMBEDDING_MODEL=openai://text-embedding-ada-002 \
-e OPENAI_API_KEY=your_openai_api_key \ -e OPENAI_API_KEY=your_openai_api_key \
@ -201,6 +201,7 @@ After starting, access SurfSense at:
- **Frontend**: [http://localhost:3000](http://localhost:3000) - **Frontend**: [http://localhost:3000](http://localhost:3000)
- **Backend API**: [http://localhost:8000](http://localhost:8000) - **Backend API**: [http://localhost:8000](http://localhost:8000)
- **API Docs**: [http://localhost:8000/docs](http://localhost:8000/docs) - **API Docs**: [http://localhost:8000/docs](http://localhost:8000/docs)
- **Electric-SQL**: [http://localhost:5133](http://localhost:5133)
**Useful Commands:** **Useful Commands:**

View file

@ -165,7 +165,7 @@ https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
**Linux/macOS:** **Linux/macOS:**
```bash ```bash
docker run -d -p 3000:3000 -p 8000:8000 \ docker run -d -p 3000:3000 -p 8000:8000 -p 5133:5133 \
-v surfsense-data:/data \ -v surfsense-data:/data \
--name surfsense \ --name surfsense \
--restart unless-stopped \ --restart unless-stopped \
@ -175,7 +175,7 @@ docker run -d -p 3000:3000 -p 8000:8000 \
**Windows (PowerShell):** **Windows (PowerShell):**
```powershell ```powershell
docker run -d -p 3000:3000 -p 8000:8000 ` docker run -d -p 3000:3000 -p 8000:8000 -p 5133:5133 `
-v surfsense-data:/data ` -v surfsense-data:/data `
--name surfsense ` --name surfsense `
--restart unless-stopped ` --restart unless-stopped `
@ -187,7 +187,7 @@ docker run -d -p 3000:3000 -p 8000:8000 `
您可以使用 `-e` 标志传递任何环境变量: 您可以使用 `-e` 标志传递任何环境变量:
```bash ```bash
docker run -d -p 3000:3000 -p 8000:8000 \ docker run -d -p 3000:3000 -p 8000:8000 -p 5133:5133 \
-v surfsense-data:/data \ -v surfsense-data:/data \
-e EMBEDDING_MODEL=openai://text-embedding-ada-002 \ -e EMBEDDING_MODEL=openai://text-embedding-ada-002 \
-e OPENAI_API_KEY=your_openai_api_key \ -e OPENAI_API_KEY=your_openai_api_key \
@ -208,6 +208,7 @@ docker run -d -p 3000:3000 -p 8000:8000 \
- **前端**: [http://localhost:3000](http://localhost:3000) - **前端**: [http://localhost:3000](http://localhost:3000)
- **后端 API**: [http://localhost:8000](http://localhost:8000) - **后端 API**: [http://localhost:8000](http://localhost:8000)
- **API 文档**: [http://localhost:8000/docs](http://localhost:8000/docs) - **API 文档**: [http://localhost:8000/docs](http://localhost:8000/docs)
- **Electric-SQL**: [http://localhost:5133](http://localhost:5133)
**常用命令:** **常用命令:**

View file

@ -4,6 +4,10 @@ DATABASE_URL=postgresql+asyncpg://postgres:postgres@localhost:5432/surfsense
CELERY_BROKER_URL=redis://localhost:6379/0 CELERY_BROKER_URL=redis://localhost:6379/0
CELERY_RESULT_BACKEND=redis://localhost:6379/0 CELERY_RESULT_BACKEND=redis://localhost:6379/0
#Electric(for migrations only)
ELECTRIC_DB_USER=electric
ELECTRIC_DB_PASSWORD=electric_password
# Periodic task interval # Periodic task interval
# # Run every minute (default) # # Run every minute (default)
# SCHEDULE_CHECKER_INTERVAL=1m # SCHEDULE_CHECKER_INTERVAL=1m

View file

@ -0,0 +1,300 @@
"""Simplify RBAC roles - Remove Admin role, keep only Owner, Editor, Viewer
Revision ID: 72
Revises: 71
Create Date: 2025-01-20
This migration:
1. Moves any users with Admin role to Editor role
2. Updates invites that reference Admin role to use Editor role
3. Deletes the Admin role from all search spaces
4. Updates Editor permissions to the new simplified set (everything except delete)
5. Updates Viewer permissions to the new simplified set (read-only + comments)
"""
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
revision = "72"
down_revision = "71"
branch_labels = None
depends_on = None
# New Editor permissions (can do everything except delete, manage roles, and update settings)
NEW_EDITOR_PERMISSIONS = [
"documents:create",
"documents:read",
"documents:update",
"chats:create",
"chats:read",
"chats:update",
"comments:create",
"comments:read",
"llm_configs:create",
"llm_configs:read",
"llm_configs:update",
"podcasts:create",
"podcasts:read",
"podcasts:update",
"connectors:create",
"connectors:read",
"connectors:update",
"logs:read",
"members:invite",
"members:view",
"roles:read",
"settings:view",
]
# New Viewer permissions (read-only + comments)
NEW_VIEWER_PERMISSIONS = [
"documents:read",
"chats:read",
"comments:create",
"comments:read",
"llm_configs:read",
"podcasts:read",
"connectors:read",
"logs:read",
"members:view",
"roles:read",
"settings:view",
]
def upgrade():
connection = op.get_bind()
# Step 1: For each search space, get the Editor role ID and Admin role ID
search_spaces = connection.execute(
sa.text("SELECT id FROM searchspaces")
).fetchall()
for (ss_id,) in search_spaces:
# Get Admin and Editor role IDs for this search space
admin_role = connection.execute(
sa.text("""
SELECT id FROM search_space_roles
WHERE search_space_id = :ss_id AND name = 'Admin'
"""),
{"ss_id": ss_id},
).fetchone()
editor_role = connection.execute(
sa.text("""
SELECT id FROM search_space_roles
WHERE search_space_id = :ss_id AND name = 'Editor'
"""),
{"ss_id": ss_id},
).fetchone()
if admin_role and editor_role:
admin_role_id = admin_role[0]
editor_role_id = editor_role[0]
# Step 2: Move all memberships from Admin to Editor
connection.execute(
sa.text("""
UPDATE search_space_memberships
SET role_id = :editor_role_id
WHERE role_id = :admin_role_id
"""),
{"editor_role_id": editor_role_id, "admin_role_id": admin_role_id},
)
# Step 3: Move all invites from Admin to Editor
connection.execute(
sa.text("""
UPDATE search_space_invites
SET role_id = :editor_role_id
WHERE role_id = :admin_role_id
"""),
{"editor_role_id": editor_role_id, "admin_role_id": admin_role_id},
)
# Step 4: Delete the Admin role
connection.execute(
sa.text("""
DELETE FROM search_space_roles
WHERE id = :admin_role_id
"""),
{"admin_role_id": admin_role_id},
)
# Step 5: Update Editor permissions for all search spaces
editor_perms_literal = (
"ARRAY[" + ",".join(f"'{p}'" for p in NEW_EDITOR_PERMISSIONS) + "]::TEXT[]"
)
connection.execute(
sa.text(f"""
UPDATE search_space_roles
SET permissions = {editor_perms_literal},
description = 'Can create and update content (no delete, role management, or settings access)'
WHERE name = 'Editor' AND is_system_role = TRUE
""")
)
# Step 6: Update Viewer permissions for all search spaces
viewer_perms_literal = (
"ARRAY[" + ",".join(f"'{p}'" for p in NEW_VIEWER_PERMISSIONS) + "]::TEXT[]"
)
connection.execute(
sa.text(f"""
UPDATE search_space_roles
SET permissions = {viewer_perms_literal}
WHERE name = 'Viewer' AND is_system_role = TRUE
""")
)
def downgrade():
"""
Downgrade recreates the Admin role and restores original permissions.
Note: Users who were moved from Admin to Editor will remain as Editor.
"""
connection = op.get_bind()
# Old Admin permissions
old_admin_permissions = [
"documents:create",
"documents:read",
"documents:update",
"documents:delete",
"chats:create",
"chats:read",
"chats:update",
"chats:delete",
"comments:create",
"comments:read",
"comments:delete",
"llm_configs:create",
"llm_configs:read",
"llm_configs:update",
"llm_configs:delete",
"podcasts:create",
"podcasts:read",
"podcasts:update",
"podcasts:delete",
"connectors:create",
"connectors:read",
"connectors:update",
"connectors:delete",
"logs:read",
"logs:delete",
"members:invite",
"members:view",
"members:remove",
"members:manage_roles",
"roles:create",
"roles:read",
"roles:update",
"roles:delete",
"settings:view",
"settings:update",
]
# Old Editor permissions
old_editor_permissions = [
"documents:create",
"documents:read",
"documents:update",
"documents:delete",
"chats:create",
"chats:read",
"chats:update",
"chats:delete",
"comments:create",
"comments:read",
"llm_configs:read",
"llm_configs:create",
"llm_configs:update",
"podcasts:create",
"podcasts:read",
"podcasts:update",
"podcasts:delete",
"connectors:create",
"connectors:read",
"connectors:update",
"logs:read",
"members:view",
"roles:read",
"settings:view",
]
# Old Viewer permissions
old_viewer_permissions = [
"documents:read",
"chats:read",
"comments:create",
"comments:read",
"llm_configs:read",
"podcasts:read",
"connectors:read",
"logs:read",
"members:view",
"roles:read",
"settings:view",
]
# Recreate Admin role for each search space
search_spaces = connection.execute(
sa.text("SELECT id FROM searchspaces")
).fetchall()
admin_perms_literal = (
"ARRAY[" + ",".join(f"'{p}'" for p in old_admin_permissions) + "]::TEXT[]"
)
for (ss_id,) in search_spaces:
# Check if Admin role already exists
existing = connection.execute(
sa.text("""
SELECT id FROM search_space_roles
WHERE search_space_id = :ss_id AND name = 'Admin'
"""),
{"ss_id": ss_id},
).fetchone()
if not existing:
connection.execute(
sa.text(f"""
INSERT INTO search_space_roles
(name, description, permissions, is_default, is_system_role, search_space_id)
VALUES (
'Admin',
'Can manage most resources except deleting the search space',
{admin_perms_literal},
FALSE,
TRUE,
:ss_id
)
"""),
{"ss_id": ss_id},
)
# Restore old Editor permissions
editor_perms_literal = (
"ARRAY[" + ",".join(f"'{p}'" for p in old_editor_permissions) + "]::TEXT[]"
)
connection.execute(
sa.text(f"""
UPDATE search_space_roles
SET permissions = {editor_perms_literal},
description = 'Can create and edit documents, chats, and podcasts'
WHERE name = 'Editor' AND is_system_role = TRUE
""")
)
# Restore old Viewer permissions
viewer_perms_literal = (
"ARRAY[" + ",".join(f"'{p}'" for p in old_viewer_permissions) + "]::TEXT[]"
)
connection.execute(
sa.text(f"""
UPDATE search_space_roles
SET permissions = {viewer_perms_literal}
WHERE name = 'Viewer' AND is_system_role = TRUE
""")
)

View file

@ -77,7 +77,7 @@ class MCPClient:
# Initialize the connection # Initialize the connection
await session.initialize() await session.initialize()
self.session = session self.session = session
if attempt > 0: if attempt > 0:
logger.info( logger.info(
"Connected to MCP server on attempt %d: %s %s", "Connected to MCP server on attempt %d: %s %s",
@ -267,30 +267,38 @@ async def test_mcp_http_connection(
""" """
try: try:
logger.info("Testing HTTP MCP connection to: %s (transport: %s)", url, transport) logger.info(
"Testing HTTP MCP connection to: %s (transport: %s)", url, transport
)
# Use streamable HTTP client for all HTTP-based transports # Use streamable HTTP client for all HTTP-based transports
async with streamablehttp_client(url, headers=headers or {}) as (read, write, _): async with (
async with ClientSession(read, write) as session: streamablehttp_client(url, headers=headers or {}) as (read, write, _),
await session.initialize() ClientSession(read, write) as session,
):
# List available tools await session.initialize()
response = await session.list_tools()
tools = [] # List available tools
for tool in response.tools: response = await session.list_tools()
tools.append({ tools = []
for tool in response.tools:
tools.append(
{
"name": tool.name, "name": tool.name,
"description": tool.description or "", "description": tool.description or "",
"input_schema": tool.inputSchema if hasattr(tool, "inputSchema") else {}, "input_schema": tool.inputSchema
}) if hasattr(tool, "inputSchema")
else {},
logger.info("HTTP MCP connection successful. Found %d tools.", len(tools)) }
return { )
"status": "success",
"message": f"Connected successfully. Found {len(tools)} tools.", logger.info("HTTP MCP connection successful. Found %d tools.", len(tools))
"tools": tools, return {
} "status": "success",
"message": f"Connected successfully. Found {len(tools)} tools.",
"tools": tools,
}
except Exception as e: except Exception as e:
logger.error("Failed to connect to HTTP MCP server: %s", e, exc_info=True) logger.error("Failed to connect to HTTP MCP server: %s", e, exc_info=True)
return { return {

View file

@ -160,27 +160,31 @@ async def _create_mcp_tool_from_definition_http(
logger.info(f"MCP HTTP tool '{tool_name}' called with params: {kwargs}") logger.info(f"MCP HTTP tool '{tool_name}' called with params: {kwargs}")
try: try:
async with streamablehttp_client(url, headers=headers) as (read, write, _): async with (
async with ClientSession(read, write) as session: streamablehttp_client(url, headers=headers) as (read, write, _),
await session.initialize() ClientSession(read, write) as session,
):
# Call the tool await session.initialize()
response = await session.call_tool(tool_name, arguments=kwargs)
# Call the tool
# Extract content from response response = await session.call_tool(tool_name, arguments=kwargs)
result = []
for content in response.content: # Extract content from response
if hasattr(content, "text"): result = []
result.append(content.text) for content in response.content:
elif hasattr(content, "data"): if hasattr(content, "text"):
result.append(str(content.data)) result.append(content.text)
else: elif hasattr(content, "data"):
result.append(str(content)) result.append(str(content.data))
else:
result_str = "\n".join(result) if result else "" result.append(str(content))
logger.info(f"MCP HTTP tool '{tool_name}' succeeded: {result_str[:200]}")
return result_str result_str = "\n".join(result) if result else ""
logger.info(
f"MCP HTTP tool '{tool_name}' succeeded: {result_str[:200]}"
)
return result_str
except Exception as e: except Exception as e:
error_msg = f"MCP HTTP tool '{tool_name}' execution failed: {e!s}" error_msg = f"MCP HTTP tool '{tool_name}' execution failed: {e!s}"
logger.exception(error_msg) logger.exception(error_msg)
@ -192,7 +196,11 @@ async def _create_mcp_tool_from_definition_http(
description=tool_description, description=tool_description,
coroutine=mcp_http_tool_call, coroutine=mcp_http_tool_call,
args_schema=input_model, args_schema=input_model,
metadata={"mcp_input_schema": input_schema, "mcp_transport": "http", "mcp_url": url}, metadata={
"mcp_input_schema": input_schema,
"mcp_transport": "http",
"mcp_url": url,
},
) )
logger.info(f"Created MCP tool (HTTP): '{tool_name}'") logger.info(f"Created MCP tool (HTTP): '{tool_name}'")
@ -205,17 +213,17 @@ async def _load_stdio_mcp_tools(
server_config: dict[str, Any], server_config: dict[str, Any],
) -> list[StructuredTool]: ) -> list[StructuredTool]:
"""Load tools from a stdio-based MCP server. """Load tools from a stdio-based MCP server.
Args: Args:
connector_id: Connector ID for logging connector_id: Connector ID for logging
connector_name: Connector name for logging connector_name: Connector name for logging
server_config: Server configuration with command, args, env server_config: Server configuration with command, args, env
Returns: Returns:
List of tools from the MCP server List of tools from the MCP server
""" """
tools: list[StructuredTool] = [] tools: list[StructuredTool] = []
# Validate required command field # Validate required command field
command = server_config.get("command") command = server_config.get("command")
if not command or not isinstance(command, str): if not command or not isinstance(command, str):
@ -262,7 +270,7 @@ async def _load_stdio_mcp_tools(
f"Failed to create tool '{tool_def.get('name')}' " f"Failed to create tool '{tool_def.get('name')}' "
f"from connector {connector_id}: {e!s}" f"from connector {connector_id}: {e!s}"
) )
return tools return tools
@ -272,17 +280,17 @@ async def _load_http_mcp_tools(
server_config: dict[str, Any], server_config: dict[str, Any],
) -> list[StructuredTool]: ) -> list[StructuredTool]:
"""Load tools from an HTTP-based MCP server. """Load tools from an HTTP-based MCP server.
Args: Args:
connector_id: Connector ID for logging connector_id: Connector ID for logging
connector_name: Connector name for logging connector_name: Connector name for logging
server_config: Server configuration with url, headers server_config: Server configuration with url, headers
Returns: Returns:
List of tools from the MCP server List of tools from the MCP server
""" """
tools: list[StructuredTool] = [] tools: list[StructuredTool] = []
# Validate required url field # Validate required url field
url = server_config.get("url") url = server_config.get("url")
if not url or not isinstance(url, str): if not url or not isinstance(url, str):
@ -301,41 +309,49 @@ async def _load_http_mcp_tools(
# Connect and discover tools via HTTP # Connect and discover tools via HTTP
try: try:
async with streamablehttp_client(url, headers=headers) as (read, write, _): async with (
async with ClientSession(read, write) as session: streamablehttp_client(url, headers=headers) as (read, write, _),
await session.initialize() ClientSession(read, write) as session,
):
# List available tools await session.initialize()
response = await session.list_tools()
tool_definitions = [] # List available tools
for tool in response.tools: response = await session.list_tools()
tool_definitions.append({ tool_definitions = []
for tool in response.tools:
tool_definitions.append(
{
"name": tool.name, "name": tool.name,
"description": tool.description or "", "description": tool.description or "",
"input_schema": tool.inputSchema if hasattr(tool, "inputSchema") else {}, "input_schema": tool.inputSchema
}) if hasattr(tool, "inputSchema")
else {},
logger.info( }
f"Discovered {len(tool_definitions)} tools from HTTP MCP server "
f"'{url}' (connector {connector_id})"
) )
logger.info(
f"Discovered {len(tool_definitions)} tools from HTTP MCP server "
f"'{url}' (connector {connector_id})"
)
# Create LangChain tools from definitions # Create LangChain tools from definitions
for tool_def in tool_definitions: for tool_def in tool_definitions:
try: try:
tool = await _create_mcp_tool_from_definition_http(tool_def, url, headers) tool = await _create_mcp_tool_from_definition_http(
tool_def, url, headers
)
tools.append(tool) tools.append(tool)
except Exception as e: except Exception as e:
logger.exception( logger.exception(
f"Failed to create HTTP tool '{tool_def.get('name')}' " f"Failed to create HTTP tool '{tool_def.get('name')}' "
f"from connector {connector_id}: {e!s}" f"from connector {connector_id}: {e!s}"
) )
except Exception as e: except Exception as e:
logger.exception( logger.exception(
f"Failed to connect to HTTP MCP server at '{url}' (connector {connector_id}): {e!s}" f"Failed to connect to HTTP MCP server at '{url}' (connector {connector_id}): {e!s}"
) )
return tools return tools
@ -372,7 +388,7 @@ async def load_mcp_tools(
# Early validation: Extract and validate connector config # Early validation: Extract and validate connector config
config = connector.config or {} config = connector.config or {}
server_config = config.get("server_config", {}) server_config = config.get("server_config", {})
# Validate server_config exists and is a dict # Validate server_config exists and is a dict
if not server_config or not isinstance(server_config, dict): if not server_config or not isinstance(server_config, dict):
logger.warning( logger.warning(
@ -382,7 +398,7 @@ async def load_mcp_tools(
# Determine transport type # Determine transport type
transport = server_config.get("transport", "stdio") transport = server_config.get("transport", "stdio")
if transport in ("streamable-http", "http", "sse"): if transport in ("streamable-http", "http", "sse"):
# HTTP-based MCP server # HTTP-based MCP server
connector_tools = await _load_http_mcp_tools( connector_tools = await _load_http_mcp_tools(
@ -393,9 +409,9 @@ async def load_mcp_tools(
connector_tools = await _load_stdio_mcp_tools( connector_tools = await _load_stdio_mcp_tools(
connector.id, connector.name, server_config connector.id, connector.name, server_config
) )
tools.extend(connector_tools) tools.extend(connector_tools)
except Exception as e: except Exception as e:
logger.exception( logger.exception(
f"Failed to load tools from MCP connector {connector.id}: {e!s}" f"Failed to load tools from MCP connector {connector.id}: {e!s}"

View file

@ -201,89 +201,42 @@ class Permission(str, Enum):
# Predefined role permission sets for convenience # Predefined role permission sets for convenience
# Note: Only Owner, Editor, and Viewer roles are supported.
# Owner has full access (*), Editor can do everything except delete, Viewer has read-only access.
DEFAULT_ROLE_PERMISSIONS = { DEFAULT_ROLE_PERMISSIONS = {
"Owner": [Permission.FULL_ACCESS.value], "Owner": [Permission.FULL_ACCESS.value],
"Admin": [
# Documents
Permission.DOCUMENTS_CREATE.value,
Permission.DOCUMENTS_READ.value,
Permission.DOCUMENTS_UPDATE.value,
Permission.DOCUMENTS_DELETE.value,
# Chats
Permission.CHATS_CREATE.value,
Permission.CHATS_READ.value,
Permission.CHATS_UPDATE.value,
Permission.CHATS_DELETE.value,
# Comments
Permission.COMMENTS_CREATE.value,
Permission.COMMENTS_READ.value,
Permission.COMMENTS_DELETE.value,
# LLM Configs
Permission.LLM_CONFIGS_CREATE.value,
Permission.LLM_CONFIGS_READ.value,
Permission.LLM_CONFIGS_UPDATE.value,
Permission.LLM_CONFIGS_DELETE.value,
# Podcasts
Permission.PODCASTS_CREATE.value,
Permission.PODCASTS_READ.value,
Permission.PODCASTS_UPDATE.value,
Permission.PODCASTS_DELETE.value,
# Connectors
Permission.CONNECTORS_CREATE.value,
Permission.CONNECTORS_READ.value,
Permission.CONNECTORS_UPDATE.value,
Permission.CONNECTORS_DELETE.value,
# Logs
Permission.LOGS_READ.value,
Permission.LOGS_DELETE.value,
# Members
Permission.MEMBERS_INVITE.value,
Permission.MEMBERS_VIEW.value,
Permission.MEMBERS_REMOVE.value,
Permission.MEMBERS_MANAGE_ROLES.value,
# Roles
Permission.ROLES_CREATE.value,
Permission.ROLES_READ.value,
Permission.ROLES_UPDATE.value,
Permission.ROLES_DELETE.value,
# Settings (no delete)
Permission.SETTINGS_VIEW.value,
Permission.SETTINGS_UPDATE.value,
],
"Editor": [ "Editor": [
# Documents # Documents (no delete)
Permission.DOCUMENTS_CREATE.value, Permission.DOCUMENTS_CREATE.value,
Permission.DOCUMENTS_READ.value, Permission.DOCUMENTS_READ.value,
Permission.DOCUMENTS_UPDATE.value, Permission.DOCUMENTS_UPDATE.value,
Permission.DOCUMENTS_DELETE.value, # Chats (no delete)
# Chats
Permission.CHATS_CREATE.value, Permission.CHATS_CREATE.value,
Permission.CHATS_READ.value, Permission.CHATS_READ.value,
Permission.CHATS_UPDATE.value, Permission.CHATS_UPDATE.value,
Permission.CHATS_DELETE.value,
# Comments (no delete) # Comments (no delete)
Permission.COMMENTS_CREATE.value, Permission.COMMENTS_CREATE.value,
Permission.COMMENTS_READ.value, Permission.COMMENTS_READ.value,
# LLM Configs (read only) # LLM Configs (no delete)
Permission.LLM_CONFIGS_READ.value,
Permission.LLM_CONFIGS_CREATE.value, Permission.LLM_CONFIGS_CREATE.value,
Permission.LLM_CONFIGS_READ.value,
Permission.LLM_CONFIGS_UPDATE.value, Permission.LLM_CONFIGS_UPDATE.value,
# Podcasts # Podcasts (no delete)
Permission.PODCASTS_CREATE.value, Permission.PODCASTS_CREATE.value,
Permission.PODCASTS_READ.value, Permission.PODCASTS_READ.value,
Permission.PODCASTS_UPDATE.value, Permission.PODCASTS_UPDATE.value,
Permission.PODCASTS_DELETE.value, # Connectors (no delete)
# Connectors (full access for editors)
Permission.CONNECTORS_CREATE.value, Permission.CONNECTORS_CREATE.value,
Permission.CONNECTORS_READ.value, Permission.CONNECTORS_READ.value,
Permission.CONNECTORS_UPDATE.value, Permission.CONNECTORS_UPDATE.value,
# Logs # Logs (read only)
Permission.LOGS_READ.value, Permission.LOGS_READ.value,
# Members (view only) # Members (can invite and view only, cannot manage roles or remove)
Permission.MEMBERS_INVITE.value,
Permission.MEMBERS_VIEW.value, Permission.MEMBERS_VIEW.value,
# Roles (read only) # Roles (read only - cannot create, update, or delete)
Permission.ROLES_READ.value, Permission.ROLES_READ.value,
# Settings (view only) # Settings (view only, no update or delete)
Permission.SETTINGS_VIEW.value, Permission.SETTINGS_VIEW.value,
], ],
"Viewer": [ "Viewer": [
@ -291,7 +244,7 @@ DEFAULT_ROLE_PERMISSIONS = {
Permission.DOCUMENTS_READ.value, Permission.DOCUMENTS_READ.value,
# Chats (read only) # Chats (read only)
Permission.CHATS_READ.value, Permission.CHATS_READ.value,
# Comments (no delete) # Comments (can create and read, but not delete)
Permission.COMMENTS_CREATE.value, Permission.COMMENTS_CREATE.value,
Permission.COMMENTS_READ.value, Permission.COMMENTS_READ.value,
# LLM Configs (read only) # LLM Configs (read only)
@ -865,7 +818,7 @@ class SearchSpaceRole(BaseModel, TimestampMixin):
permissions = Column(ARRAY(String), nullable=False, default=[]) permissions = Column(ARRAY(String), nullable=False, default=[])
# Whether this role is assigned to new members by default when they join via invite # Whether this role is assigned to new members by default when they join via invite
is_default = Column(Boolean, nullable=False, default=False) is_default = Column(Boolean, nullable=False, default=False)
# System roles (Owner, Admin, Editor, Viewer) cannot be deleted # System roles (Owner, Editor, Viewer) cannot be deleted
is_system_role = Column(Boolean, nullable=False, default=False) is_system_role = Column(Boolean, nullable=False, default=False)
search_space_id = Column( search_space_id = Column(
@ -1221,6 +1174,11 @@ def get_default_roles_config() -> list[dict]:
Get the configuration for default system roles. Get the configuration for default system roles.
These roles are created automatically when a search space is created. These roles are created automatically when a search space is created.
Only 3 roles are supported:
- Owner: Full access to everything (assigned to search space creator)
- Editor: Can create/update content but cannot delete, manage roles, or change settings
- Viewer: Read-only access to resources (can add comments)
Returns: Returns:
List of role configurations with name, description, permissions, and flags List of role configurations with name, description, permissions, and flags
""" """
@ -1232,16 +1190,9 @@ def get_default_roles_config() -> list[dict]:
"is_default": False, "is_default": False,
"is_system_role": True, "is_system_role": True,
}, },
{
"name": "Admin",
"description": "Can manage most resources except deleting the search space",
"permissions": DEFAULT_ROLE_PERMISSIONS["Admin"],
"is_default": False,
"is_system_role": True,
},
{ {
"name": "Editor", "name": "Editor",
"description": "Can create and edit documents, chats, and podcasts", "description": "Can create and update content (no delete, role management, or settings access)",
"permissions": DEFAULT_ROLE_PERMISSIONS["Editor"], "permissions": DEFAULT_ROLE_PERMISSIONS["Editor"],
"is_default": True, # Default role for new members via invite "is_default": True, # Default role for new members via invite
"is_system_role": True, "is_system_role": True,

View file

@ -67,16 +67,15 @@ async def check_thread_access(
Access is granted if: Access is granted if:
- User is the creator of the thread - User is the creator of the thread
- Thread visibility is SEARCH_SPACE (any member can access) - Thread visibility is SEARCH_SPACE (any member can access) - for read/update operations only
- Thread is a legacy thread (created_by_id is NULL) - only if user is search space owner - Thread is a legacy thread (created_by_id is NULL) - only if user is search space owner
Args: Args:
session: Database session session: Database session
thread: The thread to check access for thread: The thread to check access for
user: The user requesting access user: The user requesting access
require_ownership: If True, only the creator can access (for edit/delete operations) require_ownership: If True, ONLY the creator can perform this action (e.g., changing visibility).
For SEARCH_SPACE threads, any member with permission can access This is checked FIRST, before visibility rules.
Legacy threads (NULL creator) are accessible by search space owner
Returns: Returns:
True if access is granted True if access is granted
@ -87,11 +86,18 @@ async def check_thread_access(
is_owner = thread.created_by_id == user.id is_owner = thread.created_by_id == user.id
is_legacy = thread.created_by_id is None is_legacy = thread.created_by_id is None
# Shared threads (SEARCH_SPACE) are accessible by any member # If ownership is required (e.g., changing visibility), ONLY the creator can do it
# This check comes first so shared threads are always accessible # This check comes first to ensure ownership-required operations are always creator-only
if require_ownership:
if not is_owner:
raise HTTPException(
status_code=403,
detail="Only the creator of this chat can perform this action",
)
return True
# Shared threads (SEARCH_SPACE) are accessible by any member for read/update operations
if thread.visibility == ChatVisibility.SEARCH_SPACE: if thread.visibility == ChatVisibility.SEARCH_SPACE:
# For ownership-required operations on shared threads, any member can proceed
# (permission check is done at route level)
return True return True
# For legacy threads (created before visibility feature), # For legacy threads (created before visibility feature),
@ -112,15 +118,6 @@ async def check_thread_access(
detail="You don't have access to this chat", detail="You don't have access to this chat",
) )
# If ownership is required, only the creator can access
if require_ownership:
if not is_owner:
raise HTTPException(
status_code=403,
detail="Only the creator of this chat can perform this action",
)
return True
# For read access: owner can access their own private threads # For read access: owner can access their own private threads
if is_owner: if is_owner:
return True return True

View file

@ -2403,25 +2403,29 @@ async def test_mcp_server_connection(
) )
transport = server_config.get("transport", "stdio") transport = server_config.get("transport", "stdio")
# HTTP transport (streamable-http, http, sse) # HTTP transport (streamable-http, http, sse)
if transport in ("streamable-http", "http", "sse"): if transport in ("streamable-http", "http", "sse"):
url = server_config.get("url") url = server_config.get("url")
headers = server_config.get("headers", {}) headers = server_config.get("headers", {})
if not url: if not url:
raise HTTPException(status_code=400, detail="Server URL is required for HTTP transport") raise HTTPException(
status_code=400, detail="Server URL is required for HTTP transport"
)
result = await test_mcp_http_connection(url, headers, transport) result = await test_mcp_http_connection(url, headers, transport)
return result return result
# stdio transport (default) # stdio transport (default)
command = server_config.get("command") command = server_config.get("command")
args = server_config.get("args", []) args = server_config.get("args", [])
env = server_config.get("env", {}) env = server_config.get("env", {})
if not command: if not command:
raise HTTPException(status_code=400, detail="Server command is required for stdio transport") raise HTTPException(
status_code=400, detail="Server command is required for stdio transport"
)
# Test the connection # Test the connection
result = await test_mcp_connection(command, args, env) result = await test_mcp_connection(command, args, env)

View file

@ -84,7 +84,7 @@ class SearchSourceConnectorRead(SearchSourceConnectorBase, IDModel, TimestampMod
class MCPServerConfig(BaseModel): class MCPServerConfig(BaseModel):
"""Configuration for an MCP server connection. """Configuration for an MCP server connection.
Supports two transport types: Supports two transport types:
- stdio: Local process (command, args, env) - stdio: Local process (command, args, env)
- streamable-http/http/sse: Remote HTTP server (url, headers) - streamable-http/http/sse: Remote HTTP server (url, headers)
@ -94,13 +94,13 @@ class MCPServerConfig(BaseModel):
command: str | None = None # e.g., "uvx", "node", "python" command: str | None = None # e.g., "uvx", "node", "python"
args: list[str] = [] # e.g., ["mcp-server-git", "--repository", "/path"] args: list[str] = [] # e.g., ["mcp-server-git", "--repository", "/path"]
env: dict[str, str] = {} # Environment variables for the server process env: dict[str, str] = {} # Environment variables for the server process
# HTTP transport fields # HTTP transport fields
url: str | None = None # e.g., "https://mcp-server.com/mcp" url: str | None = None # e.g., "https://mcp-server.com/mcp"
headers: dict[str, str] = {} # HTTP headers for authentication headers: dict[str, str] = {} # HTTP headers for authentication
transport: str = "stdio" # "stdio" | "streamable-http" | "http" | "sse" transport: str = "stdio" # "stdio" | "streamable-http" | "http" | "sse"
def is_http_transport(self) -> bool: def is_http_transport(self) -> bool:
"""Check if this config uses HTTP transport.""" """Check if this config uses HTTP transport."""
return self.transport in ("streamable-http", "http", "sse") return self.transport in ("streamable-http", "http", "sse")

View file

@ -767,20 +767,18 @@ function RolesTab({
className={cn( className={cn(
"h-10 w-10 rounded-lg flex items-center justify-center", "h-10 w-10 rounded-lg flex items-center justify-center",
role.name === "Owner" && "bg-amber-500/20", role.name === "Owner" && "bg-amber-500/20",
role.name === "Admin" && "bg-red-500/20",
role.name === "Editor" && "bg-blue-500/20", role.name === "Editor" && "bg-blue-500/20",
role.name === "Viewer" && "bg-gray-500/20", role.name === "Viewer" && "bg-gray-500/20",
!["Owner", "Admin", "Editor", "Viewer"].includes(role.name) && "bg-primary/20" !["Owner", "Editor", "Viewer"].includes(role.name) && "bg-primary/20"
)} )}
> >
<ShieldCheck <ShieldCheck
className={cn( className={cn(
"h-5 w-5", "h-5 w-5",
role.name === "Owner" && "text-amber-600", role.name === "Owner" && "text-amber-600",
role.name === "Admin" && "text-red-600",
role.name === "Editor" && "text-blue-600", role.name === "Editor" && "text-blue-600",
role.name === "Viewer" && "text-gray-600", role.name === "Viewer" && "text-gray-600",
!["Owner", "Admin", "Editor", "Viewer"].includes(role.name) && !["Owner", "Editor", "Viewer"].includes(role.name) &&
"text-primary" "text-primary"
)} )}
/> />
@ -1310,6 +1308,49 @@ function CreateInviteDialog({
// ============ Create Role Dialog ============ // ============ Create Role Dialog ============
// Preset permission sets for quick role creation
// Editor: can create/read/update content, but cannot manage roles, remove members, or change settings
// Viewer: read-only access with ability to create comments
const PRESET_PERMISSIONS = {
editor: [
"documents:create",
"documents:read",
"documents:update",
"chats:create",
"chats:read",
"chats:update",
"comments:create",
"comments:read",
"llm_configs:create",
"llm_configs:read",
"llm_configs:update",
"podcasts:create",
"podcasts:read",
"podcasts:update",
"connectors:create",
"connectors:read",
"connectors:update",
"logs:read",
"members:invite",
"members:view",
"roles:read",
"settings:view",
],
viewer: [
"documents:read",
"chats:read",
"comments:create",
"comments:read",
"llm_configs:read",
"podcasts:read",
"connectors:read",
"logs:read",
"members:view",
"roles:read",
"settings:view",
],
};
function CreateRoleDialog({ function CreateRoleDialog({
groupedPermissions, groupedPermissions,
onCreateRole, onCreateRole,
@ -1369,6 +1410,11 @@ function CreateRoleDialog({
} }
}; };
const applyPreset = (preset: "editor" | "viewer") => {
setSelectedPermissions(PRESET_PERMISSIONS[preset]);
toast.success(`Applied ${preset === "editor" ? "Editor" : "Viewer"} preset permissions`);
};
return ( return (
<Dialog open={open} onOpenChange={setOpen}> <Dialog open={open} onOpenChange={setOpen}>
<DialogTrigger asChild> <DialogTrigger asChild>
@ -1416,7 +1462,34 @@ function CreateRoleDialog({
/> />
</div> </div>
<div className="space-y-2"> <div className="space-y-2">
<Label>Permissions ({selectedPermissions.length} selected)</Label> <div className="flex items-center justify-between">
<Label>Permissions ({selectedPermissions.length} selected)</Label>
<div className="flex gap-2">
<Button
type="button"
variant="outline"
size="sm"
className="h-7 text-xs gap-1"
onClick={() => applyPreset("editor")}
>
<ShieldCheck className="h-3 w-3 text-blue-600" />
Editor Preset
</Button>
<Button
type="button"
variant="outline"
size="sm"
className="h-7 text-xs gap-1"
onClick={() => applyPreset("viewer")}
>
<ShieldCheck className="h-3 w-3 text-gray-600" />
Viewer Preset
</Button>
</div>
</div>
<p className="text-xs text-muted-foreground">
Use presets to quickly apply Editor (create/read/update) or Viewer (read-only) permissions
</p>
<ScrollArea className="h-64 rounded-lg border p-4"> <ScrollArea className="h-64 rounded-lg border p-4">
<div className="space-y-4"> <div className="space-y-4">
{Object.entries(groupedPermissions).map(([category, perms]) => { {Object.entries(groupedPermissions).map(([category, perms]) => {
@ -1427,10 +1500,8 @@ function CreateRoleDialog({
return ( return (
<div key={category} className="space-y-2"> <div key={category} className="space-y-2">
<button <label
type="button"
className="flex items-center gap-2 cursor-pointer hover:bg-muted/50 p-1 rounded w-full text-left" className="flex items-center gap-2 cursor-pointer hover:bg-muted/50 p-1 rounded w-full text-left"
onClick={() => toggleCategory(category)}
> >
<Checkbox <Checkbox
checked={allSelected} checked={allSelected}
@ -1439,21 +1510,19 @@ function CreateRoleDialog({
<span className="text-sm font-medium capitalize"> <span className="text-sm font-medium capitalize">
{category} ({categorySelected}/{perms.length}) {category} ({categorySelected}/{perms.length})
</span> </span>
</button> </label>
<div className="grid grid-cols-2 gap-2 ml-6"> <div className="grid grid-cols-2 gap-2 ml-6">
{perms.map((perm) => ( {perms.map((perm) => (
<button <label
type="button"
key={perm.value} key={perm.value}
className="flex items-center gap-2 cursor-pointer text-left" className="flex items-center gap-2 cursor-pointer text-left"
onClick={() => togglePermission(perm.value)}
> >
<Checkbox <Checkbox
checked={selectedPermissions.includes(perm.value)} checked={selectedPermissions.includes(perm.value)}
onCheckedChange={() => togglePermission(perm.value)} onCheckedChange={() => togglePermission(perm.value)}
/> />
<span className="text-xs">{perm.value.split(":")[1]}</span> <span className="text-xs">{perm.value.split(":")[1]}</span>
</button> </label>
))} ))}
</div> </div>
</div> </div>

View file

@ -204,7 +204,9 @@ export const AssistantMessage: FC = () => {
> >
<MessageSquare className={cn("size-4", hasComments && "fill-current")} /> <MessageSquare className={cn("size-4", hasComments && "fill-current")} />
{hasComments ? ( {hasComments ? (
<span>{commentCount} {commentCount === 1 ? "comment" : "comments"}</span> <span>
{commentCount} {commentCount === 1 ? "comment" : "comments"}
</span>
) : ( ) : (
<span>Add comment</span> <span>Add comment</span>
)} )}

View file

@ -22,7 +22,6 @@ import { useIndexingConnectors } from "./connector-popup/hooks/use-indexing-conn
import { ActiveConnectorsTab } from "./connector-popup/tabs/active-connectors-tab"; import { ActiveConnectorsTab } from "./connector-popup/tabs/active-connectors-tab";
import { AllConnectorsTab } from "./connector-popup/tabs/all-connectors-tab"; import { AllConnectorsTab } from "./connector-popup/tabs/all-connectors-tab";
import { ConnectorAccountsListView } from "./connector-popup/views/connector-accounts-list-view"; import { ConnectorAccountsListView } from "./connector-popup/views/connector-accounts-list-view";
import { MCPConnectorListView } from "./connector-popup/views/mcp-connector-list-view";
import { YouTubeCrawlerView } from "./connector-popup/views/youtube-crawler-view"; import { YouTubeCrawlerView } from "./connector-popup/views/youtube-crawler-view";
export const ConnectorIndicator: FC = () => { export const ConnectorIndicator: FC = () => {
@ -178,18 +177,16 @@ export const ConnectorIndicator: FC = () => {
{isYouTubeView && searchSpaceId ? ( {isYouTubeView && searchSpaceId ? (
<YouTubeCrawlerView searchSpaceId={searchSpaceId} onBack={handleBackFromYouTube} /> <YouTubeCrawlerView searchSpaceId={searchSpaceId} onBack={handleBackFromYouTube} />
) : viewingMCPList ? ( ) : viewingMCPList ? (
<div className="p-6 sm:p-12 h-full overflow-hidden"> <ConnectorAccountsListView
<MCPConnectorListView connectorType="MCP_CONNECTOR"
mcpConnectors={ connectorTitle="MCP Connectors"
(allConnectors || []).filter( connectors={(allConnectors || []) as SearchSourceConnector[]}
(c: SearchSourceConnector) => c.connector_type === "MCP_CONNECTOR" indexingConnectorIds={indexingConnectorIds}
) as SearchSourceConnector[] onBack={handleBackFromMCPList}
} onManage={handleStartEdit}
onAddNew={handleAddNewMCPFromList} onAddAccount={handleAddNewMCPFromList}
onManageConnector={handleStartEdit} addButtonText="Add New MCP Server"
onBack={handleBackFromMCPList} />
/>
</div>
) : viewingAccountsType ? ( ) : viewingAccountsType ? (
<ConnectorAccountsListView <ConnectorAccountsListView
connectorType={viewingAccountsType.connectorType} connectorType={viewingAccountsType.connectorType}

View file

@ -4,6 +4,7 @@ import { IconBrandYoutube } from "@tabler/icons-react";
import { FileText, Loader2 } from "lucide-react"; import { FileText, Loader2 } from "lucide-react";
import type { FC } from "react"; import type { FC } from "react";
import { Button } from "@/components/ui/button"; import { Button } from "@/components/ui/button";
import { EnumConnectorName } from "@/contracts/enums/connector";
import { getConnectorIcon } from "@/contracts/enums/connectorIcons"; import { getConnectorIcon } from "@/contracts/enums/connectorIcons";
import { cn } from "@/lib/utils"; import { cn } from "@/lib/utils";
import { useConnectorStatus } from "../hooks/use-connector-status"; import { useConnectorStatus } from "../hooks/use-connector-status";
@ -18,6 +19,7 @@ interface ConnectorCardProps {
isConnecting?: boolean; isConnecting?: boolean;
documentCount?: number; documentCount?: number;
accountCount?: number; accountCount?: number;
connectorCount?: number;
isIndexing?: boolean; isIndexing?: boolean;
onConnect?: () => void; onConnect?: () => void;
onManage?: () => void; onManage?: () => void;
@ -46,10 +48,12 @@ export const ConnectorCard: FC<ConnectorCardProps> = ({
isConnecting = false, isConnecting = false,
documentCount, documentCount,
accountCount, accountCount,
connectorCount,
isIndexing = false, isIndexing = false,
onConnect, onConnect,
onManage, onManage,
}) => { }) => {
const isMCP = connectorType === EnumConnectorName.MCP_CONNECTOR;
// Get connector status // Get connector status
const { getConnectorStatus, isConnectorEnabled, getConnectorStatusMessage, shouldShowWarnings } = const { getConnectorStatus, isConnectorEnabled, getConnectorStatusMessage, shouldShowWarnings } =
useConnectorStatus(); useConnectorStatus();
@ -112,13 +116,21 @@ export const ConnectorCard: FC<ConnectorCardProps> = ({
</p> </p>
) : isConnected ? ( ) : isConnected ? (
<p className="text-[10px] text-muted-foreground mt-1 flex items-center gap-1.5"> <p className="text-[10px] text-muted-foreground mt-1 flex items-center gap-1.5">
<span>{formatDocumentCount(documentCount)}</span> {isMCP && connectorCount !== undefined ? (
{accountCount !== undefined && accountCount > 0 && ( <span>
{connectorCount} {connectorCount === 1 ? "server" : "servers"}
</span>
) : (
<> <>
<span className="text-muted-foreground/50"></span> <span>{formatDocumentCount(documentCount)}</span>
<span> {accountCount !== undefined && accountCount > 0 && (
{accountCount} {accountCount === 1 ? "Account" : "Accounts"} <>
</span> <span className="text-muted-foreground/50"></span>
<span>
{accountCount} {accountCount === 1 ? "Account" : "Accounts"}
</span>
</>
)}
</> </>
)} )}
</p> </p>

View file

@ -1,7 +1,7 @@
"use client"; "use client";
import type { FC } from "react";
import { AlertCircle } from "lucide-react"; import { AlertCircle } from "lucide-react";
import type { FC } from "react";
import { Label } from "@/components/ui/label"; import { Label } from "@/components/ui/label";
import { import {
Select, Select,

View file

@ -4,18 +4,16 @@ import { CheckCircle2, ChevronDown, ChevronUp, Server, XCircle } from "lucide-re
import { type FC, useRef, useState } from "react"; import { type FC, useRef, useState } from "react";
import { Alert, AlertDescription, AlertTitle } from "@/components/ui/alert"; import { Alert, AlertDescription, AlertTitle } from "@/components/ui/alert";
import { Button } from "@/components/ui/button"; import { Button } from "@/components/ui/button";
import { Input } from "@/components/ui/input";
import { Label } from "@/components/ui/label"; import { Label } from "@/components/ui/label";
import { Textarea } from "@/components/ui/textarea"; import { Textarea } from "@/components/ui/textarea";
import { EnumConnectorName } from "@/contracts/enums/connector"; import { EnumConnectorName } from "@/contracts/enums/connector";
import type { MCPToolDefinition } from "@/contracts/types/mcp.types";
import type { ConnectFormProps } from "..";
import { import {
extractServerName, extractServerName,
type MCPConnectionTestResult,
parseMCPConfig, parseMCPConfig,
testMCPConnection, testMCPConnection,
type MCPConnectionTestResult,
} from "../../utils/mcp-config-validator"; } from "../../utils/mcp-config-validator";
import type { ConnectFormProps } from "..";
export const MCPConnectForm: FC<ConnectFormProps> = ({ onSubmit, isSubmitting }) => { export const MCPConnectForm: FC<ConnectFormProps> = ({ onSubmit, isSubmitting }) => {
const isSubmittingRef = useRef(false); const isSubmittingRef = useRef(false);
@ -46,7 +44,7 @@ export const MCPConnectForm: FC<ConnectFormProps> = ({ onSubmit, isSubmitting })
name: "My Remote MCP Server", name: "My Remote MCP Server",
url: "https://your-mcp-server.com/mcp", url: "https://your-mcp-server.com/mcp",
headers: { headers: {
"API_KEY": "your_api_key_here", API_KEY: "your_api_key_here",
}, },
transport: "streamable-http", transport: "streamable-http",
}, },
@ -178,29 +176,47 @@ export const MCPConnectForm: FC<ConnectFormProps> = ({ onSubmit, isSubmitting })
id="config" id="config"
value={configJson} value={configJson}
onChange={(e) => handleConfigChange(e.target.value)} onChange={(e) => handleConfigChange(e.target.value)}
onKeyDown={(e) => {
if (e.key === "Tab") {
e.preventDefault();
const target = e.target as HTMLTextAreaElement;
const start = target.selectionStart;
const end = target.selectionEnd;
const indent = " "; // 2 spaces for JSON
const newValue =
configJson.substring(0, start) + indent + configJson.substring(end);
handleConfigChange(newValue);
// Set cursor position after the inserted tab
requestAnimationFrame(() => {
target.selectionStart = target.selectionEnd = start + indent.length;
});
}
}}
placeholder={DEFAULT_CONFIG} placeholder={DEFAULT_CONFIG}
rows={16} rows={16}
className={`font-mono text-xs ${jsonError ? "border-red-500" : ""}`} className={`font-mono text-xs ${jsonError ? "border-red-500" : ""}`}
/> />
{jsonError && <p className="text-xs text-red-500">JSON Error: {jsonError}</p>} {jsonError && <p className="text-xs text-red-500">JSON Error: {jsonError}</p>}
<p className="text-[10px] sm:text-xs text-muted-foreground"> <p className="text-[10px] sm:text-xs text-muted-foreground">
<strong>Local (stdio):</strong> command, args, env, transport: "stdio"<br /> Paste a single MCP server configuration. Must include: name, command, args (optional),
<strong>Remote (HTTP):</strong> url, headers, transport: "streamable-http" env (optional), transport (optional).
</p> </p>
</div> </div>
{/* Test Connection */}
<div className="pt-4"> <div className="pt-4">
<Button <Button
type="button" type="button"
onClick={handleTestConnection} onClick={handleTestConnection}
disabled={isTesting} disabled={isTesting}
variant="outline" variant="secondary"
className="w-full" className="w-full h-8 text-[13px] px-3 rounded-lg font-medium bg-white text-slate-700 hover:bg-slate-50 border-0 shadow-xs dark:bg-secondary dark:text-secondary-foreground dark:hover:bg-secondary/80"
> >
{isTesting ? "Testing Connection..." : "Test Connection"} {isTesting ? "Testing Connection" : "Test Connection"}
</Button> </Button>
</div> </div>
{/* Test Result */}
{testResult && ( {testResult && (
<Alert <Alert
className={ className={
@ -226,7 +242,7 @@ export const MCPConnectForm: FC<ConnectFormProps> = ({ onSubmit, isSubmitting })
type="button" type="button"
variant="ghost" variant="ghost"
size="sm" size="sm"
className="h-6 px-2" className="h-6 px-2 self-start sm:self-auto text-xs"
onClick={(e) => { onClick={(e) => {
e.preventDefault(); e.preventDefault();
e.stopPropagation(); e.stopPropagation();
@ -236,18 +252,20 @@ export const MCPConnectForm: FC<ConnectFormProps> = ({ onSubmit, isSubmitting })
{showDetails ? ( {showDetails ? (
<> <>
<ChevronUp className="h-3 w-3 mr-1" /> <ChevronUp className="h-3 w-3 mr-1" />
Hide Details <span className="hidden sm:inline">Hide Details</span>
<span className="sm:hidden">Hide</span>
</> </>
) : ( ) : (
<> <>
<ChevronDown className="h-3 w-3 mr-1" /> <ChevronDown className="h-3 w-3 mr-1" />
Show Details <span className="hidden sm:inline">Show Details</span>
<span className="sm:hidden">Show</span>
</> </>
)} )}
</Button> </Button>
)} )}
</div> </div>
<AlertDescription className="text-xs mt-1"> <AlertDescription className="text-[10px] sm:text-xs mt-1">
{testResult.message} {testResult.message}
{showDetails && testResult.tools.length > 0 && ( {showDetails && testResult.tools.length > 0 && (
<div className="mt-3 pt-3 border-t border-green-500/20"> <div className="mt-3 pt-3 border-t border-green-500/20">

View file

@ -1,6 +1,6 @@
"use client"; "use client";
import { File, FileText, FileSpreadsheet, FolderClosed, Image, Presentation } from "lucide-react"; import { File, FileSpreadsheet, FileText, FolderClosed, Image, Presentation } from "lucide-react";
import type { FC } from "react"; import type { FC } from "react";
import { useEffect, useState } from "react"; import { useEffect, useState } from "react";
import { GoogleDriveFolderTree } from "@/components/connectors/google-drive-folder-tree"; import { GoogleDriveFolderTree } from "@/components/connectors/google-drive-folder-tree";

View file

@ -10,12 +10,12 @@ import { Label } from "@/components/ui/label";
import { Textarea } from "@/components/ui/textarea"; import { Textarea } from "@/components/ui/textarea";
import { EnumConnectorName } from "@/contracts/enums/connector"; import { EnumConnectorName } from "@/contracts/enums/connector";
import type { MCPServerConfig } from "@/contracts/types/mcp.types"; import type { MCPServerConfig } from "@/contracts/types/mcp.types";
import type { ConnectorConfigProps } from "../index";
import { import {
type MCPConnectionTestResult,
parseMCPConfig, parseMCPConfig,
testMCPConnection, testMCPConnection,
type MCPConnectionTestResult,
} from "../../utils/mcp-config-validator"; } from "../../utils/mcp-config-validator";
import type { ConnectorConfigProps } from "../index";
interface MCPConfigProps extends ConnectorConfigProps { interface MCPConfigProps extends ConnectorConfigProps {
onNameChange?: (name: string) => void; onNameChange?: (name: string) => void;
@ -47,10 +47,10 @@ export const MCPConfig: FC<MCPConfigProps> = ({ connector, onConfigChange, onNam
const serverConfig = connector.config?.server_config as MCPServerConfig | undefined; const serverConfig = connector.config?.server_config as MCPServerConfig | undefined;
if (serverConfig) { if (serverConfig) {
const transport = serverConfig.transport || "stdio"; const transport = serverConfig.transport || "stdio";
// Build config object based on transport type // Build config object based on transport type
let configObj: Record<string, unknown>; let configObj: Record<string, unknown>;
if (transport === "streamable-http" || transport === "http" || transport === "sse") { if (transport === "streamable-http" || transport === "http" || transport === "sse") {
// HTTP transport - use url and headers // HTTP transport - use url and headers
configObj = { configObj = {
@ -67,7 +67,7 @@ export const MCPConfig: FC<MCPConfigProps> = ({ connector, onConfigChange, onNam
transport: transport, transport: transport,
}; };
} }
setConfigJson(JSON.stringify(configObj, null, 2)); setConfigJson(JSON.stringify(configObj, null, 2));
} }
}, [isValidConnector, connector.name, connector.config?.server_config]); }, [isValidConnector, connector.name, connector.config?.server_config]);
@ -148,15 +148,23 @@ export const MCPConfig: FC<MCPConfigProps> = ({ connector, onConfigChange, onNam
return ( return (
<div className="space-y-6"> <div className="space-y-6">
{/* Server Name */} {/* Server Name */}
<div className="space-y-2"> <div className="rounded-xl border border-border bg-slate-400/5 dark:bg-white/5 p-3 sm:p-6 space-y-3 sm:space-y-4">
<Label htmlFor="name">Server Name *</Label> <div className="space-y-2">
<Input <Label htmlFor="name" className="text-xs sm:text-sm">
id="name" Server Name
value={name} </Label>
onChange={(e) => handleNameChange(e.target.value)} <Input
placeholder="e.g., Filesystem Server" id="name"
required value={name}
/> onChange={(e) => handleNameChange(e.target.value)}
placeholder="e.g., Filesystem Server"
className="border-slate-400/20 focus-visible:border-slate-400/40"
required
/>
<p className="text-[10px] sm:text-xs text-muted-foreground">
A friendly name to identify this connector.
</p>
</div>
</div> </div>
{/* Server Configuration */} {/* Server Configuration */}
@ -173,12 +181,29 @@ export const MCPConfig: FC<MCPConfigProps> = ({ connector, onConfigChange, onNam
id="config" id="config"
value={configJson} value={configJson}
onChange={(e) => handleConfigChange(e.target.value)} onChange={(e) => handleConfigChange(e.target.value)}
onKeyDown={(e) => {
if (e.key === "Tab") {
e.preventDefault();
const target = e.target as HTMLTextAreaElement;
const start = target.selectionStart;
const end = target.selectionEnd;
const indent = " "; // 2 spaces for JSON
const newValue =
configJson.substring(0, start) + indent + configJson.substring(end);
handleConfigChange(newValue);
// Set cursor position after the inserted tab
requestAnimationFrame(() => {
target.selectionStart = target.selectionEnd = start + indent.length;
});
}
}}
rows={16} rows={16}
className={`font-mono text-xs ${jsonError ? "border-red-500" : ""}`} className={`font-mono text-xs ${jsonError ? "border-red-500" : ""}`}
/> />
{jsonError && <p className="text-xs text-red-500">JSON Error: {jsonError}</p>} {jsonError && <p className="text-xs text-red-500">JSON Error: {jsonError}</p>}
<p className="text-[10px] sm:text-xs text-muted-foreground"> <p className="text-[10px] sm:text-xs text-muted-foreground">
<strong>Local (stdio):</strong> command, args, env, transport: "stdio"<br /> <strong>Local (stdio):</strong> command, args, env, transport: "stdio"
<br />
<strong>Remote (HTTP):</strong> url, headers, transport: "streamable-http" <strong>Remote (HTTP):</strong> url, headers, transport: "streamable-http"
</p> </p>
</div> </div>
@ -189,10 +214,10 @@ export const MCPConfig: FC<MCPConfigProps> = ({ connector, onConfigChange, onNam
type="button" type="button"
onClick={handleTestConnection} onClick={handleTestConnection}
disabled={isTesting} disabled={isTesting}
variant="outline" variant="secondary"
className="w-full" className="w-full h-8 text-[13px] px-3 rounded-lg font-medium bg-white text-slate-700 hover:bg-slate-50 border-0 shadow-xs dark:bg-secondary dark:text-secondary-foreground dark:hover:bg-secondary/80"
> >
{isTesting ? "Testing Connection..." : "Test Connection"} {isTesting ? "Testing Connection" : "Test Connection"}
</Button> </Button>
</div> </div>
@ -211,7 +236,7 @@ export const MCPConfig: FC<MCPConfigProps> = ({ connector, onConfigChange, onNam
<XCircle className="h-4 w-4 text-red-600" /> <XCircle className="h-4 w-4 text-red-600" />
)} )}
<div className="flex-1"> <div className="flex-1">
<div className="flex items-center justify-between"> <div className="flex flex-col sm:flex-row sm:items-center sm:justify-between gap-2 sm:gap-0">
<AlertTitle className="text-sm"> <AlertTitle className="text-sm">
{testResult.status === "success" {testResult.status === "success"
? "Connection Successful" ? "Connection Successful"
@ -222,7 +247,7 @@ export const MCPConfig: FC<MCPConfigProps> = ({ connector, onConfigChange, onNam
type="button" type="button"
variant="ghost" variant="ghost"
size="sm" size="sm"
className="h-6 px-2" className="h-6 px-2 self-start sm:self-auto text-xs"
onClick={(e) => { onClick={(e) => {
e.preventDefault(); e.preventDefault();
e.stopPropagation(); e.stopPropagation();
@ -232,12 +257,14 @@ export const MCPConfig: FC<MCPConfigProps> = ({ connector, onConfigChange, onNam
{showDetails ? ( {showDetails ? (
<> <>
<ChevronUp className="h-3 w-3 mr-1" /> <ChevronUp className="h-3 w-3 mr-1" />
Hide Details <span className="hidden sm:inline">Hide Details</span>
<span className="sm:hidden">Hide</span>
</> </>
) : ( ) : (
<> <>
<ChevronDown className="h-3 w-3 mr-1" /> <ChevronDown className="h-3 w-3 mr-1" />
Show Details <span className="hidden sm:inline">Show Details</span>
<span className="sm:hidden">Show</span>
</> </>
)} )}
</Button> </Button>

View file

@ -151,7 +151,7 @@ export const ConnectorEditView: FC<ConnectorEditViewProps> = ({
</div> </div>
<div className="flex-1 min-w-0"> <div className="flex-1 min-w-0">
<h2 className="text-xl sm:text-2xl font-semibold tracking-tight text-wrap whitespace-normal wrap-break-word"> <h2 className="text-xl sm:text-2xl font-semibold tracking-tight text-wrap whitespace-normal wrap-break-word">
{connector.connector_type === "MCP_CONNECTOR" ? "MCP Server" : connector.name} {connector.name}
</h2> </h2>
<p className="text-xs sm:text-base text-muted-foreground mt-1"> <p className="text-xs sm:text-base text-muted-foreground mt-1">
Manage your connector settings and sync configuration Manage your connector settings and sync configuration

View file

@ -646,7 +646,7 @@ export const useConnectorDialog = () => {
const successMessage = const successMessage =
currentConnectorType === "MCP_CONNECTOR" currentConnectorType === "MCP_CONNECTOR"
? `${connector.name} MCP server added successfully` ? `${connector.name} added successfully`
: `${connectorTitle} connected and indexing started!`; : `${connectorTitle} connected and indexing started!`;
toast.success(successMessage, { toast.success(successMessage, {
description: periodicEnabledForIndexing description: periodicEnabledForIndexing
@ -711,7 +711,7 @@ export const useConnectorDialog = () => {
// Other non-indexable connectors - just show success message and close // Other non-indexable connectors - just show success message and close
const successMessage = const successMessage =
currentConnectorType === "MCP_CONNECTOR" currentConnectorType === "MCP_CONNECTOR"
? `${connector.name} MCP server added successfully` ? `${connector.name} added successfully`
: `${connectorTitle} connected successfully!`; : `${connectorTitle} connected successfully!`;
toast.success(successMessage); toast.success(successMessage);

View file

@ -1,6 +1,7 @@
"use client"; "use client";
import type { FC } from "react"; import type { FC } from "react";
import { EnumConnectorName } from "@/contracts/enums/connector";
import type { SearchSourceConnector } from "@/contracts/types/connector.types"; import type { SearchSourceConnector } from "@/contracts/types/connector.types";
import { ConnectorCard } from "../components/connector-card"; import { ConnectorCard } from "../components/connector-card";
import { CRAWLERS, OAUTH_CONNECTORS, OTHER_CONNECTORS } from "../constants/connector-constants"; import { CRAWLERS, OAUTH_CONNECTORS, OTHER_CONNECTORS } from "../constants/connector-constants";
@ -161,6 +162,16 @@ export const AllConnectorsTab: FC<AllConnectorsTabProps> = ({
); );
const isIndexing = actualConnector && indexingConnectorIds?.has(actualConnector.id); const isIndexing = actualConnector && indexingConnectorIds?.has(actualConnector.id);
// For MCP connectors, count total MCP connectors instead of document count
const isMCP = connector.connectorType === EnumConnectorName.MCP_CONNECTOR;
const mcpConnectorCount =
isMCP && allConnectors
? allConnectors.filter(
(c: SearchSourceConnector) =>
c.connector_type === EnumConnectorName.MCP_CONNECTOR
).length
: undefined;
const handleConnect = onConnectNonOAuth const handleConnect = onConnectNonOAuth
? () => onConnectNonOAuth(connector.connectorType) ? () => onConnectNonOAuth(connector.connectorType)
: () => {}; // Fallback - connector popup should handle all connector types : () => {}; // Fallback - connector popup should handle all connector types
@ -175,6 +186,7 @@ export const AllConnectorsTab: FC<AllConnectorsTabProps> = ({
isConnected={isConnected} isConnected={isConnected}
isConnecting={isConnecting} isConnecting={isConnecting}
documentCount={documentCount} documentCount={documentCount}
connectorCount={mcpConnectorCount}
isIndexing={isIndexing} isIndexing={isIndexing}
onConnect={handleConnect} onConnect={handleConnect}
onManage={ onManage={

View file

@ -138,35 +138,37 @@ export const parseMCPConfig = (configJson: string): MCPConfigValidationResult =>
// Replace technical error messages with user-friendly ones // Replace technical error messages with user-friendly ones
if (errorMsg.includes("expected string, received undefined")) { if (errorMsg.includes("expected string, received undefined")) {
errorMsg = "This field is required"; errorMsg = fieldPath ? `The '${fieldPath}' field is required` : "This field is required";
} else if (errorMsg.includes("Invalid input")) { } else if (errorMsg.includes("Invalid input")) {
errorMsg = "Invalid value"; errorMsg = fieldPath ? `The '${fieldPath}' field has an invalid value` : "Invalid value";
} else if (fieldPath && !errorMsg.toLowerCase().includes(fieldPath.toLowerCase())) {
// If error message doesn't mention the field name, prepend it
errorMsg = `The '${fieldPath}' field: ${errorMsg}`;
} }
const formattedError = fieldPath ? `${fieldPath}: ${errorMsg}` : errorMsg; console.error("[MCP Validator] ❌ Validation error:", errorMsg);
console.error("[MCP Validator] ❌ Validation error:", formattedError);
console.error("[MCP Validator] Full Zod errors:", result.error.issues); console.error("[MCP Validator] Full Zod errors:", result.error.issues);
return { return {
config: null, config: null,
error: formattedError, error: errorMsg,
}; };
} }
// Build config based on transport type // Build config based on transport type
const config: MCPServerConfig = result.data.transport === "stdio" || !result.data.transport const config: MCPServerConfig =
? { result.data.transport === "stdio" || !result.data.transport
command: (result.data as z.infer<typeof StdioConfigSchema>).command, ? {
args: (result.data as z.infer<typeof StdioConfigSchema>).args, command: (result.data as z.infer<typeof StdioConfigSchema>).command,
env: (result.data as z.infer<typeof StdioConfigSchema>).env, args: (result.data as z.infer<typeof StdioConfigSchema>).args,
transport: "stdio" as const, env: (result.data as z.infer<typeof StdioConfigSchema>).env,
} transport: "stdio" as const,
: { }
url: (result.data as z.infer<typeof HttpConfigSchema>).url, : {
headers: (result.data as z.infer<typeof HttpConfigSchema>).headers, url: (result.data as z.infer<typeof HttpConfigSchema>).url,
transport: result.data.transport as "streamable-http" | "http" | "sse", headers: (result.data as z.infer<typeof HttpConfigSchema>).headers,
}; transport: result.data.transport as "streamable-http" | "http" | "sse",
};
// Cache the successfully parsed config // Cache the successfully parsed config
configCache.set(configJson, { configCache.set(configJson, {

View file

@ -1,9 +1,10 @@
"use client"; "use client";
import { differenceInDays, differenceInMinutes, format, isToday, isYesterday } from "date-fns"; import { differenceInDays, differenceInMinutes, format, isToday, isYesterday } from "date-fns";
import { ArrowLeft, Loader2, Plus } from "lucide-react"; import { ArrowLeft, Loader2, Plus, Server } from "lucide-react";
import type { FC } from "react"; import type { FC } from "react";
import { Button } from "@/components/ui/button"; import { Button } from "@/components/ui/button";
import { EnumConnectorName } from "@/contracts/enums/connector";
import { getConnectorIcon } from "@/contracts/enums/connectorIcons"; import { getConnectorIcon } from "@/contracts/enums/connectorIcons";
import type { SearchSourceConnector } from "@/contracts/types/connector.types"; import type { SearchSourceConnector } from "@/contracts/types/connector.types";
import { cn } from "@/lib/utils"; import { cn } from "@/lib/utils";
@ -19,6 +20,7 @@ interface ConnectorAccountsListViewProps {
onManage: (connector: SearchSourceConnector) => void; onManage: (connector: SearchSourceConnector) => void;
onAddAccount: () => void; onAddAccount: () => void;
isConnecting?: boolean; isConnecting?: boolean;
addButtonText?: string;
} }
/** /**
@ -70,6 +72,7 @@ export const ConnectorAccountsListView: FC<ConnectorAccountsListViewProps> = ({
onManage, onManage,
onAddAccount, onAddAccount,
isConnecting = false, isConnecting = false,
addButtonText,
}) => { }) => {
// Get connector status // Get connector status
const { isConnectorEnabled, getConnectorStatusMessage } = useConnectorStatus(); const { isConnectorEnabled, getConnectorStatusMessage } = useConnectorStatus();
@ -80,6 +83,22 @@ export const ConnectorAccountsListView: FC<ConnectorAccountsListViewProps> = ({
// Filter connectors to only show those of this type // Filter connectors to only show those of this type
const typeConnectors = connectors.filter((c) => c.connector_type === connectorType); const typeConnectors = connectors.filter((c) => c.connector_type === connectorType);
// Determine button text - default to "Add Account" unless specified
const buttonText =
addButtonText ||
(connectorType === EnumConnectorName.MCP_CONNECTOR ? "Add New MCP Server" : "Add Account");
const isMCP = connectorType === EnumConnectorName.MCP_CONNECTOR;
// Helper to get display name for connector (handles MCP server name extraction)
const getDisplayName = (connector: SearchSourceConnector): string => {
if (isMCP) {
// For MCP, extract server name from config if available
const serverName = connector.config?.server_config?.name || connector.name;
return serverName;
}
return getConnectorDisplayName(connector.name);
};
return ( return (
<div className="flex flex-col h-full"> <div className="flex flex-col h-full">
{/* Header */} {/* Header */}
@ -115,22 +134,22 @@ export const ConnectorAccountsListView: FC<ConnectorAccountsListViewProps> = ({
onClick={onAddAccount} onClick={onAddAccount}
disabled={isConnecting || !isEnabled} disabled={isConnecting || !isEnabled}
className={cn( className={cn(
"flex items-center gap-1.5 sm:gap-2 px-2 sm:px-3 py-1.5 sm:py-2 rounded-lg border-2 border-dashed text-left transition-all duration-200 shrink-0 self-center sm:self-auto sm:w-auto", "flex items-center justify-center gap-1.5 h-8 px-3 rounded-md border-2 border-dashed text-xs sm:text-sm transition-all duration-200 shrink-0 w-full sm:w-auto",
!isEnabled !isEnabled
? "border-border/30 opacity-50 cursor-not-allowed" ? "border-border/30 opacity-50 cursor-not-allowed"
: "border-primary/50 hover:bg-primary/5", : "border-slate-400/20 dark:border-white/20 hover:bg-primary/5",
isConnecting && "opacity-50 cursor-not-allowed" isConnecting && "opacity-50 cursor-not-allowed"
)} )}
> >
<div className="flex h-5 w-5 sm:h-6 sm:w-6 items-center justify-center rounded-md bg-primary/10 shrink-0"> <div className="flex h-5 w-5 items-center justify-center rounded-md bg-primary/10 shrink-0">
{isConnecting ? ( {isConnecting ? (
<Loader2 className="size-3 sm:size-3.5 animate-spin text-primary" /> <Loader2 className="size-3 animate-spin text-primary" />
) : ( ) : (
<Plus className="size-3 sm:size-3.5 text-primary" /> <Plus className="size-3 text-primary" />
)} )}
</div> </div>
<span className="text-[11px] sm:text-[12px] font-medium"> <span className="text-xs sm:text-sm font-medium">
{isConnecting ? "Connecting" : "Add Account"} {isConnecting ? "Connecting" : buttonText}
</span> </span>
</button> </button>
</div> </div>
@ -139,61 +158,81 @@ export const ConnectorAccountsListView: FC<ConnectorAccountsListViewProps> = ({
{/* Content */} {/* Content */}
<div className="flex-1 overflow-y-auto px-6 sm:px-12 pt-0 sm:pt-6 pb-6 sm:pb-8"> <div className="flex-1 overflow-y-auto px-6 sm:px-12 pt-0 sm:pt-6 pb-6 sm:pb-8">
{/* Connected Accounts Grid */} {/* Connected Accounts Grid */}
<div className="grid grid-cols-1 sm:grid-cols-2 gap-3"> {typeConnectors.length === 0 ? (
{typeConnectors.map((connector) => { <div className="flex flex-col items-center justify-center py-12 text-center">
const isIndexing = indexingConnectorIds.has(connector.id); <div className="h-16 w-16 rounded-full bg-slate-400/5 dark:bg-white/5 flex items-center justify-center mb-4">
{isMCP ? (
<Server className="h-8 w-8 text-muted-foreground" />
) : (
getConnectorIcon(connectorType, "size-8")
)}
</div>
<h3 className="text-sm font-medium mb-1">
{isMCP ? "No MCP Servers" : `No ${connectorTitle} Accounts`}
</h3>
<p className="text-xs text-muted-foreground max-w-[280px]">
{isMCP
? "Get started by adding your first Model Context Protocol server"
: `Get started by connecting your first ${connectorTitle} account`}
</p>
</div>
) : (
<div className="grid grid-cols-1 sm:grid-cols-2 gap-3">
{typeConnectors.map((connector) => {
const isIndexing = indexingConnectorIds.has(connector.id);
return ( return (
<div
key={connector.id}
className={cn(
"flex items-center gap-4 p-4 rounded-xl transition-all",
isIndexing
? "bg-primary/5 border-0"
: "bg-slate-400/5 dark:bg-white/5 hover:bg-slate-400/10 dark:hover:bg-white/10 border border-border"
)}
>
<div <div
key={connector.id}
className={cn( className={cn(
"flex h-12 w-12 items-center justify-center rounded-lg border shrink-0", "flex items-center gap-4 p-4 rounded-xl transition-all",
isIndexing isIndexing
? "bg-primary/10 border-primary/20" ? "bg-primary/5 border-0"
: "bg-slate-400/5 dark:bg-white/5 border-slate-400/5 dark:border-white/5" : "bg-slate-400/5 dark:bg-white/5 hover:bg-slate-400/10 dark:hover:bg-white/10 border border-border"
)} )}
> >
{getConnectorIcon(connector.connector_type, "size-6")} <div
</div> className={cn(
<div className="flex-1 min-w-0"> "flex h-12 w-12 items-center justify-center rounded-lg border shrink-0",
<p className="text-[14px] font-semibold leading-tight truncate"> isIndexing
{getConnectorDisplayName(connector.name)} ? "bg-primary/10 border-primary/20"
</p> : "bg-slate-400/5 dark:bg-white/5 border-slate-400/5 dark:border-white/5"
{isIndexing ? ( )}
<p className="text-[11px] text-primary mt-1 flex items-center gap-1.5"> >
<Loader2 className="size-3 animate-spin" /> {getConnectorIcon(connector.connector_type, "size-6")}
Syncing </div>
<div className="flex-1 min-w-0">
<p className="text-[14px] font-semibold leading-tight truncate">
{getDisplayName(connector)}
</p> </p>
) : ( {isIndexing ? (
<p className="text-[10px] text-muted-foreground mt-1 whitespace-nowrap truncate"> <p className="text-[11px] text-primary mt-1 flex items-center gap-1.5">
{isIndexableConnector(connector.connector_type) <Loader2 className="size-3 animate-spin" />
? connector.last_indexed_at Syncing
? `Last indexed: ${formatLastIndexedDate(connector.last_indexed_at)}` </p>
: "Never indexed" ) : (
: "Active"} <p className="text-[10px] text-muted-foreground mt-1 whitespace-nowrap truncate">
</p> {isIndexableConnector(connector.connector_type)
)} ? connector.last_indexed_at
? `Last indexed: ${formatLastIndexedDate(connector.last_indexed_at)}`
: "Never indexed"
: "Active"}
</p>
)}
</div>
<Button
variant="secondary"
size="sm"
className="h-8 text-[11px] px-3 rounded-lg font-medium bg-white text-slate-700 hover:bg-slate-50 border-0 shadow-xs dark:bg-secondary dark:text-secondary-foreground dark:hover:bg-secondary/80 shrink-0"
onClick={() => onManage(connector)}
>
Manage
</Button>
</div> </div>
<Button );
variant="secondary" })}
size="sm" </div>
className="h-8 text-[11px] px-3 rounded-lg font-medium bg-white text-slate-700 hover:bg-slate-50 border-0 shadow-xs dark:bg-secondary dark:text-secondary-foreground dark:hover:bg-secondary/80 shrink-0" )}
onClick={() => onManage(connector)}
>
Manage
</Button>
</div>
);
})}
</div>
</div> </div>
</div> </div>
); );

View file

@ -1,134 +0,0 @@
"use client";
import { Plus, Server, XCircle } from "lucide-react";
import type { FC } from "react";
import { Alert, AlertDescription, AlertTitle } from "@/components/ui/alert";
import { Button } from "@/components/ui/button";
import { EnumConnectorName } from "@/contracts/enums/connector";
import { getConnectorIcon } from "@/contracts/enums/connectorIcons";
import type { SearchSourceConnector } from "@/contracts/types/connector.types";
import { cn } from "@/lib/utils";
interface MCPConnectorListViewProps {
mcpConnectors: SearchSourceConnector[];
onAddNew: () => void;
onManageConnector: (connector: SearchSourceConnector) => void;
onBack: () => void;
}
export const MCPConnectorListView: FC<MCPConnectorListViewProps> = ({
mcpConnectors,
onAddNew,
onManageConnector,
onBack,
}) => {
// Validate that all connectors are MCP connectors
const invalidConnectors = mcpConnectors.filter(
(c) => c.connector_type !== EnumConnectorName.MCP_CONNECTOR
);
if (invalidConnectors.length > 0) {
console.error(
"MCPConnectorListView received non-MCP connectors:",
invalidConnectors.map((c) => c.connector_type)
);
return (
<Alert className="border-red-500/50 bg-red-500/10">
<XCircle className="h-4 w-4 text-red-600" />
<AlertTitle>Invalid Connector Type</AlertTitle>
<AlertDescription>
This view can only display MCP connectors. Found {invalidConnectors.length} invalid
connector(s).
</AlertDescription>
</Alert>
);
}
return (
<div className="flex flex-col h-full">
{/* Header */}
<div className="flex items-center justify-between mb-6 shrink-0">
<div className="flex items-center gap-3">
<Button variant="ghost" size="icon" onClick={onBack} className="h-8 w-8">
<svg
xmlns="http://www.w3.org/2000/svg"
width="16"
height="16"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
strokeWidth="2"
strokeLinecap="round"
strokeLinejoin="round"
>
<path d="m15 18-6-6 6-6" />
</svg>
</Button>
<div>
<h2 className="text-lg sm:text-xl font-semibold">MCP Connectors</h2>
<p className="text-xs sm:text-sm text-muted-foreground">
Manage your Model Context Protocol servers
</p>
</div>
</div>
</div>
{/* Add New Button */}
<div className="mb-4 shrink-0">
<Button onClick={onAddNew} className="w-full" variant="outline">
<Plus className="h-4 w-4 mr-2" />
Add New MCP Server
</Button>
</div>
{/* MCP Connectors List */}
<div className="space-y-3 flex-1 overflow-y-auto">
{mcpConnectors.length === 0 ? (
<div className="flex flex-col items-center justify-center py-12 text-center">
<div className="h-16 w-16 rounded-full bg-slate-400/5 dark:bg-white/5 flex items-center justify-center mb-4">
<Server className="h-8 w-8 text-muted-foreground" />
</div>
<h3 className="text-sm font-medium mb-1">No MCP Servers</h3>
<p className="text-xs text-muted-foreground max-w-[280px]">
Get started by adding your first Model Context Protocol server
</p>
</div>
) : (
mcpConnectors.map((connector) => {
// Extract server name from config
const serverName = connector.config?.server_config?.name || connector.name;
return (
<div
key={connector.id}
className={cn(
"flex items-center gap-4 p-4 rounded-xl border border-border transition-all",
"bg-slate-400/5 dark:bg-white/5 hover:bg-slate-400/10 dark:hover:bg-white/10"
)}
>
<div
className={cn(
"flex h-12 w-12 items-center justify-center rounded-lg border shrink-0",
"bg-slate-400/5 dark:bg-white/5 border-slate-400/5 dark:border-white/5"
)}
>
{getConnectorIcon("MCP_CONNECTOR", "size-6")}
</div>
<div className="flex-1 min-w-0">
<p className="text-[14px] font-semibold leading-tight truncate">{serverName}</p>
</div>
<Button
variant="secondary"
size="sm"
className="h-8 text-[11px] px-3 rounded-lg font-medium bg-white text-slate-700 hover:bg-slate-50 border-0 shadow-xs dark:bg-secondary dark:text-secondary-foreground dark:hover:bg-secondary/80 shrink-0"
onClick={() => onManageConnector(connector)}
>
Manage
</Button>
</div>
);
})
)}
</div>
</div>
);
};

View file

@ -11,8 +11,8 @@ import {
useState, useState,
} from "react"; } from "react";
import ReactDOMServer from "react-dom/server"; import ReactDOMServer from "react-dom/server";
import type { Document } from "@/contracts/types/document.types";
import { getConnectorIcon } from "@/contracts/enums/connectorIcons"; import { getConnectorIcon } from "@/contracts/enums/connectorIcons";
import type { Document } from "@/contracts/types/document.types";
import { cn } from "@/lib/utils"; import { cn } from "@/lib/utils";
export interface MentionedDocument { export interface MentionedDocument {

View file

@ -36,10 +36,12 @@ export function CommentPanel({
if (isLoading) { if (isLoading) {
return ( return (
<div className={cn( <div
"flex min-h-[120px] items-center justify-center p-4", className={cn(
!isMobile && "w-96 rounded-lg border bg-card" "flex min-h-[120px] items-center justify-center p-4",
)}> !isMobile && "w-96 rounded-lg border bg-card"
)}
>
<div className="flex items-center gap-2 text-sm text-muted-foreground"> <div className="flex items-center gap-2 text-sm text-muted-foreground">
<div className="size-4 animate-spin rounded-full border-2 border-current border-t-transparent" /> <div className="size-4 animate-spin rounded-full border-2 border-current border-t-transparent" />
Loading comments... Loading comments...
@ -57,10 +59,7 @@ export function CommentPanel({
return ( return (
<div <div
className={cn( className={cn("flex flex-col", isMobile ? "w-full" : "w-85 rounded-lg border bg-card")}
"flex flex-col",
isMobile ? "w-full" : "w-85 rounded-lg border bg-card"
)}
style={!isMobile && effectiveMaxHeight ? { maxHeight: effectiveMaxHeight } : undefined} style={!isMobile && effectiveMaxHeight ? { maxHeight: effectiveMaxHeight } : undefined}
> >
{hasThreads && ( {hasThreads && (
@ -92,11 +91,7 @@ export function CommentPanel({
</div> </div>
)} )}
<div className={cn( <div className={cn("p-3", showEmptyState && !isMobile && "border-t", isMobile && "border-t")}>
"p-3",
showEmptyState && !isMobile && "border-t",
isMobile && "border-t"
)}>
{isComposerOpen ? ( {isComposerOpen ? (
<CommentComposer <CommentComposer
members={members} members={members}

View file

@ -1,12 +1,7 @@
"use client"; "use client";
import { MessageSquare } from "lucide-react"; import { MessageSquare } from "lucide-react";
import { import { Sheet, SheetContent, SheetHeader, SheetTitle } from "@/components/ui/sheet";
Sheet,
SheetContent,
SheetHeader,
SheetTitle,
} from "@/components/ui/sheet";
import { cn } from "@/lib/utils"; import { cn } from "@/lib/utils";
import { CommentPanelContainer } from "../comment-panel-container/comment-panel-container"; import { CommentPanelContainer } from "../comment-panel-container/comment-panel-container";
import type { CommentSheetProps } from "./types"; import type { CommentSheetProps } from "./types";
@ -26,9 +21,7 @@ export function CommentSheet({
side={side} side={side}
className={cn( className={cn(
"flex flex-col p-0", "flex flex-col p-0",
isBottomSheet isBottomSheet ? "h-[85vh] max-h-[85vh] rounded-t-xl" : "h-full w-full max-w-md"
? "h-[85vh] max-h-[85vh] rounded-t-xl"
: "h-full w-full max-w-md"
)} )}
> >
{/* Drag handle indicator - only for bottom sheet */} {/* Drag handle indicator - only for bottom sheet */}
@ -37,10 +30,7 @@ export function CommentSheet({
<div className="h-1 w-10 rounded-full bg-muted-foreground/30" /> <div className="h-1 w-10 rounded-full bg-muted-foreground/30" />
</div> </div>
)} )}
<SheetHeader className={cn( <SheetHeader className={cn("flex-shrink-0 border-b px-4", isBottomSheet ? "pb-3" : "py-4")}>
"flex-shrink-0 border-b px-4",
isBottomSheet ? "pb-3" : "py-4"
)}>
<SheetTitle className="flex items-center gap-2 text-base font-semibold"> <SheetTitle className="flex items-center gap-2 text-base font-semibold">
<MessageSquare className="size-5" /> <MessageSquare className="size-5" />
Comments Comments
@ -52,11 +42,7 @@ export function CommentSheet({
</SheetTitle> </SheetTitle>
</SheetHeader> </SheetHeader>
<div className="min-h-0 flex-1 overflow-y-auto"> <div className="min-h-0 flex-1 overflow-y-auto">
<CommentPanelContainer <CommentPanelContainer messageId={messageId} isOpen={true} variant="mobile" />
messageId={messageId}
isOpen={true}
variant="mobile"
/>
</div> </div>
</SheetContent> </SheetContent>
</Sheet> </Sheet>

View file

@ -4,6 +4,7 @@ import {
ChevronDown, ChevronDown,
ChevronRight, ChevronRight,
File, File,
FileSpreadsheet,
FileText, FileText,
FolderClosed, FolderClosed,
FolderOpen, FolderOpen,
@ -11,7 +12,6 @@ import {
Image, Image,
Loader2, Loader2,
Presentation, Presentation,
FileSpreadsheet,
} from "lucide-react"; } from "lucide-react";
import { useState } from "react"; import { useState } from "react";
import { Checkbox } from "@/components/ui/checkbox"; import { Checkbox } from "@/components/ui/checkbox";

View file

@ -28,21 +28,22 @@ export function Header({
const hasThread = isChatPage && currentThreadState.id !== null; const hasThread = isChatPage && currentThreadState.id !== null;
// Create minimal thread object for ChatShareButton (used for API calls) // Create minimal thread object for ChatShareButton (used for API calls)
const threadForButton: ThreadRecord | null = hasThread const threadForButton: ThreadRecord | null =
? { hasThread && currentThreadState.id !== null
id: currentThreadState.id!, ? {
visibility: currentThreadState.visibility ?? "PRIVATE", id: currentThreadState.id,
// These fields are not used by ChatShareButton for display, only for checks visibility: currentThreadState.visibility ?? "PRIVATE",
created_by_id: null, // These fields are not used by ChatShareButton for display, only for checks
search_space_id: 0, created_by_id: null,
title: "", search_space_id: 0,
archived: false, title: "",
created_at: "", archived: false,
updated_at: "", created_at: "",
} updated_at: "",
: null; }
: null;
const handleVisibilityChange = (visibility: ChatVisibility) => { const handleVisibilityChange = (_visibility: ChatVisibility) => {
// Visibility change is handled by ChatShareButton internally via Jotai // Visibility change is handled by ChatShareButton internally via Jotai
// This callback can be used for additional side effects if needed // This callback can be used for additional side effects if needed
}; };

View file

@ -1,16 +1,16 @@
"use client"; "use client";
import { useState } from "react"; import { useAtomValue } from "jotai";
import { Bell } from "lucide-react"; import { Bell } from "lucide-react";
import { useParams } from "next/navigation";
import { useState } from "react";
import { currentUserAtom } from "@/atoms/user/user-query.atoms";
import { Button } from "@/components/ui/button"; import { Button } from "@/components/ui/button";
import { Popover, PopoverContent, PopoverTrigger } from "@/components/ui/popover"; import { Popover, PopoverContent, PopoverTrigger } from "@/components/ui/popover";
import { Tooltip, TooltipContent, TooltipTrigger } from "@/components/ui/tooltip"; import { Tooltip, TooltipContent, TooltipTrigger } from "@/components/ui/tooltip";
import { useNotifications } from "@/hooks/use-notifications"; import { useNotifications } from "@/hooks/use-notifications";
import { useAtomValue } from "jotai";
import { currentUserAtom } from "@/atoms/user/user-query.atoms";
import { NotificationPopup } from "./NotificationPopup";
import { cn } from "@/lib/utils"; import { cn } from "@/lib/utils";
import { useParams } from "next/navigation"; import { NotificationPopup } from "./NotificationPopup";
export function NotificationButton() { export function NotificationButton() {
const [open, setOpen] = useState(false); const [open, setOpen] = useState(false);

View file

@ -1,14 +1,14 @@
"use client"; "use client";
import { Bell, CheckCheck, Loader2, AlertCircle, CheckCircle2 } from "lucide-react"; import { formatDistanceToNow } from "date-fns";
import { AlertCircle, Bell, CheckCheck, CheckCircle2, Loader2 } from "lucide-react";
import { useRouter } from "next/navigation"; import { useRouter } from "next/navigation";
import { convertRenderedToDisplay } from "@/components/chat-comments/comment-item/comment-item";
import { Button } from "@/components/ui/button"; import { Button } from "@/components/ui/button";
import { ScrollArea } from "@/components/ui/scroll-area"; import { ScrollArea } from "@/components/ui/scroll-area";
import { Separator } from "@/components/ui/separator"; import { Separator } from "@/components/ui/separator";
import type { Notification } from "@/hooks/use-notifications"; import type { Notification } from "@/hooks/use-notifications";
import { formatDistanceToNow } from "date-fns";
import { cn } from "@/lib/utils"; import { cn } from "@/lib/utils";
import { convertRenderedToDisplay } from "@/components/chat-comments/comment-item/comment-item";
interface NotificationPopupProps { interface NotificationPopupProps {
notifications: Notification[]; notifications: Notification[];

View file

@ -1,13 +1,13 @@
"use client"; "use client";
import { useEffect, useState, useRef } from "react";
import { useAtomValue } from "jotai"; import { useAtomValue } from "jotai";
import { useEffect, useRef, useState } from "react";
import { currentUserAtom } from "@/atoms/user/user-query.atoms"; import { currentUserAtom } from "@/atoms/user/user-query.atoms";
import { import {
initElectric,
cleanupElectric, cleanupElectric,
isElectricInitialized,
type ElectricClient, type ElectricClient,
initElectric,
isElectricInitialized,
} from "@/lib/electric/client"; } from "@/lib/electric/client";
import { ElectricContext } from "@/lib/electric/context"; import { ElectricContext } from "@/lib/electric/context";

View file

@ -26,7 +26,7 @@ Make sure to include the `-v surfsense-data:/data` in your Docker command. This
**Linux/macOS:** **Linux/macOS:**
```bash ```bash
docker run -d -p 3000:3000 -p 8000:8000 \ docker run -d -p 3000:3000 -p 8000:8000 -p 5133:5133 \
-v surfsense-data:/data \ -v surfsense-data:/data \
--name surfsense \ --name surfsense \
--restart unless-stopped \ --restart unless-stopped \
@ -36,7 +36,7 @@ docker run -d -p 3000:3000 -p 8000:8000 \
**Windows (PowerShell):** **Windows (PowerShell):**
```powershell ```powershell
docker run -d -p 3000:3000 -p 8000:8000 ` docker run -d -p 3000:3000 -p 8000:8000 -p 5133:5133 `
-v surfsense-data:/data ` -v surfsense-data:/data `
--name surfsense ` --name surfsense `
--restart unless-stopped ` --restart unless-stopped `
@ -50,7 +50,7 @@ docker run -d -p 3000:3000 -p 8000:8000 `
You can pass any [environment variable](/docs/manual-installation#backend-environment-variables) using `-e` flags: You can pass any [environment variable](/docs/manual-installation#backend-environment-variables) using `-e` flags:
```bash ```bash
docker run -d -p 3000:3000 -p 8000:8000 \ docker run -d -p 3000:3000 -p 8000:8000 -p 5133:5133 \
-v surfsense-data:/data \ -v surfsense-data:/data \
-e EMBEDDING_MODEL=openai://text-embedding-ada-002 \ -e EMBEDDING_MODEL=openai://text-embedding-ada-002 \
-e OPENAI_API_KEY=your_openai_api_key \ -e OPENAI_API_KEY=your_openai_api_key \
@ -93,6 +93,7 @@ After starting, access SurfSense at:
- **Frontend**: [http://localhost:3000](http://localhost:3000) - **Frontend**: [http://localhost:3000](http://localhost:3000)
- **Backend API**: [http://localhost:8000](http://localhost:8000) - **Backend API**: [http://localhost:8000](http://localhost:8000)
- **API Docs**: [http://localhost:8000/docs](http://localhost:8000/docs) - **API Docs**: [http://localhost:8000/docs](http://localhost:8000/docs)
- **Electric-SQL**: [http://localhost:5133](http://localhost:5133)
### Quick Start Environment Variables ### Quick Start Environment Variables
@ -195,6 +196,11 @@ Before you begin, ensure you have:
| NEXT_PUBLIC_FASTAPI_BACKEND_URL | URL of the backend API (used by frontend during build and runtime) | http://localhost:8000 | | NEXT_PUBLIC_FASTAPI_BACKEND_URL | URL of the backend API (used by frontend during build and runtime) | http://localhost:8000 |
| NEXT_PUBLIC_FASTAPI_BACKEND_AUTH_TYPE | Authentication method for frontend: `LOCAL` or `GOOGLE` | LOCAL | | NEXT_PUBLIC_FASTAPI_BACKEND_AUTH_TYPE | Authentication method for frontend: `LOCAL` or `GOOGLE` | LOCAL |
| NEXT_PUBLIC_ETL_SERVICE | Document parsing service for frontend UI: `UNSTRUCTURED`, `LLAMACLOUD`, or `DOCLING` | DOCLING | | NEXT_PUBLIC_ETL_SERVICE | Document parsing service for frontend UI: `UNSTRUCTURED`, `LLAMACLOUD`, or `DOCLING` | DOCLING |
| ELECTRIC_PORT | Port for Electric-SQL service | 5133 |
| POSTGRES_HOST | PostgreSQL host for Electric connection (`db` for Docker PostgreSQL, `host.docker.internal` for local PostgreSQL) | db |
| ELECTRIC_DB_USER | PostgreSQL username for Electric connection | electric |
| ELECTRIC_DB_PASSWORD | PostgreSQL password for Electric connection | electric_password |
| NEXT_PUBLIC_ELECTRIC_URL | URL for Electric-SQL service (used by frontend) | http://localhost:5133 |
**Note:** Frontend environment variables with the `NEXT_PUBLIC_` prefix are embedded into the Next.js production build at build time. Since the frontend now runs as a production build in Docker, these variables must be set in the root `.env` file (Docker-specific configuration) and will be passed as build arguments during the Docker build process. **Note:** Frontend environment variables with the `NEXT_PUBLIC_` prefix are embedded into the Next.js production build at build time. Since the frontend now runs as a production build in Docker, these variables must be set in the root `.env` file (Docker-specific configuration) and will be passed as build arguments during the Docker build process.
@ -209,7 +215,8 @@ Before you begin, ensure you have:
| AUTH_TYPE | Authentication method: `GOOGLE` for OAuth with Google, `LOCAL` for email/password authentication | | AUTH_TYPE | Authentication method: `GOOGLE` for OAuth with Google, `LOCAL` for email/password authentication |
| GOOGLE_OAUTH_CLIENT_ID | (Optional) Client ID from Google Cloud Console (required if AUTH_TYPE=GOOGLE) | | GOOGLE_OAUTH_CLIENT_ID | (Optional) Client ID from Google Cloud Console (required if AUTH_TYPE=GOOGLE) |
| GOOGLE_OAUTH_CLIENT_SECRET | (Optional) Client secret from Google Cloud Console (required if AUTH_TYPE=GOOGLE) | | GOOGLE_OAUTH_CLIENT_SECRET | (Optional) Client secret from Google Cloud Console (required if AUTH_TYPE=GOOGLE) |
| GOOGLE_DRIVE_REDIRECT_URI | (Optional) Redirect URI for Google Drive connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/google/drive/connector/callback`). Required when using Google Drive connector. | | ELECTRIC_DB_USER | (Optional) PostgreSQL username for Electric-SQL connection (default: `electric`) |
| ELECTRIC_DB_PASSWORD | (Optional) PostgreSQL password for Electric-SQL connection (default: `electric_password`) |
| EMBEDDING_MODEL | Name of the embedding model (e.g., `sentence-transformers/all-MiniLM-L6-v2`, `openai://text-embedding-ada-002`) | | EMBEDDING_MODEL | Name of the embedding model (e.g., `sentence-transformers/all-MiniLM-L6-v2`, `openai://text-embedding-ada-002`) |
| RERANKERS_ENABLED | (Optional) Enable or disable document reranking for improved search results (e.g., `TRUE` or `FALSE`, default: `FALSE`) | | RERANKERS_ENABLED | (Optional) Enable or disable document reranking for improved search results (e.g., `TRUE` or `FALSE`, default: `FALSE`) |
| RERANKERS_MODEL_NAME | Name of the reranker model (e.g., `ms-marco-MiniLM-L-12-v2`) (required if RERANKERS_ENABLED=TRUE) | | RERANKERS_MODEL_NAME | Name of the reranker model (e.g., `ms-marco-MiniLM-L-12-v2`) (required if RERANKERS_ENABLED=TRUE) |
@ -230,6 +237,44 @@ Before you begin, ensure you have:
| REGISTRATION_ENABLED | (Optional) Enable or disable new user registration (e.g., `TRUE` or `FALSE`, default: `TRUE`) | | REGISTRATION_ENABLED | (Optional) Enable or disable new user registration (e.g., `TRUE` or `FALSE`, default: `TRUE`) |
| PAGES_LIMIT | (Optional) Maximum pages limit per user for ETL services (default: `999999999` for unlimited in OSS version) | | PAGES_LIMIT | (Optional) Maximum pages limit per user for ETL services (default: `999999999` for unlimited in OSS version) |
**Google Connector OAuth Configuration:**
| ENV VARIABLE | DESCRIPTION |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| GOOGLE_CALENDAR_REDIRECT_URI | (Optional) Redirect URI for Google Calendar connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/google/calendar/connector/callback`) |
| GOOGLE_GMAIL_REDIRECT_URI | (Optional) Redirect URI for Gmail connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/google/gmail/connector/callback`) |
| GOOGLE_DRIVE_REDIRECT_URI | (Optional) Redirect URI for Google Drive connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/google/drive/connector/callback`) |
**Connector OAuth Configurations (Optional):**
| ENV VARIABLE | DESCRIPTION |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| AIRTABLE_CLIENT_ID | (Optional) Airtable OAuth client ID from [Airtable Developer Hub](https://airtable.com/create/oauth) |
| AIRTABLE_CLIENT_SECRET | (Optional) Airtable OAuth client secret |
| AIRTABLE_REDIRECT_URI | (Optional) Redirect URI for Airtable connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/airtable/connector/callback`) |
| CLICKUP_CLIENT_ID | (Optional) ClickUp OAuth client ID |
| CLICKUP_CLIENT_SECRET | (Optional) ClickUp OAuth client secret |
| CLICKUP_REDIRECT_URI | (Optional) Redirect URI for ClickUp connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/clickup/connector/callback`) |
| DISCORD_CLIENT_ID | (Optional) Discord OAuth client ID |
| DISCORD_CLIENT_SECRET | (Optional) Discord OAuth client secret |
| DISCORD_REDIRECT_URI | (Optional) Redirect URI for Discord connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/discord/connector/callback`) |
| DISCORD_BOT_TOKEN | (Optional) Discord bot token from Developer Portal |
| ATLASSIAN_CLIENT_ID | (Optional) Atlassian OAuth client ID (for Jira and Confluence) |
| ATLASSIAN_CLIENT_SECRET | (Optional) Atlassian OAuth client secret |
| JIRA_REDIRECT_URI | (Optional) Redirect URI for Jira connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/jira/connector/callback`) |
| CONFLUENCE_REDIRECT_URI | (Optional) Redirect URI for Confluence connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/confluence/connector/callback`) |
| LINEAR_CLIENT_ID | (Optional) Linear OAuth client ID |
| LINEAR_CLIENT_SECRET | (Optional) Linear OAuth client secret |
| LINEAR_REDIRECT_URI | (Optional) Redirect URI for Linear connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/linear/connector/callback`) |
| NOTION_CLIENT_ID | (Optional) Notion OAuth client ID |
| NOTION_CLIENT_SECRET | (Optional) Notion OAuth client secret |
| NOTION_REDIRECT_URI | (Optional) Redirect URI for Notion connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/notion/connector/callback`) |
| SLACK_CLIENT_ID | (Optional) Slack OAuth client ID |
| SLACK_CLIENT_SECRET | (Optional) Slack OAuth client secret |
| SLACK_REDIRECT_URI | (Optional) Redirect URI for Slack connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/slack/connector/callback`) |
| TEAMS_CLIENT_ID | (Optional) Microsoft Teams OAuth client ID |
| TEAMS_CLIENT_SECRET | (Optional) Microsoft Teams OAuth client secret |
| TEAMS_REDIRECT_URI | (Optional) Redirect URI for Teams connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/teams/connector/callback`) |
**Optional Backend LangSmith Observability:** **Optional Backend LangSmith Observability:**
| ENV VARIABLE | DESCRIPTION | | ENV VARIABLE | DESCRIPTION |
@ -282,6 +327,8 @@ For more details, see the [Uvicorn documentation](https://www.uvicorn.org/#comma
- `NEXT_PUBLIC_FASTAPI_BACKEND_URL` - URL of the backend service - `NEXT_PUBLIC_FASTAPI_BACKEND_URL` - URL of the backend service
- `NEXT_PUBLIC_FASTAPI_BACKEND_AUTH_TYPE` - Authentication method (`LOCAL` or `GOOGLE`) - `NEXT_PUBLIC_FASTAPI_BACKEND_AUTH_TYPE` - Authentication method (`LOCAL` or `GOOGLE`)
- `NEXT_PUBLIC_ETL_SERVICE` - Document parsing service (should match backend `ETL_SERVICE`) - `NEXT_PUBLIC_ETL_SERVICE` - Document parsing service (should match backend `ETL_SERVICE`)
- `NEXT_PUBLIC_ELECTRIC_URL` - URL for Electric-SQL service (default: `http://localhost:5133`)
- `NEXT_PUBLIC_ELECTRIC_AUTH_MODE` - Electric-SQL authentication mode (default: `insecure`)
These variables are embedded into the application during the Docker build process and affect the frontend's behavior and available features. These variables are embedded into the application during the Docker build process and affect the frontend's behavior and available features.
@ -312,6 +359,7 @@ These variables are embedded into the application during the Docker build proces
- Frontend: [http://localhost:3000](http://localhost:3000) - Frontend: [http://localhost:3000](http://localhost:3000)
- Backend API: [http://localhost:8000](http://localhost:8000) - Backend API: [http://localhost:8000](http://localhost:8000)
- API Documentation: [http://localhost:8000/docs](http://localhost:8000/docs) - API Documentation: [http://localhost:8000/docs](http://localhost:8000/docs)
- Electric-SQL: [http://localhost:5133](http://localhost:5133)
- pgAdmin: [http://localhost:5050](http://localhost:5050) - pgAdmin: [http://localhost:5050](http://localhost:5050)
## Docker Services Overview ## Docker Services Overview
@ -322,6 +370,7 @@ The Docker setup includes several services that work together:
- **Frontend**: Next.js web application - **Frontend**: Next.js web application
- **PostgreSQL (db)**: Database with pgvector extension - **PostgreSQL (db)**: Database with pgvector extension
- **Redis**: Message broker for Celery - **Redis**: Message broker for Celery
- **Electric-SQL**: Real-time sync service for database operations
- **Celery Worker**: Handles background tasks (document processing, indexing, etc.) - **Celery Worker**: Handles background tasks (document processing, indexing, etc.)
- **Celery Beat**: Scheduler for periodic tasks (enables scheduled connector indexing) - **Celery Beat**: Scheduler for periodic tasks (enables scheduled connector indexing)
- The schedule interval can be configured using the `SCHEDULE_CHECKER_INTERVAL` environment variable in your backend `.env` file - The schedule interval can be configured using the `SCHEDULE_CHECKER_INTERVAL` environment variable in your backend `.env` file

View file

@ -72,7 +72,8 @@ Edit the `.env` file and set the following variables:
| AUTH_TYPE | Authentication method: `GOOGLE` for OAuth with Google, `LOCAL` for email/password authentication | | AUTH_TYPE | Authentication method: `GOOGLE` for OAuth with Google, `LOCAL` for email/password authentication |
| GOOGLE_OAUTH_CLIENT_ID | (Optional) Client ID from Google Cloud Console (required if AUTH_TYPE=GOOGLE) | | GOOGLE_OAUTH_CLIENT_ID | (Optional) Client ID from Google Cloud Console (required if AUTH_TYPE=GOOGLE) |
| GOOGLE_OAUTH_CLIENT_SECRET | (Optional) Client secret from Google Cloud Console (required if AUTH_TYPE=GOOGLE) | | GOOGLE_OAUTH_CLIENT_SECRET | (Optional) Client secret from Google Cloud Console (required if AUTH_TYPE=GOOGLE) |
| GOOGLE_DRIVE_REDIRECT_URI | (Optional) Redirect URI for Google Drive connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/google/drive/connector/callback`). Required when using Google Drive connector. | | ELECTRIC_DB_USER | (Optional) PostgreSQL username for Electric-SQL connection (default: `electric`) |
| ELECTRIC_DB_PASSWORD | (Optional) PostgreSQL password for Electric-SQL connection (default: `electric_password`) |
| EMBEDDING_MODEL | Name of the embedding model (e.g., `sentence-transformers/all-MiniLM-L6-v2`, `openai://text-embedding-ada-002`) | | EMBEDDING_MODEL | Name of the embedding model (e.g., `sentence-transformers/all-MiniLM-L6-v2`, `openai://text-embedding-ada-002`) |
| RERANKERS_ENABLED | (Optional) Enable or disable document reranking for improved search results (e.g., `TRUE` or `FALSE`, default: `FALSE`) | | RERANKERS_ENABLED | (Optional) Enable or disable document reranking for improved search results (e.g., `TRUE` or `FALSE`, default: `FALSE`) |
| RERANKERS_MODEL_NAME | Name of the reranker model (e.g., `ms-marco-MiniLM-L-12-v2`) (required if RERANKERS_ENABLED=TRUE) | | RERANKERS_MODEL_NAME | Name of the reranker model (e.g., `ms-marco-MiniLM-L-12-v2`) (required if RERANKERS_ENABLED=TRUE) |
@ -83,6 +84,7 @@ Edit the `.env` file and set the following variables:
| STT_SERVICE | Speech-to-Text API provider for Audio Files (e.g., `local/base`, `openai/whisper-1`). See [supported providers](https://docs.litellm.ai/docs/audio_transcription#supported-providers) | | STT_SERVICE | Speech-to-Text API provider for Audio Files (e.g., `local/base`, `openai/whisper-1`). See [supported providers](https://docs.litellm.ai/docs/audio_transcription#supported-providers) |
| STT_SERVICE_API_KEY | (Optional if local) API key for the Speech-to-Text service | | STT_SERVICE_API_KEY | (Optional if local) API key for the Speech-to-Text service |
| STT_SERVICE_API_BASE | (Optional) Custom API base URL for the Speech-to-Text service | | STT_SERVICE_API_BASE | (Optional) Custom API base URL for the Speech-to-Text service |
| FIRECRAWL_API_KEY | (Optional) API key for Firecrawl service for web crawling |
| ETL_SERVICE | Document parsing service: `UNSTRUCTURED` (supports 34+ formats), `LLAMACLOUD` (supports 50+ formats including legacy document types), or `DOCLING` (local processing, supports PDF, Office docs, images, HTML, CSV) | | ETL_SERVICE | Document parsing service: `UNSTRUCTURED` (supports 34+ formats), `LLAMACLOUD` (supports 50+ formats including legacy document types), or `DOCLING` (local processing, supports PDF, Office docs, images, HTML, CSV) |
| UNSTRUCTURED_API_KEY | API key for Unstructured.io service for document parsing (required if ETL_SERVICE=UNSTRUCTURED) | | UNSTRUCTURED_API_KEY | API key for Unstructured.io service for document parsing (required if ETL_SERVICE=UNSTRUCTURED) |
| LLAMA_CLOUD_API_KEY | API key for LlamaCloud service for document parsing (required if ETL_SERVICE=LLAMACLOUD) | | LLAMA_CLOUD_API_KEY | API key for LlamaCloud service for document parsing (required if ETL_SERVICE=LLAMACLOUD) |
@ -92,6 +94,43 @@ Edit the `.env` file and set the following variables:
| REGISTRATION_ENABLED | (Optional) Enable or disable new user registration (e.g., `TRUE` or `FALSE`, default: `TRUE`) | | REGISTRATION_ENABLED | (Optional) Enable or disable new user registration (e.g., `TRUE` or `FALSE`, default: `TRUE`) |
| PAGES_LIMIT | (Optional) Maximum pages limit per user for ETL services (default: `999999999` for unlimited in OSS version) | | PAGES_LIMIT | (Optional) Maximum pages limit per user for ETL services (default: `999999999` for unlimited in OSS version) |
**Google Connector OAuth Configuration:**
| ENV VARIABLE | DESCRIPTION |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| GOOGLE_CALENDAR_REDIRECT_URI | (Optional) Redirect URI for Google Calendar connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/google/calendar/connector/callback`) |
| GOOGLE_GMAIL_REDIRECT_URI | (Optional) Redirect URI for Gmail connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/google/gmail/connector/callback`) |
| GOOGLE_DRIVE_REDIRECT_URI | (Optional) Redirect URI for Google Drive connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/google/drive/connector/callback`) |
**Connector OAuth Configurations (Optional):**
| ENV VARIABLE | DESCRIPTION |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| AIRTABLE_CLIENT_ID | (Optional) Airtable OAuth client ID from [Airtable Developer Hub](https://airtable.com/create/oauth) |
| AIRTABLE_CLIENT_SECRET | (Optional) Airtable OAuth client secret |
| AIRTABLE_REDIRECT_URI | (Optional) Redirect URI for Airtable connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/airtable/connector/callback`) |
| CLICKUP_CLIENT_ID | (Optional) ClickUp OAuth client ID |
| CLICKUP_CLIENT_SECRET | (Optional) ClickUp OAuth client secret |
| CLICKUP_REDIRECT_URI | (Optional) Redirect URI for ClickUp connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/clickup/connector/callback`) |
| DISCORD_CLIENT_ID | (Optional) Discord OAuth client ID |
| DISCORD_CLIENT_SECRET | (Optional) Discord OAuth client secret |
| DISCORD_REDIRECT_URI | (Optional) Redirect URI for Discord connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/discord/connector/callback`) |
| DISCORD_BOT_TOKEN | (Optional) Discord bot token from Developer Portal |
| ATLASSIAN_CLIENT_ID | (Optional) Atlassian OAuth client ID (for Jira and Confluence) |
| ATLASSIAN_CLIENT_SECRET | (Optional) Atlassian OAuth client secret |
| JIRA_REDIRECT_URI | (Optional) Redirect URI for Jira connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/jira/connector/callback`) |
| CONFLUENCE_REDIRECT_URI | (Optional) Redirect URI for Confluence connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/confluence/connector/callback`) |
| LINEAR_CLIENT_ID | (Optional) Linear OAuth client ID |
| LINEAR_CLIENT_SECRET | (Optional) Linear OAuth client secret |
| LINEAR_REDIRECT_URI | (Optional) Redirect URI for Linear connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/linear/connector/callback`) |
| NOTION_CLIENT_ID | (Optional) Notion OAuth client ID |
| NOTION_CLIENT_SECRET | (Optional) Notion OAuth client secret |
| NOTION_REDIRECT_URI | (Optional) Redirect URI for Notion connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/notion/connector/callback`) |
| SLACK_CLIENT_ID | (Optional) Slack OAuth client ID |
| SLACK_CLIENT_SECRET | (Optional) Slack OAuth client secret |
| SLACK_REDIRECT_URI | (Optional) Redirect URI for Slack connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/slack/connector/callback`) |
| TEAMS_CLIENT_ID | (Optional) Microsoft Teams OAuth client ID |
| TEAMS_CLIENT_SECRET | (Optional) Microsoft Teams OAuth client secret |
| TEAMS_REDIRECT_URI | (Optional) Redirect URI for Teams connector OAuth callback (e.g., `http://localhost:8000/api/v1/auth/teams/connector/callback`) |
**(Optional) Backend LangSmith Observability:** **(Optional) Backend LangSmith Observability:**
| ENV VARIABLE | DESCRIPTION | | ENV VARIABLE | DESCRIPTION |
@ -368,6 +407,8 @@ Edit the `.env` file and set:
| NEXT_PUBLIC_FASTAPI_BACKEND_URL | Backend URL (e.g., `http://localhost:8000`) | | NEXT_PUBLIC_FASTAPI_BACKEND_URL | Backend URL (e.g., `http://localhost:8000`) |
| NEXT_PUBLIC_FASTAPI_BACKEND_AUTH_TYPE | Same value as set in backend AUTH_TYPE i.e `GOOGLE` for OAuth with Google, `LOCAL` for email/password authentication | | NEXT_PUBLIC_FASTAPI_BACKEND_AUTH_TYPE | Same value as set in backend AUTH_TYPE i.e `GOOGLE` for OAuth with Google, `LOCAL` for email/password authentication |
| NEXT_PUBLIC_ETL_SERVICE | Document parsing service (should match backend ETL_SERVICE): `UNSTRUCTURED`, `LLAMACLOUD`, or `DOCLING` - affects supported file formats in upload interface | | NEXT_PUBLIC_ETL_SERVICE | Document parsing service (should match backend ETL_SERVICE): `UNSTRUCTURED`, `LLAMACLOUD`, or `DOCLING` - affects supported file formats in upload interface |
| NEXT_PUBLIC_ELECTRIC_URL | URL for Electric-SQL service (e.g., `http://localhost:5133`) |
| NEXT_PUBLIC_ELECTRIC_AUTH_MODE | Electric-SQL authentication mode (default: `insecure`) |
### 2. Install Dependencies ### 2. Install Dependencies

View file

@ -65,7 +65,7 @@ export const getConnectorIcon = (connectorType: EnumConnectorName | string, clas
case EnumConnectorName.CIRCLEBACK_CONNECTOR: case EnumConnectorName.CIRCLEBACK_CONNECTOR:
return <IconUsersGroup {...iconProps} />; return <IconUsersGroup {...iconProps} />;
case EnumConnectorName.MCP_CONNECTOR: case EnumConnectorName.MCP_CONNECTOR:
return <Webhook {...iconProps} />; return <Image src="/connectors/modelcontextprotocol.svg" alt="MCP" {...imgProps} />;
// Additional cases for non-enum connector types // Additional cases for non-enum connector types
case "YOUTUBE_CONNECTOR": case "YOUTUBE_CONNECTOR":
return <Image src="/connectors/youtube.svg" alt="YouTube" {...imgProps} />; return <Image src="/connectors/youtube.svg" alt="YouTube" {...imgProps} />;

View file

@ -1,9 +1,9 @@
"use client"; "use client";
import { useEffect, useState, useCallback, useRef } from "react"; import { useCallback, useEffect, useRef, useState } from "react";
import { useElectricClient } from "@/lib/electric/context";
import type { SyncHandle } from "@/lib/electric/client";
import type { SearchSourceConnector } from "@/contracts/types/connector.types"; import type { SearchSourceConnector } from "@/contracts/types/connector.types";
import type { SyncHandle } from "@/lib/electric/client";
import { useElectricClient } from "@/lib/electric/context";
/** /**
* Hook for managing connectors with Electric SQL real-time sync * Hook for managing connectors with Electric SQL real-time sync

View file

@ -1,8 +1,8 @@
"use client"; "use client";
import { useEffect, useState, useRef, useMemo } from "react"; import { useEffect, useMemo, useRef, useState } from "react";
import { useElectricClient } from "@/lib/electric/context";
import type { SyncHandle } from "@/lib/electric/client"; import type { SyncHandle } from "@/lib/electric/client";
import { useElectricClient } from "@/lib/electric/context";
interface Document { interface Document {
id: number; id: number;

View file

@ -1,10 +1,10 @@
"use client"; "use client";
import { useEffect, useState, useCallback, useRef } from "react"; import { useCallback, useEffect, useRef, useState } from "react";
import { useElectricClient } from "@/lib/electric/context";
import type { SyncHandle } from "@/lib/electric/client";
import type { Notification } from "@/contracts/types/notification.types"; import type { Notification } from "@/contracts/types/notification.types";
import { authenticatedFetch } from "@/lib/auth-utils"; import { authenticatedFetch } from "@/lib/auth-utils";
import type { SyncHandle } from "@/lib/electric/client";
import { useElectricClient } from "@/lib/electric/context";
export type { Notification } from "@/contracts/types/notification.types"; export type { Notification } from "@/contracts/types/notification.types";

View file

@ -13,8 +13,8 @@
*/ */
import { PGlite } from "@electric-sql/pglite"; import { PGlite } from "@electric-sql/pglite";
import { electricSync } from "@electric-sql/pglite-sync";
import { live } from "@electric-sql/pglite/live"; import { live } from "@electric-sql/pglite/live";
import { electricSync } from "@electric-sql/pglite-sync";
// Types // Types
export interface ElectricClient { export interface ElectricClient {
@ -270,365 +270,375 @@ export async function initElectric(userId: string): Promise<ElectricClient> {
// Create and track the sync promise to prevent race conditions // Create and track the sync promise to prevent race conditions
const syncPromise = (async (): Promise<SyncHandle> => { const syncPromise = (async (): Promise<SyncHandle> => {
// Build params for the shape request // Build params for the shape request
// Electric SQL expects params as URL query parameters // Electric SQL expects params as URL query parameters
const params: Record<string, string> = { table }; const params: Record<string, string> = { table };
// Validate and fix WHERE clause to ensure string literals are properly quoted // Validate and fix WHERE clause to ensure string literals are properly quoted
let validatedWhere = where; let validatedWhere = where;
if (where) { if (where) {
// Check if where uses positional parameters // Check if where uses positional parameters
if (where.includes("$1")) { if (where.includes("$1")) {
// Extract the value from the where clause if it's embedded // Extract the value from the where clause if it's embedded
// For now, we'll use the where clause as-is and let Electric handle it // For now, we'll use the where clause as-is and let Electric handle it
params.where = where;
validatedWhere = where;
} else {
// Validate that string literals are properly quoted
// Count single quotes - should be even (pairs) for properly quoted strings
const singleQuoteCount = (where.match(/'/g) || []).length;
if (singleQuoteCount % 2 !== 0) {
// Odd number of quotes means unterminated string literal
console.warn("Where clause has unmatched quotes, fixing:", where);
// Add closing quote at the end
validatedWhere = `${where}'`;
params.where = validatedWhere;
} else {
// Use the where clause directly (already formatted)
params.where = where; params.where = where;
validatedWhere = where; validatedWhere = where;
} else {
// Validate that string literals are properly quoted
// Count single quotes - should be even (pairs) for properly quoted strings
const singleQuoteCount = (where.match(/'/g) || []).length;
if (singleQuoteCount % 2 !== 0) {
// Odd number of quotes means unterminated string literal
console.warn("Where clause has unmatched quotes, fixing:", where);
// Add closing quote at the end
validatedWhere = `${where}'`;
params.where = validatedWhere;
} else {
// Use the where clause directly (already formatted)
params.where = where;
validatedWhere = where;
}
} }
} }
}
if (columns) params.columns = columns.join(","); if (columns) params.columns = columns.join(",");
console.log("[Electric] Syncing shape with params:", params); console.log("[Electric] Syncing shape with params:", params);
console.log("[Electric] Electric URL:", `${electricUrl}/v1/shape`); console.log("[Electric] Electric URL:", `${electricUrl}/v1/shape`);
console.log("[Electric] Where clause:", where, "Validated:", validatedWhere); console.log("[Electric] Where clause:", where, "Validated:", validatedWhere);
try {
// Debug: Test Electric SQL connection directly first
// Use validatedWhere to ensure proper URL encoding
const testUrl = `${electricUrl}/v1/shape?table=${table}&offset=-1${validatedWhere ? `&where=${encodeURIComponent(validatedWhere)}` : ""}`;
console.log("[Electric] Testing Electric SQL directly:", testUrl);
try { try {
const testResponse = await fetch(testUrl); // Debug: Test Electric SQL connection directly first
const testHeaders = { // Use validatedWhere to ensure proper URL encoding
handle: testResponse.headers.get("electric-handle"), const testUrl = `${electricUrl}/v1/shape?table=${table}&offset=-1${validatedWhere ? `&where=${encodeURIComponent(validatedWhere)}` : ""}`;
offset: testResponse.headers.get("electric-offset"), console.log("[Electric] Testing Electric SQL directly:", testUrl);
upToDate: testResponse.headers.get("electric-up-to-date"), try {
}; const testResponse = await fetch(testUrl);
console.log("[Electric] Direct Electric SQL response headers:", testHeaders); const testHeaders = {
const testData = await testResponse.json(); handle: testResponse.headers.get("electric-handle"),
console.log( offset: testResponse.headers.get("electric-offset"),
"[Electric] Direct Electric SQL data count:", upToDate: testResponse.headers.get("electric-up-to-date"),
Array.isArray(testData) ? testData.length : "not array", };
testData console.log("[Electric] Direct Electric SQL response headers:", testHeaders);
); const testData = await testResponse.json();
} catch (testErr) {
console.error("[Electric] Direct Electric SQL test failed:", testErr);
}
// Use PGlite's electric sync plugin to sync the shape
// According to Electric SQL docs, the shape config uses params for table, where, columns
// Note: mapColumns is OPTIONAL per pglite-sync types.ts
// Create a promise that resolves when initial sync is complete
// Using recommended approach: check isUpToDate immediately, watch stream, shorter timeout
// IMPORTANT: We don't unsubscribe from the stream - it must stay active for real-time updates
let syncResolved = false;
// Initialize with no-op functions to satisfy TypeScript
let resolveInitialSync: () => void = () => {};
let rejectInitialSync: (error: Error) => void = () => {};
const initialSyncPromise = new Promise<void>((resolve, reject) => {
resolveInitialSync = () => {
if (!syncResolved) {
syncResolved = true;
// DON'T unsubscribe from stream - it needs to stay active for real-time updates
resolve();
}
};
rejectInitialSync = (error: Error) => {
if (!syncResolved) {
syncResolved = true;
// DON'T unsubscribe from stream even on error - let Electric handle it
reject(error);
}
};
// Shorter timeout (5 seconds) as fallback
setTimeout(() => {
if (!syncResolved) {
console.warn(
`[Electric] ⚠️ Sync timeout for ${table} - checking isUpToDate one more time...`
);
// Check isUpToDate one more time before resolving
// This will be checked after shape is created
setTimeout(() => {
if (!syncResolved) {
console.warn(
`[Electric] ⚠️ Sync timeout for ${table} - resolving anyway after 5s`
);
resolveInitialSync();
}
}, 100);
}
}, 5000);
});
// Include userId in shapeKey for user-specific sync state
const shapeConfig = {
shape: {
url: `${electricUrl}/v1/shape`,
params: {
table,
...(validatedWhere ? { where: validatedWhere } : {}),
...(columns ? { columns: columns.join(",") } : {}),
},
},
table,
primaryKey,
shapeKey: `${userId}_v${SYNC_VERSION}_${table}_${where?.replace(/[^a-zA-Z0-9]/g, "_") || "all"}`, // User-specific versioned key
onInitialSync: () => {
console.log( console.log(
`[Electric] ✅ Initial sync complete for ${table} - data should now be in PGlite` "[Electric] Direct Electric SQL data count:",
Array.isArray(testData) ? testData.length : "not array",
testData
);
} catch (testErr) {
console.error("[Electric] Direct Electric SQL test failed:", testErr);
}
// Use PGlite's electric sync plugin to sync the shape
// According to Electric SQL docs, the shape config uses params for table, where, columns
// Note: mapColumns is OPTIONAL per pglite-sync types.ts
// Create a promise that resolves when initial sync is complete
// Using recommended approach: check isUpToDate immediately, watch stream, shorter timeout
// IMPORTANT: We don't unsubscribe from the stream - it must stay active for real-time updates
let syncResolved = false;
// Initialize with no-op functions to satisfy TypeScript
let resolveInitialSync: () => void = () => {};
let rejectInitialSync: (error: Error) => void = () => {};
const initialSyncPromise = new Promise<void>((resolve, reject) => {
resolveInitialSync = () => {
if (!syncResolved) {
syncResolved = true;
// DON'T unsubscribe from stream - it needs to stay active for real-time updates
resolve();
}
};
rejectInitialSync = (error: Error) => {
if (!syncResolved) {
syncResolved = true;
// DON'T unsubscribe from stream even on error - let Electric handle it
reject(error);
}
};
// Shorter timeout (5 seconds) as fallback
setTimeout(() => {
if (!syncResolved) {
console.warn(
`[Electric] ⚠️ Sync timeout for ${table} - checking isUpToDate one more time...`
);
// Check isUpToDate one more time before resolving
// This will be checked after shape is created
setTimeout(() => {
if (!syncResolved) {
console.warn(
`[Electric] ⚠️ Sync timeout for ${table} - resolving anyway after 5s`
);
resolveInitialSync();
}
}, 100);
}
}, 5000);
});
// Include userId in shapeKey for user-specific sync state
const shapeConfig = {
shape: {
url: `${electricUrl}/v1/shape`,
params: {
table,
...(validatedWhere ? { where: validatedWhere } : {}),
...(columns ? { columns: columns.join(",") } : {}),
},
},
table,
primaryKey,
shapeKey: `${userId}_v${SYNC_VERSION}_${table}_${where?.replace(/[^a-zA-Z0-9]/g, "_") || "all"}`, // User-specific versioned key
onInitialSync: () => {
console.log(
`[Electric] ✅ Initial sync complete for ${table} - data should now be in PGlite`
);
resolveInitialSync();
},
onError: (error: Error) => {
console.error(`[Electric] ❌ Shape sync error for ${table}:`, error);
console.error(
"[Electric] Error details:",
JSON.stringify(error, Object.getOwnPropertyNames(error))
);
rejectInitialSync(error);
},
};
console.log(
"[Electric] syncShapeToTable config:",
JSON.stringify(shapeConfig, null, 2)
);
// Type assertion to PGlite with electric extension
const pgWithElectric = db as PGlite & {
electric: {
syncShapeToTable: (
config: typeof shapeConfig
) => Promise<{ unsubscribe: () => void; isUpToDate: boolean; stream: unknown }>;
};
};
let shape: { unsubscribe: () => void; isUpToDate: boolean; stream: unknown };
try {
shape = await pgWithElectric.electric.syncShapeToTable(shapeConfig);
} catch (syncError) {
// Handle "Already syncing" error - pglite-sync might not have fully cleaned up yet
const errorMessage =
syncError instanceof Error ? syncError.message : String(syncError);
if (errorMessage.includes("Already syncing")) {
console.warn(
`[Electric] Already syncing ${table}, waiting for existing sync to settle...`
);
// Wait a short time for pglite-sync to settle
await new Promise((resolve) => setTimeout(resolve, 100));
// Check if an active handle now exists (another sync might have completed)
const existingHandle = activeSyncHandles.get(cacheKey);
if (existingHandle) {
console.log(`[Electric] Found existing handle after waiting: ${cacheKey}`);
return existingHandle;
}
// Retry once after waiting
console.log(`[Electric] Retrying sync for ${table}...`);
try {
shape = await pgWithElectric.electric.syncShapeToTable(shapeConfig);
} catch (retryError) {
const retryMessage =
retryError instanceof Error ? retryError.message : String(retryError);
if (retryMessage.includes("Already syncing")) {
// Still syncing - create a placeholder handle that indicates the table is being synced
console.warn(
`[Electric] ${table} still syncing, creating placeholder handle`
);
const placeholderHandle: SyncHandle = {
unsubscribe: () => {
console.log(`[Electric] Placeholder unsubscribe for: ${cacheKey}`);
activeSyncHandles.delete(cacheKey);
},
get isUpToDate() {
return false; // We don't know the real state
},
stream: undefined,
initialSyncPromise: Promise.resolve(), // Already syncing means data should be coming
};
activeSyncHandles.set(cacheKey, placeholderHandle);
return placeholderHandle;
}
throw retryError;
}
} else {
throw syncError;
}
}
if (!shape) {
throw new Error("syncShapeToTable returned undefined");
}
// Log the actual shape result structure
console.log("[Electric] Shape sync result (initial):", {
hasUnsubscribe: typeof shape?.unsubscribe === "function",
isUpToDate: shape?.isUpToDate,
hasStream: !!shape?.stream,
streamType: typeof shape?.stream,
});
// Recommended Approach Step 1: Check isUpToDate immediately
if (shape.isUpToDate) {
console.log(
`[Electric] ✅ Sync already up-to-date for ${table} (resuming from previous state)`
); );
resolveInitialSync(); resolveInitialSync();
},
onError: (error: Error) => {
console.error(`[Electric] ❌ Shape sync error for ${table}:`, error);
console.error(
"[Electric] Error details:",
JSON.stringify(error, Object.getOwnPropertyNames(error))
);
rejectInitialSync(error);
},
};
console.log(
"[Electric] syncShapeToTable config:",
JSON.stringify(shapeConfig, null, 2)
);
// Type assertion to PGlite with electric extension
const pgWithElectric = db as PGlite & {
electric: {
syncShapeToTable: (
config: typeof shapeConfig
) => Promise<{ unsubscribe: () => void; isUpToDate: boolean; stream: unknown }>;
};
};
let shape: { unsubscribe: () => void; isUpToDate: boolean; stream: unknown };
try {
shape = await pgWithElectric.electric.syncShapeToTable(shapeConfig);
} catch (syncError) {
// Handle "Already syncing" error - pglite-sync might not have fully cleaned up yet
const errorMessage = syncError instanceof Error ? syncError.message : String(syncError);
if (errorMessage.includes("Already syncing")) {
console.warn(`[Electric] Already syncing ${table}, waiting for existing sync to settle...`);
// Wait a short time for pglite-sync to settle
await new Promise(resolve => setTimeout(resolve, 100));
// Check if an active handle now exists (another sync might have completed)
const existingHandle = activeSyncHandles.get(cacheKey);
if (existingHandle) {
console.log(`[Electric] Found existing handle after waiting: ${cacheKey}`);
return existingHandle;
}
// Retry once after waiting
console.log(`[Electric] Retrying sync for ${table}...`);
try {
shape = await pgWithElectric.electric.syncShapeToTable(shapeConfig);
} catch (retryError) {
const retryMessage = retryError instanceof Error ? retryError.message : String(retryError);
if (retryMessage.includes("Already syncing")) {
// Still syncing - create a placeholder handle that indicates the table is being synced
console.warn(`[Electric] ${table} still syncing, creating placeholder handle`);
const placeholderHandle: SyncHandle = {
unsubscribe: () => {
console.log(`[Electric] Placeholder unsubscribe for: ${cacheKey}`);
activeSyncHandles.delete(cacheKey);
},
get isUpToDate() {
return false; // We don't know the real state
},
stream: undefined,
initialSyncPromise: Promise.resolve(), // Already syncing means data should be coming
};
activeSyncHandles.set(cacheKey, placeholderHandle);
return placeholderHandle;
}
throw retryError;
}
} else { } else {
throw syncError; // Recommended Approach Step 2: Subscribe to stream and watch for "up-to-date" message
} if (shape?.stream) {
} const stream = shape.stream as any;
console.log("[Electric] Shape stream details:", {
if (!shape) { shapeHandle: stream?.shapeHandle,
throw new Error("syncShapeToTable returned undefined"); lastOffset: stream?.lastOffset,
} isUpToDate: stream?.isUpToDate,
error: stream?.error,
// Log the actual shape result structure hasSubscribe: typeof stream?.subscribe === "function",
console.log("[Electric] Shape sync result (initial):", { hasUnsubscribe: typeof stream?.unsubscribe === "function",
hasUnsubscribe: typeof shape?.unsubscribe === "function",
isUpToDate: shape?.isUpToDate,
hasStream: !!shape?.stream,
streamType: typeof shape?.stream,
});
// Recommended Approach Step 1: Check isUpToDate immediately
if (shape.isUpToDate) {
console.log(
`[Electric] ✅ Sync already up-to-date for ${table} (resuming from previous state)`
);
resolveInitialSync();
} else {
// Recommended Approach Step 2: Subscribe to stream and watch for "up-to-date" message
if (shape?.stream) {
const stream = shape.stream as any;
console.log("[Electric] Shape stream details:", {
shapeHandle: stream?.shapeHandle,
lastOffset: stream?.lastOffset,
isUpToDate: stream?.isUpToDate,
error: stream?.error,
hasSubscribe: typeof stream?.subscribe === "function",
hasUnsubscribe: typeof stream?.unsubscribe === "function",
});
// Subscribe to the stream to watch for "up-to-date" control message
// NOTE: We keep this subscription active - don't unsubscribe!
// The stream is what Electric SQL uses for real-time updates
if (typeof stream?.subscribe === "function") {
console.log(
"[Electric] Subscribing to shape stream to watch for up-to-date message..."
);
// Subscribe but don't store unsubscribe - we want it to stay active
stream.subscribe((messages: unknown[]) => {
// Continue receiving updates even after sync is resolved
if (!syncResolved) {
console.log(
"[Electric] 🔵 Shape stream received messages:",
messages?.length || 0
);
}
// Check if any message indicates sync is complete
if (messages && messages.length > 0) {
for (const message of messages) {
const msg = message as any;
// Check for "up-to-date" control message
if (
msg?.headers?.control === "up-to-date" ||
msg?.headers?.electric_up_to_date === "true" ||
(typeof msg === "object" && "up-to-date" in msg)
) {
if (!syncResolved) {
console.log(`[Electric] ✅ Received up-to-date message for ${table}`);
resolveInitialSync();
}
// Continue listening for real-time updates - don't return!
}
}
if (!syncResolved && messages.length > 0) {
console.log(
"[Electric] First message:",
JSON.stringify(messages[0], null, 2)
);
}
}
// Also check stream's isUpToDate property after receiving messages
if (!syncResolved && stream?.isUpToDate) {
console.log(`[Electric] ✅ Stream isUpToDate is true for ${table}`);
resolveInitialSync();
}
}); });
// Also check stream's isUpToDate property immediately // Subscribe to the stream to watch for "up-to-date" control message
if (stream?.isUpToDate) { // NOTE: We keep this subscription active - don't unsubscribe!
console.log(`[Electric] ✅ Stream isUpToDate is true immediately for ${table}`); // The stream is what Electric SQL uses for real-time updates
resolveInitialSync(); if (typeof stream?.subscribe === "function") {
console.log(
"[Electric] Subscribing to shape stream to watch for up-to-date message..."
);
// Subscribe but don't store unsubscribe - we want it to stay active
stream.subscribe((messages: unknown[]) => {
// Continue receiving updates even after sync is resolved
if (!syncResolved) {
console.log(
"[Electric] 🔵 Shape stream received messages:",
messages?.length || 0
);
}
// Check if any message indicates sync is complete
if (messages && messages.length > 0) {
for (const message of messages) {
const msg = message as any;
// Check for "up-to-date" control message
if (
msg?.headers?.control === "up-to-date" ||
msg?.headers?.electric_up_to_date === "true" ||
(typeof msg === "object" && "up-to-date" in msg)
) {
if (!syncResolved) {
console.log(`[Electric] ✅ Received up-to-date message for ${table}`);
resolveInitialSync();
}
// Continue listening for real-time updates - don't return!
}
}
if (!syncResolved && messages.length > 0) {
console.log(
"[Electric] First message:",
JSON.stringify(messages[0], null, 2)
);
}
}
// Also check stream's isUpToDate property after receiving messages
if (!syncResolved && stream?.isUpToDate) {
console.log(`[Electric] ✅ Stream isUpToDate is true for ${table}`);
resolveInitialSync();
}
});
// Also check stream's isUpToDate property immediately
if (stream?.isUpToDate) {
console.log(
`[Electric] ✅ Stream isUpToDate is true immediately for ${table}`
);
resolveInitialSync();
}
} }
// Also poll isUpToDate periodically as a backup (every 200ms)
const pollInterval = setInterval(() => {
if (syncResolved) {
clearInterval(pollInterval);
return;
}
if (shape.isUpToDate || stream?.isUpToDate) {
console.log(
`[Electric] ✅ Sync completed (detected via polling) for ${table}`
);
clearInterval(pollInterval);
resolveInitialSync();
}
}, 200);
// Clean up polling when promise resolves
initialSyncPromise.finally(() => {
clearInterval(pollInterval);
});
} else {
console.warn(
`[Electric] ⚠️ No stream available for ${table}, relying on callback and timeout`
);
} }
// Also poll isUpToDate periodically as a backup (every 200ms)
const pollInterval = setInterval(() => {
if (syncResolved) {
clearInterval(pollInterval);
return;
}
if (shape.isUpToDate || stream?.isUpToDate) {
console.log(`[Electric] ✅ Sync completed (detected via polling) for ${table}`);
clearInterval(pollInterval);
resolveInitialSync();
}
}, 200);
// Clean up polling when promise resolves
initialSyncPromise.finally(() => {
clearInterval(pollInterval);
});
} else {
console.warn(
`[Electric] ⚠️ No stream available for ${table}, relying on callback and timeout`
);
} }
}
// Create the sync handle with proper cleanup // Create the sync handle with proper cleanup
const syncHandle: SyncHandle = { const syncHandle: SyncHandle = {
unsubscribe: () => { unsubscribe: () => {
console.log(`[Electric] Unsubscribing from: ${cacheKey}`); console.log(`[Electric] Unsubscribing from: ${cacheKey}`);
// Remove from cache first // Remove from cache first
activeSyncHandles.delete(cacheKey); activeSyncHandles.delete(cacheKey);
// Then unsubscribe from the shape // Then unsubscribe from the shape
if (shape && typeof shape.unsubscribe === "function") { if (shape && typeof shape.unsubscribe === "function") {
shape.unsubscribe(); shape.unsubscribe();
} }
}, },
// Use getter to always return current state // Use getter to always return current state
get isUpToDate() { get isUpToDate() {
return shape?.isUpToDate ?? false; return shape?.isUpToDate ?? false;
}, },
stream: shape?.stream, stream: shape?.stream,
initialSyncPromise, // Expose promise so callers can wait for sync initialSyncPromise, // Expose promise so callers can wait for sync
}; };
// Cache the sync handle for reuse (memory optimization) // Cache the sync handle for reuse (memory optimization)
activeSyncHandles.set(cacheKey, syncHandle); activeSyncHandles.set(cacheKey, syncHandle);
console.log(
`[Electric] Cached sync handle for: ${cacheKey} (total cached: ${activeSyncHandles.size})`
);
return syncHandle;
} catch (error) {
console.error("[Electric] Failed to sync shape:", error);
// Check if Electric SQL server is reachable
try {
const response = await fetch(`${electricUrl}/v1/shape?table=${table}&offset=-1`, {
method: "GET",
});
console.log( console.log(
"[Electric] Electric SQL server response:", `[Electric] Cached sync handle for: ${cacheKey} (total cached: ${activeSyncHandles.size})`
response.status,
response.statusText
); );
if (!response.ok) {
console.error("[Electric] Electric SQL server error:", await response.text()); return syncHandle;
} catch (error) {
console.error("[Electric] Failed to sync shape:", error);
// Check if Electric SQL server is reachable
try {
const response = await fetch(`${electricUrl}/v1/shape?table=${table}&offset=-1`, {
method: "GET",
});
console.log(
"[Electric] Electric SQL server response:",
response.status,
response.statusText
);
if (!response.ok) {
console.error("[Electric] Electric SQL server error:", await response.text());
}
} catch (fetchError) {
console.error("[Electric] Cannot reach Electric SQL server:", fetchError);
console.error("[Electric] Make sure Electric SQL is running at:", electricUrl);
} }
} catch (fetchError) { throw error;
console.error("[Electric] Cannot reach Electric SQL server:", fetchError);
console.error("[Electric] Make sure Electric SQL is running at:", electricUrl);
} }
throw error;
}
})(); })();
// Track the sync promise to prevent concurrent syncs for the same shape // Track the sync promise to prevent concurrent syncs for the same shape

View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24"><path fill="#000" d="M13.85 0a4.16 4.16 0 0 0-2.95 1.217L1.456 10.66a.835.835 0 0 0 0 1.18a.835.835 0 0 0 1.18 0l9.442-9.442a2.49 2.49 0 0 1 3.541 0a2.49 2.49 0 0 1 0 3.541L8.59 12.97l-.1.1a.835.835 0 0 0 0 1.18a.835.835 0 0 0 1.18 0l.1-.098l7.03-7.034a2.49 2.49 0 0 1 3.542 0l.049.05a2.49 2.49 0 0 1 0 3.54l-8.54 8.54a1.96 1.96 0 0 0 0 2.755l1.753 1.753a.835.835 0 0 0 1.18 0a.835.835 0 0 0 0-1.18l-1.753-1.753a.266.266 0 0 1 0-.394l8.54-8.54a4.185 4.185 0 0 0 0-5.9l-.05-.05a4.16 4.16 0 0 0-2.95-1.218c-.2 0-.401.02-.6.048a4.17 4.17 0 0 0-1.17-3.552A4.16 4.16 0 0 0 13.85 0m0 3.333a.84.84 0 0 0-.59.245L6.275 10.56a4.186 4.186 0 0 0 0 5.902a4.186 4.186 0 0 0 5.902 0L19.16 9.48a.835.835 0 0 0 0-1.18a.835.835 0 0 0-1.18 0l-6.985 6.984a2.49 2.49 0 0 1-3.54 0a2.49 2.49 0 0 1 0-3.54l6.983-6.985a.835.835 0 0 0 0-1.18a.84.84 0 0 0-.59-.245"/></svg>

After

Width:  |  Height:  |  Size: 930 B