mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-04-29 10:56:24 +02:00
feat: support multiple transport types for MCP server connections, including stdio and HTTP
This commit is contained in:
parent
bb5cb846b3
commit
9625a24475
9 changed files with 435 additions and 191 deletions
|
|
@ -116,47 +116,6 @@ You have access to the following tools:
|
||||||
* This makes your response more visual and engaging.
|
* This makes your response more visual and engaging.
|
||||||
* Prioritize showing: diagrams, charts, infographics, key illustrations, or images that help explain the content.
|
* Prioritize showing: diagrams, charts, infographics, key illustrations, or images that help explain the content.
|
||||||
* Don't show every image - just the most relevant 1-3 images that enhance understanding.
|
* Don't show every image - just the most relevant 1-3 images that enhance understanding.
|
||||||
|
|
||||||
6. write_todos: Create and update a planning/todo list.
|
|
||||||
- Args:
|
|
||||||
- todos: List of todo items, each with:
|
|
||||||
* content: Description of the task (required)
|
|
||||||
* status: "pending", "in_progress", "completed", or "cancelled" (required)
|
|
||||||
|
|
||||||
STRICT MODE SELECTION - CHOOSE ONE:
|
|
||||||
|
|
||||||
[MODE A] AGENT PLAN (you will work through it)
|
|
||||||
Use when: User asks you to explain, teach, plan, or break down a concept.
|
|
||||||
Examples: "Explain how to set up Python", "Plan my trip", "Break down machine learning"
|
|
||||||
Rules:
|
|
||||||
- Create plan with first item "in_progress", rest "pending"
|
|
||||||
- After explaining each step, call write_todos again to update progress
|
|
||||||
- Only ONE item "in_progress" at a time
|
|
||||||
- Mark items "completed" as you finish explaining them
|
|
||||||
- Final call: all items "completed"
|
|
||||||
|
|
||||||
[MODE B] EXTERNAL TASK DISPLAY (from connectors - you CANNOT complete these)
|
|
||||||
Use when: User asks to show/list/display tasks from Linear, Jira, ClickUp, GitHub, Airtable, Notion, or any connector.
|
|
||||||
Examples: "Show my Linear tasks", "List Jira tickets", "Create todos from ClickUp", "Show GitHub issues"
|
|
||||||
STRICT RULES:
|
|
||||||
1. You CANNOT complete these tasks - only the user can in the actual tool
|
|
||||||
2. PRESERVE original status from source - DO NOT use agent workflow
|
|
||||||
3. Call write_todos ONCE with all tasks and their REAL statuses
|
|
||||||
4. Provide insights/summary as TEXT after the todo list, NOT as todo items
|
|
||||||
5. NO INTERNAL REASONING - Never expose your process. Do NOT say "Let me map...", "Converting statuses...", "Here's how I'll organize...", or explain mapping logic. Just call write_todos silently and provide insights.
|
|
||||||
|
|
||||||
STATUS MAPPING (apply strictly):
|
|
||||||
- "completed" ← Done, Completed, Complete, Closed, Resolved, Fixed, Merged, Shipped, Released
|
|
||||||
- "in_progress" ← In Progress, In Review, Testing, QA, Active, Doing, Started, Review, Working
|
|
||||||
- "pending" ← Todo, To Do, Backlog, Open, New, Pending, Triage, Reopened, Unstarted
|
|
||||||
- "cancelled" ← Cancelled, Canceled, Won't Fix, Duplicate, Invalid, Rejected, Archived, Obsolete
|
|
||||||
|
|
||||||
CONNECTOR-SPECIFIC:
|
|
||||||
- Linear: state.name = "Done", "In Progress", "Todo", "Backlog", "Cancelled"
|
|
||||||
- Jira: statusCategory.name = "To Do", "In Progress", "Done"
|
|
||||||
- ClickUp: status = "complete", "in progress", "open", "closed"
|
|
||||||
- GitHub: state = "open", "closed"; PRs also "merged"
|
|
||||||
- Airtable/Notion: Check field values, apply mapping above
|
|
||||||
</tools>
|
</tools>
|
||||||
<tool_call_examples>
|
<tool_call_examples>
|
||||||
- User: "How do I install SurfSense?"
|
- User: "How do I install SurfSense?"
|
||||||
|
|
@ -226,70 +185,6 @@ You have access to the following tools:
|
||||||
- Then, if the content contains useful diagrams/images like ``:
|
- Then, if the content contains useful diagrams/images like ``:
|
||||||
- Call: `display_image(src="https://example.com/nn-diagram.png", alt="Neural Network Diagram", title="Neural Network Architecture")`
|
- Call: `display_image(src="https://example.com/nn-diagram.png", alt="Neural Network Diagram", title="Neural Network Architecture")`
|
||||||
- Then provide your explanation, referencing the displayed image
|
- Then provide your explanation, referencing the displayed image
|
||||||
|
|
||||||
[MODE A EXAMPLES] Agent Plan - you work through it:
|
|
||||||
|
|
||||||
- User: "Create a plan for building a user authentication system"
|
|
||||||
- Call: `write_todos(todos=[{"content": "Design database schema for users and sessions", "status": "in_progress"}, {"content": "Implement registration and login endpoints", "status": "pending"}, {"content": "Add password reset functionality", "status": "pending"}])`
|
|
||||||
- Then explain each step in detail as you work through them
|
|
||||||
|
|
||||||
- User: "Break down how to build a REST API into steps"
|
|
||||||
- Call: `write_todos(todos=[{"content": "Design API endpoints and data models", "status": "in_progress"}, {"content": "Set up server framework and routing", "status": "pending"}, {"content": "Implement CRUD operations", "status": "pending"}, {"content": "Add authentication and error handling", "status": "pending"}])`
|
|
||||||
- Then provide detailed explanations for each step
|
|
||||||
|
|
||||||
- User: "Help me plan my trip to Japan"
|
|
||||||
- Call: `write_todos(todos=[{"content": "Research best time to visit and book flights", "status": "in_progress"}, {"content": "Plan itinerary for cities to visit", "status": "pending"}, {"content": "Book accommodations", "status": "pending"}, {"content": "Prepare travel documents and currency", "status": "pending"}])`
|
|
||||||
- Then provide travel preparation guidance
|
|
||||||
|
|
||||||
- COMPLETE WORKFLOW EXAMPLE - User: "Explain how to set up a Python project"
|
|
||||||
- STEP 1 (Create initial plan):
|
|
||||||
Call: `write_todos(todos=[{"content": "Set up virtual environment", "status": "in_progress"}, {"content": "Create project structure", "status": "pending"}, {"content": "Configure dependencies", "status": "pending"}])`
|
|
||||||
Then explain virtual environment setup in detail...
|
|
||||||
- STEP 2 (After explaining virtual environments, update progress):
|
|
||||||
Call: `write_todos(todos=[{"content": "Set up virtual environment", "status": "completed"}, {"content": "Create project structure", "status": "in_progress"}, {"content": "Configure dependencies", "status": "pending"}])`
|
|
||||||
Then explain project structure in detail...
|
|
||||||
- STEP 3 (After explaining project structure, update progress):
|
|
||||||
Call: `write_todos(todos=[{"content": "Set up virtual environment", "status": "completed"}, {"content": "Create project structure", "status": "completed"}, {"content": "Configure dependencies", "status": "in_progress"}])`
|
|
||||||
Then explain dependency configuration in detail...
|
|
||||||
- STEP 4 (After completing all explanations, mark all done):
|
|
||||||
Call: `write_todos(todos=[{"content": "Set up virtual environment", "status": "completed"}, {"content": "Create project structure", "status": "completed"}, {"content": "Configure dependencies", "status": "completed"}])`
|
|
||||||
Provide final summary
|
|
||||||
|
|
||||||
[MODE B EXAMPLES] External Tasks - preserve original status, you CANNOT complete:
|
|
||||||
|
|
||||||
- User: "Show my Linear tasks" or "Create todos for Linear tasks"
|
|
||||||
- First search: `search_knowledge_base(query="Linear tasks issues", connectors_to_search=["LINEAR_CONNECTOR"])`
|
|
||||||
- Then call write_todos ONCE with ORIGINAL statuses preserved:
|
|
||||||
Call: `write_todos(todos=[
|
|
||||||
{"content": "SUR-21: Add refresh button in manage documents page", "status": "completed"},
|
|
||||||
{"content": "SUR-22: Logs page not accessible in docker", "status": "completed"},
|
|
||||||
{"content": "SUR-27: Add Google Drive connector", "status": "in_progress"},
|
|
||||||
{"content": "SUR-28: Logs page should show all logs", "status": "pending"}
|
|
||||||
])`
|
|
||||||
- Then provide INSIGHTS as text (NOT as todos):
|
|
||||||
"You have 2 completed, 1 in progress, and 1 pending task. SUR-27 (Google Drive connector) is currently active. Consider prioritizing SUR-28 next."
|
|
||||||
|
|
||||||
- User: "List my Jira tickets"
|
|
||||||
- First search: `search_knowledge_base(query="Jira tickets issues", connectors_to_search=["JIRA_CONNECTOR"])`
|
|
||||||
- Map Jira statuses: "Done" → completed, "In Progress"/"In Review" → in_progress, "To Do" → pending
|
|
||||||
- Call write_todos ONCE with mapped statuses
|
|
||||||
- Provide summary as text after
|
|
||||||
|
|
||||||
- User: "Show ClickUp tasks"
|
|
||||||
- First search: `search_knowledge_base(query="ClickUp tasks", connectors_to_search=["CLICKUP_CONNECTOR"])`
|
|
||||||
- Map: "complete"/"closed" → completed, "in progress" → in_progress, "open" → pending
|
|
||||||
- Call write_todos ONCE, then provide insights as text
|
|
||||||
|
|
||||||
- User: "Show my GitHub issues"
|
|
||||||
- First search: `search_knowledge_base(query="GitHub issues", connectors_to_search=["GITHUB_CONNECTOR"])`
|
|
||||||
- Map: "closed"/"merged" → completed, "open" → pending
|
|
||||||
- Call write_todos ONCE, then summarize as text
|
|
||||||
|
|
||||||
CRITICAL FOR MODE B:
|
|
||||||
- NEVER use the "first item in_progress, rest pending" pattern for external tasks
|
|
||||||
- NEVER pretend you will complete external tasks - be honest that only the user can
|
|
||||||
- ALWAYS preserve the actual status from the source system
|
|
||||||
- ALWAYS provide insights/summaries as regular text, not as todo items
|
|
||||||
</tool_call_examples>
|
</tool_call_examples>
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
"""MCP Client Wrapper.
|
"""MCP Client Wrapper.
|
||||||
|
|
||||||
This module provides a client for communicating with MCP servers via stdio transport.
|
This module provides a client for communicating with MCP servers via stdio and HTTP transports.
|
||||||
It handles server lifecycle management, tool discovery, and tool execution.
|
It handles server lifecycle management, tool discovery, and tool execution.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
@ -12,6 +12,7 @@ from typing import Any
|
||||||
|
|
||||||
from mcp import ClientSession
|
from mcp import ClientSession
|
||||||
from mcp.client.stdio import StdioServerParameters, stdio_client
|
from mcp.client.stdio import StdioServerParameters, stdio_client
|
||||||
|
from mcp.client.streamable_http import streamablehttp_client
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
@ -222,7 +223,7 @@ class MCPClient:
|
||||||
async def test_mcp_connection(
|
async def test_mcp_connection(
|
||||||
command: str, args: list[str], env: dict[str, str] | None = None
|
command: str, args: list[str], env: dict[str, str] | None = None
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
"""Test connection to an MCP server and fetch available tools.
|
"""Test connection to an MCP server via stdio and fetch available tools.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
command: Command to spawn the MCP server
|
command: Command to spawn the MCP server
|
||||||
|
|
@ -249,3 +250,51 @@ async def test_mcp_connection(
|
||||||
"message": f"Failed to connect: {e!s}",
|
"message": f"Failed to connect: {e!s}",
|
||||||
"tools": [],
|
"tools": [],
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
async def test_mcp_http_connection(
|
||||||
|
url: str, headers: dict[str, str] | None = None, transport: str = "streamable-http"
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Test connection to an MCP server via HTTP and fetch available tools.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
url: URL of the MCP server
|
||||||
|
headers: Optional HTTP headers for authentication
|
||||||
|
transport: Transport type ("streamable-http", "http", or "sse")
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dict with connection status and available tools
|
||||||
|
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
logger.info("Testing HTTP MCP connection to: %s (transport: %s)", url, transport)
|
||||||
|
|
||||||
|
# Use streamable HTTP client for all HTTP-based transports
|
||||||
|
async with streamablehttp_client(url, headers=headers or {}) as (read, write, _):
|
||||||
|
async with ClientSession(read, write) as session:
|
||||||
|
await session.initialize()
|
||||||
|
|
||||||
|
# List available tools
|
||||||
|
response = await session.list_tools()
|
||||||
|
tools = []
|
||||||
|
for tool in response.tools:
|
||||||
|
tools.append({
|
||||||
|
"name": tool.name,
|
||||||
|
"description": tool.description or "",
|
||||||
|
"input_schema": tool.inputSchema if hasattr(tool, "inputSchema") else {},
|
||||||
|
})
|
||||||
|
|
||||||
|
logger.info("HTTP MCP connection successful. Found %d tools.", len(tools))
|
||||||
|
return {
|
||||||
|
"status": "success",
|
||||||
|
"message": f"Connected successfully. Found {len(tools)} tools.",
|
||||||
|
"tools": tools,
|
||||||
|
}
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error("Failed to connect to HTTP MCP server: %s", e, exc_info=True)
|
||||||
|
return {
|
||||||
|
"status": "error",
|
||||||
|
"message": f"Failed to connect: {e!s}",
|
||||||
|
"tools": [],
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -3,6 +3,10 @@
|
||||||
This module creates LangChain tools from MCP servers using the Model Context Protocol.
|
This module creates LangChain tools from MCP servers using the Model Context Protocol.
|
||||||
Tools are dynamically discovered from MCP servers - no manual configuration needed.
|
Tools are dynamically discovered from MCP servers - no manual configuration needed.
|
||||||
|
|
||||||
|
Supports both transport types:
|
||||||
|
- stdio: Local process-based MCP servers (command, args, env)
|
||||||
|
- streamable-http/http/sse: Remote HTTP-based MCP servers (url, headers)
|
||||||
|
|
||||||
This implements real MCP protocol support similar to Cursor's implementation.
|
This implements real MCP protocol support similar to Cursor's implementation.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
@ -10,6 +14,8 @@ import logging
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from langchain_core.tools import StructuredTool
|
from langchain_core.tools import StructuredTool
|
||||||
|
from mcp import ClientSession
|
||||||
|
from mcp.client.streamable_http import streamablehttp_client
|
||||||
from pydantic import BaseModel, create_model
|
from pydantic import BaseModel, create_model
|
||||||
from sqlalchemy import select
|
from sqlalchemy import select
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
@ -65,11 +71,11 @@ def _create_dynamic_input_model_from_schema(
|
||||||
return create_model(model_name, **field_definitions)
|
return create_model(model_name, **field_definitions)
|
||||||
|
|
||||||
|
|
||||||
async def _create_mcp_tool_from_definition(
|
async def _create_mcp_tool_from_definition_stdio(
|
||||||
tool_def: dict[str, Any],
|
tool_def: dict[str, Any],
|
||||||
mcp_client: MCPClient,
|
mcp_client: MCPClient,
|
||||||
) -> StructuredTool:
|
) -> StructuredTool:
|
||||||
"""Create a LangChain tool from an MCP tool definition.
|
"""Create a LangChain tool from an MCP tool definition (stdio transport).
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
tool_def: Tool definition from MCP server with name, description, input_schema
|
tool_def: Tool definition from MCP server with name, description, input_schema
|
||||||
|
|
@ -116,13 +122,223 @@ async def _create_mcp_tool_from_definition(
|
||||||
coroutine=mcp_tool_call,
|
coroutine=mcp_tool_call,
|
||||||
args_schema=input_model,
|
args_schema=input_model,
|
||||||
# Store the original MCP schema as metadata so we can access it later
|
# Store the original MCP schema as metadata so we can access it later
|
||||||
metadata={"mcp_input_schema": input_schema},
|
metadata={"mcp_input_schema": input_schema, "mcp_transport": "stdio"},
|
||||||
)
|
)
|
||||||
|
|
||||||
logger.info(f"Created MCP tool: '{tool_name}'")
|
logger.info(f"Created MCP tool (stdio): '{tool_name}'")
|
||||||
return tool
|
return tool
|
||||||
|
|
||||||
|
|
||||||
|
async def _create_mcp_tool_from_definition_http(
|
||||||
|
tool_def: dict[str, Any],
|
||||||
|
url: str,
|
||||||
|
headers: dict[str, str],
|
||||||
|
) -> StructuredTool:
|
||||||
|
"""Create a LangChain tool from an MCP tool definition (HTTP transport).
|
||||||
|
|
||||||
|
Args:
|
||||||
|
tool_def: Tool definition from MCP server with name, description, input_schema
|
||||||
|
url: URL of the MCP server
|
||||||
|
headers: HTTP headers for authentication
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
LangChain StructuredTool instance
|
||||||
|
|
||||||
|
"""
|
||||||
|
tool_name = tool_def.get("name", "unnamed_tool")
|
||||||
|
tool_description = tool_def.get("description", "No description provided")
|
||||||
|
input_schema = tool_def.get("input_schema", {"type": "object", "properties": {}})
|
||||||
|
|
||||||
|
# Log the actual schema for debugging
|
||||||
|
logger.info(f"MCP HTTP tool '{tool_name}' input schema: {input_schema}")
|
||||||
|
|
||||||
|
# Create dynamic input model from schema
|
||||||
|
input_model = _create_dynamic_input_model_from_schema(tool_name, input_schema)
|
||||||
|
|
||||||
|
async def mcp_http_tool_call(**kwargs) -> str:
|
||||||
|
"""Execute the MCP tool call via HTTP transport."""
|
||||||
|
logger.info(f"MCP HTTP tool '{tool_name}' called with params: {kwargs}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
async with streamablehttp_client(url, headers=headers) as (read, write, _):
|
||||||
|
async with ClientSession(read, write) as session:
|
||||||
|
await session.initialize()
|
||||||
|
|
||||||
|
# Call the tool
|
||||||
|
response = await session.call_tool(tool_name, arguments=kwargs)
|
||||||
|
|
||||||
|
# Extract content from response
|
||||||
|
result = []
|
||||||
|
for content in response.content:
|
||||||
|
if hasattr(content, "text"):
|
||||||
|
result.append(content.text)
|
||||||
|
elif hasattr(content, "data"):
|
||||||
|
result.append(str(content.data))
|
||||||
|
else:
|
||||||
|
result.append(str(content))
|
||||||
|
|
||||||
|
result_str = "\n".join(result) if result else ""
|
||||||
|
logger.info(f"MCP HTTP tool '{tool_name}' succeeded: {result_str[:200]}")
|
||||||
|
return result_str
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
error_msg = f"MCP HTTP tool '{tool_name}' execution failed: {e!s}"
|
||||||
|
logger.exception(error_msg)
|
||||||
|
return f"Error: {error_msg}"
|
||||||
|
|
||||||
|
# Create StructuredTool
|
||||||
|
tool = StructuredTool(
|
||||||
|
name=tool_name,
|
||||||
|
description=tool_description,
|
||||||
|
coroutine=mcp_http_tool_call,
|
||||||
|
args_schema=input_model,
|
||||||
|
metadata={"mcp_input_schema": input_schema, "mcp_transport": "http", "mcp_url": url},
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"Created MCP tool (HTTP): '{tool_name}'")
|
||||||
|
return tool
|
||||||
|
|
||||||
|
|
||||||
|
async def _load_stdio_mcp_tools(
|
||||||
|
connector_id: int,
|
||||||
|
connector_name: str,
|
||||||
|
server_config: dict[str, Any],
|
||||||
|
) -> list[StructuredTool]:
|
||||||
|
"""Load tools from a stdio-based MCP server.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
connector_id: Connector ID for logging
|
||||||
|
connector_name: Connector name for logging
|
||||||
|
server_config: Server configuration with command, args, env
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of tools from the MCP server
|
||||||
|
"""
|
||||||
|
tools: list[StructuredTool] = []
|
||||||
|
|
||||||
|
# Validate required command field
|
||||||
|
command = server_config.get("command")
|
||||||
|
if not command or not isinstance(command, str):
|
||||||
|
logger.warning(
|
||||||
|
f"MCP connector {connector_id} (name: '{connector_name}') missing or invalid command field, skipping"
|
||||||
|
)
|
||||||
|
return tools
|
||||||
|
|
||||||
|
# Validate args field (must be list if present)
|
||||||
|
args = server_config.get("args", [])
|
||||||
|
if not isinstance(args, list):
|
||||||
|
logger.warning(
|
||||||
|
f"MCP connector {connector_id} (name: '{connector_name}') has invalid args field (must be list), skipping"
|
||||||
|
)
|
||||||
|
return tools
|
||||||
|
|
||||||
|
# Validate env field (must be dict if present)
|
||||||
|
env = server_config.get("env", {})
|
||||||
|
if not isinstance(env, dict):
|
||||||
|
logger.warning(
|
||||||
|
f"MCP connector {connector_id} (name: '{connector_name}') has invalid env field (must be dict), skipping"
|
||||||
|
)
|
||||||
|
return tools
|
||||||
|
|
||||||
|
# Create MCP client
|
||||||
|
mcp_client = MCPClient(command, args, env)
|
||||||
|
|
||||||
|
# Connect and discover tools
|
||||||
|
async with mcp_client.connect():
|
||||||
|
tool_definitions = await mcp_client.list_tools()
|
||||||
|
|
||||||
|
logger.info(
|
||||||
|
f"Discovered {len(tool_definitions)} tools from stdio MCP server "
|
||||||
|
f"'{command}' (connector {connector_id})"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create LangChain tools from definitions
|
||||||
|
for tool_def in tool_definitions:
|
||||||
|
try:
|
||||||
|
tool = await _create_mcp_tool_from_definition_stdio(tool_def, mcp_client)
|
||||||
|
tools.append(tool)
|
||||||
|
except Exception as e:
|
||||||
|
logger.exception(
|
||||||
|
f"Failed to create tool '{tool_def.get('name')}' "
|
||||||
|
f"from connector {connector_id}: {e!s}"
|
||||||
|
)
|
||||||
|
|
||||||
|
return tools
|
||||||
|
|
||||||
|
|
||||||
|
async def _load_http_mcp_tools(
|
||||||
|
connector_id: int,
|
||||||
|
connector_name: str,
|
||||||
|
server_config: dict[str, Any],
|
||||||
|
) -> list[StructuredTool]:
|
||||||
|
"""Load tools from an HTTP-based MCP server.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
connector_id: Connector ID for logging
|
||||||
|
connector_name: Connector name for logging
|
||||||
|
server_config: Server configuration with url, headers
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of tools from the MCP server
|
||||||
|
"""
|
||||||
|
tools: list[StructuredTool] = []
|
||||||
|
|
||||||
|
# Validate required url field
|
||||||
|
url = server_config.get("url")
|
||||||
|
if not url or not isinstance(url, str):
|
||||||
|
logger.warning(
|
||||||
|
f"MCP connector {connector_id} (name: '{connector_name}') missing or invalid url field, skipping"
|
||||||
|
)
|
||||||
|
return tools
|
||||||
|
|
||||||
|
# Validate headers field (must be dict if present)
|
||||||
|
headers = server_config.get("headers", {})
|
||||||
|
if not isinstance(headers, dict):
|
||||||
|
logger.warning(
|
||||||
|
f"MCP connector {connector_id} (name: '{connector_name}') has invalid headers field (must be dict), skipping"
|
||||||
|
)
|
||||||
|
return tools
|
||||||
|
|
||||||
|
# Connect and discover tools via HTTP
|
||||||
|
try:
|
||||||
|
async with streamablehttp_client(url, headers=headers) as (read, write, _):
|
||||||
|
async with ClientSession(read, write) as session:
|
||||||
|
await session.initialize()
|
||||||
|
|
||||||
|
# List available tools
|
||||||
|
response = await session.list_tools()
|
||||||
|
tool_definitions = []
|
||||||
|
for tool in response.tools:
|
||||||
|
tool_definitions.append({
|
||||||
|
"name": tool.name,
|
||||||
|
"description": tool.description or "",
|
||||||
|
"input_schema": tool.inputSchema if hasattr(tool, "inputSchema") else {},
|
||||||
|
})
|
||||||
|
|
||||||
|
logger.info(
|
||||||
|
f"Discovered {len(tool_definitions)} tools from HTTP MCP server "
|
||||||
|
f"'{url}' (connector {connector_id})"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create LangChain tools from definitions
|
||||||
|
for tool_def in tool_definitions:
|
||||||
|
try:
|
||||||
|
tool = await _create_mcp_tool_from_definition_http(tool_def, url, headers)
|
||||||
|
tools.append(tool)
|
||||||
|
except Exception as e:
|
||||||
|
logger.exception(
|
||||||
|
f"Failed to create HTTP tool '{tool_def.get('name')}' "
|
||||||
|
f"from connector {connector_id}: {e!s}"
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.exception(
|
||||||
|
f"Failed to connect to HTTP MCP server at '{url}' (connector {connector_id}): {e!s}"
|
||||||
|
)
|
||||||
|
|
||||||
|
return tools
|
||||||
|
|
||||||
|
|
||||||
async def load_mcp_tools(
|
async def load_mcp_tools(
|
||||||
session: AsyncSession,
|
session: AsyncSession,
|
||||||
search_space_id: int,
|
search_space_id: int,
|
||||||
|
|
@ -130,6 +346,7 @@ async def load_mcp_tools(
|
||||||
"""Load all MCP tools from user's active MCP server connectors.
|
"""Load all MCP tools from user's active MCP server connectors.
|
||||||
|
|
||||||
This discovers tools dynamically from MCP servers using the protocol.
|
This discovers tools dynamically from MCP servers using the protocol.
|
||||||
|
Supports both stdio (local process) and HTTP (remote server) transports.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
session: Database session
|
session: Database session
|
||||||
|
|
@ -163,54 +380,22 @@ async def load_mcp_tools(
|
||||||
)
|
)
|
||||||
continue
|
continue
|
||||||
|
|
||||||
# Validate required command field
|
# Determine transport type
|
||||||
command = server_config.get("command")
|
transport = server_config.get("transport", "stdio")
|
||||||
if not command or not isinstance(command, str):
|
|
||||||
logger.warning(
|
if transport in ("streamable-http", "http", "sse"):
|
||||||
f"MCP connector {connector.id} (name: '{connector.name}') missing or invalid command field, skipping"
|
# HTTP-based MCP server
|
||||||
|
connector_tools = await _load_http_mcp_tools(
|
||||||
|
connector.id, connector.name, server_config
|
||||||
)
|
)
|
||||||
continue
|
else:
|
||||||
|
# stdio-based MCP server (default)
|
||||||
# Validate args field (must be list if present)
|
connector_tools = await _load_stdio_mcp_tools(
|
||||||
args = server_config.get("args", [])
|
connector.id, connector.name, server_config
|
||||||
if not isinstance(args, list):
|
|
||||||
logger.warning(
|
|
||||||
f"MCP connector {connector.id} (name: '{connector.name}') has invalid args field (must be list), skipping"
|
|
||||||
)
|
)
|
||||||
continue
|
|
||||||
|
tools.extend(connector_tools)
|
||||||
# Validate env field (must be dict if present)
|
|
||||||
env = server_config.get("env", {})
|
|
||||||
if not isinstance(env, dict):
|
|
||||||
logger.warning(
|
|
||||||
f"MCP connector {connector.id} (name: '{connector.name}') has invalid env field (must be dict), skipping"
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
|
|
||||||
# Create MCP client
|
|
||||||
mcp_client = MCPClient(command, args, env)
|
|
||||||
|
|
||||||
# Connect and discover tools
|
|
||||||
async with mcp_client.connect():
|
|
||||||
tool_definitions = await mcp_client.list_tools()
|
|
||||||
|
|
||||||
logger.info(
|
|
||||||
f"Discovered {len(tool_definitions)} tools from MCP server "
|
|
||||||
f"'{command}' (connector {connector.id})"
|
|
||||||
)
|
|
||||||
|
|
||||||
# Create LangChain tools from definitions
|
|
||||||
for tool_def in tool_definitions:
|
|
||||||
try:
|
|
||||||
tool = await _create_mcp_tool_from_definition(
|
|
||||||
tool_def, mcp_client
|
|
||||||
)
|
|
||||||
tools.append(tool)
|
|
||||||
except Exception as e:
|
|
||||||
logger.exception(
|
|
||||||
f"Failed to create tool '{tool_def.get('name')}' "
|
|
||||||
f"from connector {connector.id}: {e!s}"
|
|
||||||
)
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.exception(
|
logger.exception(
|
||||||
f"Failed to load tools from MCP connector {connector.id}: {e!s}"
|
f"Failed to load tools from MCP connector {connector.id}: {e!s}"
|
||||||
|
|
|
||||||
|
|
@ -2385,22 +2385,43 @@ async def test_mcp_server_connection(
|
||||||
This endpoint allows users to test their MCP server configuration
|
This endpoint allows users to test their MCP server configuration
|
||||||
before saving it, similar to Cursor's flow.
|
before saving it, similar to Cursor's flow.
|
||||||
|
|
||||||
|
Supports two transport types:
|
||||||
|
- stdio: Local process with command, args, env
|
||||||
|
- streamable-http/http/sse: Remote HTTP server with url, headers
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
server_config: Server configuration with command, args, env
|
server_config: Server configuration
|
||||||
user: Current authenticated user
|
user: Current authenticated user
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Connection status and list of available tools
|
Connection status and list of available tools
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
from app.agents.new_chat.tools.mcp_client import test_mcp_connection
|
from app.agents.new_chat.tools.mcp_client import (
|
||||||
|
test_mcp_connection,
|
||||||
|
test_mcp_http_connection,
|
||||||
|
)
|
||||||
|
|
||||||
|
transport = server_config.get("transport", "stdio")
|
||||||
|
|
||||||
|
# HTTP transport (streamable-http, http, sse)
|
||||||
|
if transport in ("streamable-http", "http", "sse"):
|
||||||
|
url = server_config.get("url")
|
||||||
|
headers = server_config.get("headers", {})
|
||||||
|
|
||||||
|
if not url:
|
||||||
|
raise HTTPException(status_code=400, detail="Server URL is required for HTTP transport")
|
||||||
|
|
||||||
|
result = await test_mcp_http_connection(url, headers, transport)
|
||||||
|
return result
|
||||||
|
|
||||||
|
# stdio transport (default)
|
||||||
command = server_config.get("command")
|
command = server_config.get("command")
|
||||||
args = server_config.get("args", [])
|
args = server_config.get("args", [])
|
||||||
env = server_config.get("env", {})
|
env = server_config.get("env", {})
|
||||||
|
|
||||||
if not command:
|
if not command:
|
||||||
raise HTTPException(status_code=400, detail="Server command is required")
|
raise HTTPException(status_code=400, detail="Server command is required for stdio transport")
|
||||||
|
|
||||||
# Test the connection
|
# Test the connection
|
||||||
result = await test_mcp_connection(command, args, env)
|
result = await test_mcp_connection(command, args, env)
|
||||||
|
|
|
||||||
|
|
@ -83,12 +83,27 @@ class SearchSourceConnectorRead(SearchSourceConnectorBase, IDModel, TimestampMod
|
||||||
|
|
||||||
|
|
||||||
class MCPServerConfig(BaseModel):
|
class MCPServerConfig(BaseModel):
|
||||||
"""Configuration for an MCP server connection (similar to Cursor's config)."""
|
"""Configuration for an MCP server connection.
|
||||||
|
|
||||||
|
Supports two transport types:
|
||||||
|
- stdio: Local process (command, args, env)
|
||||||
|
- streamable-http/http/sse: Remote HTTP server (url, headers)
|
||||||
|
"""
|
||||||
|
|
||||||
command: str # e.g., "uvx", "node", "python"
|
# stdio transport fields
|
||||||
|
command: str | None = None # e.g., "uvx", "node", "python"
|
||||||
args: list[str] = [] # e.g., ["mcp-server-git", "--repository", "/path"]
|
args: list[str] = [] # e.g., ["mcp-server-git", "--repository", "/path"]
|
||||||
env: dict[str, str] = {} # Environment variables for the server process
|
env: dict[str, str] = {} # Environment variables for the server process
|
||||||
transport: str = "stdio" # "stdio" | "sse" | "http" (stdio is most common)
|
|
||||||
|
# HTTP transport fields
|
||||||
|
url: str | None = None # e.g., "https://mcp-server.com/mcp"
|
||||||
|
headers: dict[str, str] = {} # HTTP headers for authentication
|
||||||
|
|
||||||
|
transport: str = "stdio" # "stdio" | "streamable-http" | "http" | "sse"
|
||||||
|
|
||||||
|
def is_http_transport(self) -> bool:
|
||||||
|
"""Check if this config uses HTTP transport."""
|
||||||
|
return self.transport in ("streamable-http", "http", "sse")
|
||||||
|
|
||||||
|
|
||||||
class MCPConnectorCreate(BaseModel):
|
class MCPConnectorCreate(BaseModel):
|
||||||
|
|
|
||||||
|
|
@ -25,7 +25,8 @@ export const MCPConnectForm: FC<ConnectFormProps> = ({ onSubmit, isSubmitting })
|
||||||
const [showDetails, setShowDetails] = useState(false);
|
const [showDetails, setShowDetails] = useState(false);
|
||||||
const [testResult, setTestResult] = useState<MCPConnectionTestResult | null>(null);
|
const [testResult, setTestResult] = useState<MCPConnectionTestResult | null>(null);
|
||||||
|
|
||||||
const DEFAULT_CONFIG = JSON.stringify(
|
// Default config for stdio transport (local process)
|
||||||
|
const DEFAULT_STDIO_CONFIG = JSON.stringify(
|
||||||
{
|
{
|
||||||
name: "My MCP Server",
|
name: "My MCP Server",
|
||||||
command: "npx",
|
command: "npx",
|
||||||
|
|
@ -39,6 +40,22 @@ export const MCPConnectForm: FC<ConnectFormProps> = ({ onSubmit, isSubmitting })
|
||||||
2
|
2
|
||||||
);
|
);
|
||||||
|
|
||||||
|
// Default config for HTTP transport (remote server)
|
||||||
|
const DEFAULT_HTTP_CONFIG = JSON.stringify(
|
||||||
|
{
|
||||||
|
name: "My Remote MCP Server",
|
||||||
|
url: "https://your-mcp-server.com/mcp",
|
||||||
|
headers: {
|
||||||
|
"API_KEY": "your_api_key_here",
|
||||||
|
},
|
||||||
|
transport: "streamable-http",
|
||||||
|
},
|
||||||
|
null,
|
||||||
|
2
|
||||||
|
);
|
||||||
|
|
||||||
|
const DEFAULT_CONFIG = DEFAULT_STDIO_CONFIG;
|
||||||
|
|
||||||
const parseConfig = () => {
|
const parseConfig = () => {
|
||||||
const result = parseMCPConfig(configJson);
|
const result = parseMCPConfig(configJson);
|
||||||
if (result.error) {
|
if (result.error) {
|
||||||
|
|
@ -132,7 +149,31 @@ export const MCPConnectForm: FC<ConnectFormProps> = ({ onSubmit, isSubmitting })
|
||||||
<form id="mcp-connect-form" onSubmit={handleSubmit} className="space-y-6">
|
<form id="mcp-connect-form" onSubmit={handleSubmit} className="space-y-6">
|
||||||
<div className="rounded-xl border border-border bg-slate-400/5 dark:bg-white/5 p-4 sm:p-6 space-y-4">
|
<div className="rounded-xl border border-border bg-slate-400/5 dark:bg-white/5 p-4 sm:p-6 space-y-4">
|
||||||
<div className="space-y-2">
|
<div className="space-y-2">
|
||||||
<Label htmlFor="config">MCP Server Configuration (JSON)</Label>
|
<div className="flex items-center justify-between flex-wrap gap-2">
|
||||||
|
<Label htmlFor="config">MCP Server Configuration (JSON)</Label>
|
||||||
|
{!configJson && (
|
||||||
|
<div className="flex gap-1">
|
||||||
|
<Button
|
||||||
|
type="button"
|
||||||
|
variant="ghost"
|
||||||
|
size="sm"
|
||||||
|
className="h-6 px-2 text-xs text-muted-foreground hover:text-foreground"
|
||||||
|
onClick={() => handleConfigChange(DEFAULT_STDIO_CONFIG)}
|
||||||
|
>
|
||||||
|
Local Example
|
||||||
|
</Button>
|
||||||
|
<Button
|
||||||
|
type="button"
|
||||||
|
variant="ghost"
|
||||||
|
size="sm"
|
||||||
|
className="h-6 px-2 text-xs text-muted-foreground hover:text-foreground"
|
||||||
|
onClick={() => handleConfigChange(DEFAULT_HTTP_CONFIG)}
|
||||||
|
>
|
||||||
|
Remote Example
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
<Textarea
|
<Textarea
|
||||||
id="config"
|
id="config"
|
||||||
value={configJson}
|
value={configJson}
|
||||||
|
|
@ -143,8 +184,8 @@ export const MCPConnectForm: FC<ConnectFormProps> = ({ onSubmit, isSubmitting })
|
||||||
/>
|
/>
|
||||||
{jsonError && <p className="text-xs text-red-500">JSON Error: {jsonError}</p>}
|
{jsonError && <p className="text-xs text-red-500">JSON Error: {jsonError}</p>}
|
||||||
<p className="text-[10px] sm:text-xs text-muted-foreground">
|
<p className="text-[10px] sm:text-xs text-muted-foreground">
|
||||||
Paste a single MCP server configuration. Must include: name, command, args (optional),
|
<strong>Local (stdio):</strong> command, args, env, transport: "stdio"<br />
|
||||||
env (optional), transport (optional).
|
<strong>Remote (HTTP):</strong> url, headers, transport: "streamable-http"
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -46,13 +46,28 @@ export const MCPConfig: FC<MCPConfigProps> = ({ connector, onConfigChange, onNam
|
||||||
|
|
||||||
const serverConfig = connector.config?.server_config as MCPServerConfig | undefined;
|
const serverConfig = connector.config?.server_config as MCPServerConfig | undefined;
|
||||||
if (serverConfig) {
|
if (serverConfig) {
|
||||||
// Convert server config to JSON string for editing (name is in separate field)
|
const transport = serverConfig.transport || "stdio";
|
||||||
const configObj = {
|
|
||||||
command: serverConfig.command || "",
|
// Build config object based on transport type
|
||||||
args: serverConfig.args || [],
|
let configObj: Record<string, unknown>;
|
||||||
env: serverConfig.env || {},
|
|
||||||
transport: serverConfig.transport || "stdio",
|
if (transport === "streamable-http" || transport === "http" || transport === "sse") {
|
||||||
};
|
// HTTP transport - use url and headers
|
||||||
|
configObj = {
|
||||||
|
url: (serverConfig as any).url || "",
|
||||||
|
headers: (serverConfig as any).headers || {},
|
||||||
|
transport: transport,
|
||||||
|
};
|
||||||
|
} else {
|
||||||
|
// stdio transport (default) - use command, args, env
|
||||||
|
configObj = {
|
||||||
|
command: (serverConfig as any).command || "",
|
||||||
|
args: (serverConfig as any).args || [],
|
||||||
|
env: (serverConfig as any).env || {},
|
||||||
|
transport: transport,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
setConfigJson(JSON.stringify(configObj, null, 2));
|
setConfigJson(JSON.stringify(configObj, null, 2));
|
||||||
}
|
}
|
||||||
}, [isValidConnector, connector.name, connector.config?.server_config]);
|
}, [isValidConnector, connector.name, connector.config?.server_config]);
|
||||||
|
|
@ -163,8 +178,8 @@ export const MCPConfig: FC<MCPConfigProps> = ({ connector, onConfigChange, onNam
|
||||||
/>
|
/>
|
||||||
{jsonError && <p className="text-xs text-red-500">JSON Error: {jsonError}</p>}
|
{jsonError && <p className="text-xs text-red-500">JSON Error: {jsonError}</p>}
|
||||||
<p className="text-[10px] sm:text-xs text-muted-foreground">
|
<p className="text-[10px] sm:text-xs text-muted-foreground">
|
||||||
Edit your MCP server configuration. Must include: name, command, args (optional), env
|
<strong>Local (stdio):</strong> command, args, env, transport: "stdio"<br />
|
||||||
(optional), transport (optional).
|
<strong>Remote (HTTP):</strong> url, headers, transport: "streamable-http"
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -35,20 +35,27 @@ import { connectorsApiService } from "@/lib/apis/connectors-api.service";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Zod schema for MCP server configuration
|
* Zod schema for MCP server configuration
|
||||||
* Provides compile-time and runtime type safety
|
* Supports both stdio (local process) and HTTP (remote server) transports
|
||||||
*
|
*
|
||||||
* Exported for advanced use cases (e.g., form builders)
|
* Exported for advanced use cases (e.g., form builders)
|
||||||
*/
|
*/
|
||||||
export const MCPServerConfigSchema = z.object({
|
const StdioConfigSchema = z.object({
|
||||||
name: z.string().optional(),
|
name: z.string().optional(),
|
||||||
command: z
|
command: z.string().min(1, "Command cannot be empty"),
|
||||||
.string({ required_error: "Command field is required" })
|
|
||||||
.min(1, "Command cannot be empty"),
|
|
||||||
args: z.array(z.string()).optional().default([]),
|
args: z.array(z.string()).optional().default([]),
|
||||||
env: z.record(z.string(), z.string()).optional().default({}),
|
env: z.record(z.string(), z.string()).optional().default({}),
|
||||||
transport: z.enum(["stdio", "sse"]).optional().default("stdio"),
|
transport: z.enum(["stdio"]).optional().default("stdio"),
|
||||||
});
|
});
|
||||||
|
|
||||||
|
const HttpConfigSchema = z.object({
|
||||||
|
name: z.string().optional(),
|
||||||
|
url: z.string().url("URL must be a valid URL"),
|
||||||
|
headers: z.record(z.string(), z.string()).optional().default({}),
|
||||||
|
transport: z.enum(["streamable-http", "http", "sse"]),
|
||||||
|
});
|
||||||
|
|
||||||
|
export const MCPServerConfigSchema = z.union([StdioConfigSchema, HttpConfigSchema]);
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Shared MCP configuration validation result
|
* Shared MCP configuration validation result
|
||||||
*/
|
*/
|
||||||
|
|
@ -147,12 +154,19 @@ export const parseMCPConfig = (configJson: string): MCPConfigValidationResult =>
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
const config: MCPServerConfig = {
|
// Build config based on transport type
|
||||||
command: result.data.command,
|
const config: MCPServerConfig = result.data.transport === "stdio" || !result.data.transport
|
||||||
args: result.data.args,
|
? {
|
||||||
env: result.data.env,
|
command: (result.data as z.infer<typeof StdioConfigSchema>).command,
|
||||||
transport: result.data.transport,
|
args: (result.data as z.infer<typeof StdioConfigSchema>).args,
|
||||||
};
|
env: (result.data as z.infer<typeof StdioConfigSchema>).env,
|
||||||
|
transport: "stdio" as const,
|
||||||
|
}
|
||||||
|
: {
|
||||||
|
url: (result.data as z.infer<typeof HttpConfigSchema>).url,
|
||||||
|
headers: (result.data as z.infer<typeof HttpConfigSchema>).headers,
|
||||||
|
transport: result.data.transport as "streamable-http" | "http" | "sse",
|
||||||
|
};
|
||||||
|
|
||||||
// Cache the successfully parsed config
|
// Cache the successfully parsed config
|
||||||
configCache.set(configJson, {
|
configCache.set(configJson, {
|
||||||
|
|
|
||||||
|
|
@ -1,15 +1,24 @@
|
||||||
import { z } from "zod";
|
import { z } from "zod";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* MCP Server Configuration Schema (similar to Cursor's config)
|
* MCP Server Configuration Schema
|
||||||
|
* Supports both stdio (local process) and HTTP (remote server) transports
|
||||||
*/
|
*/
|
||||||
export const mcpServerConfig = z.object({
|
const stdioConfigSchema = z.object({
|
||||||
command: z.string().min(1, "Command is required"),
|
command: z.string().min(1, "Command is required"),
|
||||||
args: z.array(z.string()).default([]),
|
args: z.array(z.string()).default([]),
|
||||||
env: z.record(z.string(), z.string()).default({}),
|
env: z.record(z.string(), z.string()).default({}),
|
||||||
transport: z.enum(["stdio", "sse", "http"]).default("stdio"),
|
transport: z.enum(["stdio"]).default("stdio"),
|
||||||
});
|
});
|
||||||
|
|
||||||
|
const httpConfigSchema = z.object({
|
||||||
|
url: z.string().url("URL must be a valid URL"),
|
||||||
|
headers: z.record(z.string(), z.string()).default({}),
|
||||||
|
transport: z.enum(["streamable-http", "http", "sse"]),
|
||||||
|
});
|
||||||
|
|
||||||
|
export const mcpServerConfig = z.union([stdioConfigSchema, httpConfigSchema]);
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* MCP Connector Schemas
|
* MCP Connector Schemas
|
||||||
*/
|
*/
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue