apunkt/dograh
1
0
Fork 0
mirror of https://github.com/dograh-hq/dograh.git synced 2026-07-25 12:01:04 +02:00
Code Issues Projects Releases Packages Wiki Activity

feat: add tool test panel for HTTP API tools (#547)

Browse source 
* feat: add tool test panel for HTTP API tools

Lets developers run a saved HTTP API tool against its real endpoint
from the tool detail page, without needing a live call. Reuses the
production execute_http_tool path so test behavior matches call-time
behavior.

- New POST /tools/{tool_uuid}/test route
- Test panel with per-parameter typed inputs and auto-detected
  context variable inputs (from preset parameter templates)
- Validate parameter name uniqueness on save, matching the existing
  transferParameters check
- Fix stale FunctionCallsFromLLMInfoFrame import causing test
  collection failures against pipecat-ai 1.5.0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: revert local-venv pipecat drift fix, apply ruff import formatting

test_custom_tools.py and test_unregistered_function_call.py were edited
locally to drop FunctionCallsFromLLMInfoFrame after hitting an
ImportError — that error was from a stale local pipecat-ai package, not
a real drift. CI's pipecat build emits this frame and the test asserted
on it, so removing it broke test_llm_calls_custom_tool_handler and its
unregistered-call counterpart. Reverted both files to match main.

Also applied ruff's import-sort/format fix to test_mcp_tool_route.py to
clear the drift-check job (split the aliased import into its own
`from ... import (...)` block, wrapped a long monkeypatch.setattr call).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: avoid pytest collecting test_tool import as a test, sync OpenAPI spec

pytest's default discovery matches any top-level test_* name in a test
module, including imported functions — importing the route handler as
`test_tool as test_tool_route` still matched the pattern, so pytest
tried to run it as a test and failed injecting fixtures for tool_uuid/
request/user. Renamed the alias to call_test_tool_route.

Also regenerated docs/api-reference/openapi.json for the new
POST /tools/{tool_uuid}/test route and its two schemas (couldn't run
the dump script locally — pipecat-ai version mismatch documented
separately — so hand-built the diff to exactly match FastAPI's
get_openapi() output format, verified against neighboring routes).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: address cubic review findings on test panel

- Resolve dotted context-variable keys into nested objects before
  posting the test request. render_template's get_nested_value walks
  nested dicts, so a flat key like "runtime_configuration.realtime_model"
  never matched — templates referencing nested context always resolved
  to empty.
- Restrict isHttpApiTool to an explicit category equality check instead
  of inferring it from exclusions. native/integration tools are
  currently disabled in the create-tool UI so this wasn't reachable
  today, but the exclusion list silently goes stale as new categories
  are added.
- Stop showing a green success badge for non-2xx responses.
  execute_http_tool returns status: "success" for any HTTP exchange
  that completes, regardless of status code — only transport-level
  errors (timeout, connection failure) get status: "error". The test
  panel now checks status_code is in the 2xx range before treating the
  call as a success.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: address second round of cubic/greptile findings

- Guard setNestedValue against prototype-pollution keys (__proto__,
  constructor, prototype) in the dotted context-var path before
  traversing.
- Normalize status to "error" in the test route when the upstream
  status_code is >= 400. execute_http_tool only distinguishes
  transport-level failures (timeout, connection error) from
  "success" — a completed 4xx/5xx exchange still came back as
  "success" from the executor.
- Seed testArgValues defaults for number/boolean parameters via a
  useEffect keyed on the parameters array, so a required number or
  boolean field isn't silently omitted from the test request if the
  user never touches its input.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: only seed test-arg defaults for required number/boolean params

Seeding optional number/boolean parameters silently changed the test
request — an optional boolean flag the tester never touched was sent
as true, which can flip upstream behavior unintentionally. Restrict
seeding to required parameters.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: match whitespace and fallback-filter syntax in context var detection

extractContextVars required an exact {{initial_context.foo}} with no
whitespace and no filter suffix, but the backend's TEMPLATE_VAR_PATTERN
(and render_template) accepts {{ initial_context.foo }} and
{{initial_context.foo | fallback:value}}. A preset parameter saved with
either of those forms resolved fine in production but showed no input
in the test panel, so testing always sent it empty context.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(tool-test): add hint and request_* fields to ToolTestResponse

Extends ToolTestResponse with hint, request_method, request_url,
request_body, and request_params so the frontend can surface what was
actually sent and a human-readable hint about why a test call failed.

* feat(tool-test): add status-code hints and request_method/url/body/params to test_tool()

* style: ruff-format test_mcp_tool_route.py

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(tool-test): wire hint banner and request block into result panel

Backend has returned hint/request_method/request_url/request_body/
request_params since d45ea851/60aaf31d but the frontend never
displayed them. Extends ToolTestResult with the new fields and renders
an amber hint banner (for 400/401/403/404/405/408/409/415/422/429/5xx)
plus a Request block above the response showing exactly what was sent.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(tool-test): include resolved preset params in request_body/params

Found via live GET/POST testing: the Request preview showed only the
model-provided arguments, not what execute_http_tool actually sends.
execute_http_tool merges resolved_arguments = {**arguments,
**preset_arguments} before building the outbound body/params — preset
params (e.g. {{initial_context.metadata.channel}}) are invisible to
the model but still go out on the wire. The preview now mirrors that
merge via the same _resolve_preset_parameters helper, so a dev sees
exactly what was sent, not just what the model provided.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(tool-test): add generateSampleValue helper for sample-fill button

* feat(tool-test): add Fill sample values button for arguments and context vars

Moved generateSampleValue out of page.tsx into a sibling helpers module:
Next.js's typed-route checker rejects extra named exports on a page.tsx
file (tsc error TS2344 on .next/types), so the helper and its test import
now live in testPanelHelpers.ts instead.

* feat(tool-test): add JSON edit modal state and handlers

* feat(tool-test): collapsed preview + edit modal for object/array test parameters

* fix(tool-test): validate JSON on modal open, not just on edit

Opening the JSON edit modal on an untouched object/array param (no value
yet in testArgValues) loaded an empty draft with jsonEditError hardcoded
to null, so Save was enabled despite invalid JSON and silently no-op'd on
click. Now runs the same JSON.parse check used by the live textarea
validation when the modal opens.

* chore: regenerate openapi.json for ToolTestResponse hint/request_* fields

drift-check on PR #547 was failing because the earlier hint/request_method/
request_url/request_body/request_params fields added to ToolTestResponse
were never reflected in the dumped spec. Regenerated via
scripts.dump_docs_openapi.

* fix(tool-test): serialize object/array args for GET/DELETE query params, add unsaved-changes banner

httpx raises a TypeError when a query param value is a dict/list, which
was silently caught and surfaced as a generic tool-execution error —
this is what actually broke test requests, not just the "[object
Object]" display. JSON-stringify object/array arguments before they
become query params, in both the live execute_http_tool() path and the
test route's request_params display shaping.

Also adds an unsaved-changes warning banner above Test Tool, shown
when the live form state diverges from the last-saved HTTP API config,
since Test Tool always runs the saved config.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(tool-test): keep empty request body in preview; normalize headers in snapshot

- POST/PUT/PATCH with no arguments now shows `{}` in the request body
  preview instead of null — matches what execute_http_tool actually sends
  over the wire (json={})
- buildHttpToolTestSnapshot normalizes headers from KeyValueItem[] to a
  deduped key→value map before serializing, matching the shape saved to
  the backend; duplicate header keys no longer cause a false unsaved-
  changes warning

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(tool-test): update assertion for empty POST body preview

test_tool_test_no_arguments_leaves_body_and_params_none expected
request_body=None for a POST with no arguments. The fix to preserve {}
in the preview makes request_body={} the correct assertion.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(tools): refine HTTP tool testing

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Abhishek Kumar <abhishek@a6k.me>
This commit is contained in:
Rushil 2026-07-18 16:00:50 +05:30 • committed by GitHub
parent 104dc56706
commit f4c5954a39
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
15 changed files with 1757 additions and 67 deletions
Download patch file Download diff file Expand all files Collapse all files

118
api/tests/test_custom_tools.py
View file

@ -30,6 +30,7 @@ from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.services.llm_service import FunctionCallParams
from api.enums import WorkflowRunMode
from api.services.configuration.masking import mask_key
from api.services.workflow.pipecat_engine_custom_tools import get_function_schema
from api.services.workflow.tools.custom_tool import (
_coerce_parameter_value,
@ -430,6 +431,7 @@ class TestExecuteHttpTool:
assert result["status"] == "success"
assert result["status_code"] == 201
assert result["data"]["id"] == 123
assert "request_headers" not in result
@pytest.mark.asyncio
async def test_post_request_sends_nested_json_body(self):
@ -546,6 +548,55 @@ class TestExecuteHttpTool:
}
assert result["status"] == "success"
@pytest.mark.asyncio
async def test_post_request_accepts_pre_resolved_preset_params(self):
"""The test endpoint can supply preset values without call context."""
tool = MockToolModel(
tool_uuid="test-uuid-preset-override",
name="Create Lead",
description="Create a lead with caller context",
category="http_api",
definition={
"schema_version": 1,
"type": "http_api",
"config": {
"method": "POST",
"url": "https://api.example.com/leads",
"timeout_ms": 5000,
"preset_parameters": [
{
"name": "phone_number",
"type": "string",
"value_template": "{{initial_context.phone_number}}",
"required": True,
}
],
},
},
)
with patch(
"api.services.workflow.tools.custom_tool.httpx.AsyncClient"
) as mock_client_class:
mock_client = AsyncMock()
mock_response = Mock()
mock_response.status_code = 201
mock_response.json.return_value = {"id": 123}
mock_client.request.return_value = mock_response
mock_client_class.return_value.__aenter__.return_value = mock_client
result = await execute_http_tool(
tool,
{"name": "John"},
preset_params={"phone_number": "+14155550123"},
)
assert mock_client.request.call_args.kwargs["json"] == {
"name": "John",
"phone_number": "+14155550123",
}
assert result["status"] == "success"
@pytest.mark.asyncio
async def test_missing_required_preset_parameter_returns_error(self):
"""Test that required preset parameters fail before the HTTP request."""
@ -619,6 +670,52 @@ class TestExecuteHttpTool:
assert result["status"] == "success"
@pytest.mark.asyncio
async def test_get_request_serializes_object_and_array_query_params(self):
"""Object/array-typed arguments must be JSON-stringified for GET query
params — httpx raises a TypeError if a dict/list is passed as-is."""
tool = MockToolModel(
tool_uuid="test-uuid",
name="Search Users",
description="Search for users",
category="http_api",
definition={
"schema_version": 1,
"type": "http_api",
"config": {
"method": "GET",
"url": "https://api.example.com/users/search",
"timeout_ms": 5000,
},
},
)
arguments = {
"source": "voice",
"metadata": {"campaign": "spring"},
"tags": ["a", "b"],
}
with patch(
"api.services.workflow.tools.custom_tool.httpx.AsyncClient"
) as mock_client_class:
mock_client = AsyncMock()
mock_response = Mock()
mock_response.status_code = 200
mock_response.json.return_value = {"users": []}
mock_client.request.return_value = mock_response
mock_client_class.return_value.__aenter__.return_value = mock_client
result = await execute_http_tool(tool, arguments)
call_kwargs = mock_client.request.call_args.kwargs
assert call_kwargs["params"] == {
"source": "voice",
"metadata": '{"campaign": "spring"}',
"tags": '["a", "b"]',
}
assert result["status"] == "success"
@pytest.mark.asyncio
async def test_get_request_without_arguments_preserves_url_query_params(self):
"""Empty runtime args should not override query params already in the URL."""
@ -762,11 +859,17 @@ class TestExecuteHttpTool:
mock_client.request.return_value = mock_response
mock_client_class.return_value.__aenter__.return_value = mock_client
await execute_http_tool(tool, {"data": "test"})
result = await execute_http_tool(
tool, {"data": "test"}, include_request_headers=True
)
call_kwargs = mock_client.request.call_args.kwargs
assert call_kwargs["headers"]["X-API-Key"] == "secret-key"
assert call_kwargs["headers"]["X-Custom-Header"] == "custom-value"
assert result["request_headers"] == {
"X-API-Key": "secret-key",
"X-Custom-Header": "custom-value",
}
@pytest.mark.asyncio
async def test_request_includes_auth_header_from_credential(self):
@ -807,7 +910,12 @@ class TestExecuteHttpTool:
with patch("api.services.workflow.tools.custom_tool.db_client") as mock_db:
mock_db.get_credential_by_uuid = AsyncMock(return_value=mock_credential)
await execute_http_tool(tool, {"data": "test"}, organization_id=1)
result = await execute_http_tool(
tool,
{"data": "test"},
organization_id=1,
include_request_headers=True,
)
# Verify credential was fetched
mock_db.get_credential_by_uuid.assert_called_once_with(
@ -819,6 +927,12 @@ class TestExecuteHttpTool:
assert (
call_kwargs["headers"]["Authorization"] == "Bearer my-secret-token"
)
assert result["request_headers"]["Authorization"] == mask_key(
"Bearer my-secret-token"
)
assert (
"my-secret-token" not in result["request_headers"]["Authorization"]
)
@pytest.mark.asyncio
async def test_no_credential_lookup_without_organization_id(self):

352
api/tests/test_mcp_tool_route.py
View file

@ -26,9 +26,15 @@ from api.routes.tool import (
CreateToolRequest,
McpToolConfig,
McpToolDefinition,
ToolTestRequest,
ToolTestResponse,
UpdateToolRequest,
_populate_discovered_tools,
refresh_mcp_tools,
router,
)
from api.routes.tool import (
test_tool as call_test_tool_route,
)
from api.services.workflow.tools.mcp_tool import (
validate_mcp_definition,
@ -420,6 +426,314 @@ def _mcp_tool_model(org_id=1):
return t
def _http_tool_model(method="GET"):
t = MagicMock()
t.tool_uuid = "tu-http"
t.name = "Mock HTTP"
t.category = "http_api"
t.definition = {
"schema_version": 1,
"type": "http_api",
"config": {"method": method, "url": "https://example.com/search"},
}
return t
@pytest.mark.asyncio
async def test_tool_executes_http_api_tool_with_llm_and_preset_params(monkeypatch):
import api.routes.tool as tool_route
tool = _http_tool_model()
monkeypatch.setattr(
tool_route.db_client, "get_tool_by_uuid", AsyncMock(return_value=tool)
)
executor = AsyncMock(
return_value={
"status": "success",
"status_code": 200,
"data": {"ok": True},
}
)
monkeypatch.setattr(tool_route, "execute_http_tool", executor)
resp = await call_test_tool_route(
"tu-http",
request=ToolTestRequest(
llm_params={"query": "cart"},
preset_params={
"customer_id": "c_123",
"sentiment": "cooperative",
},
),
user=_fake_user(),
)
assert resp.status == "success"
assert resp.status_code == 200
assert resp.data == {"ok": True}
assert resp.error is None
executor.assert_awaited_once_with(
tool,
{"query": "cart"},
preset_params={
"customer_id": "c_123",
"sentiment": "cooperative",
},
organization_id=1,
include_request_headers=True,
)
assert resp.hint is None
assert resp.request_method == "GET"
assert resp.request_url == "https://example.com/search"
assert resp.request_headers == {}
assert resp.request_body is None
assert resp.request_params == {
"query": "cart",
"customer_id": "c_123",
"sentiment": "cooperative",
}
@pytest.mark.asyncio
async def test_tool_test_sets_request_body_for_post_method(monkeypatch):
import api.routes.tool as tool_route
tool = _http_tool_model(method="POST")
monkeypatch.setattr(
tool_route.db_client, "get_tool_by_uuid", AsyncMock(return_value=tool)
)
monkeypatch.setattr(
tool_route,
"execute_http_tool",
AsyncMock(
return_value={"status": "success", "status_code": 200, "data": {"id": 1}}
),
)
resp = await call_test_tool_route(
"tu-http",
request=ToolTestRequest(llm_params={"name": "Ada"}),
user=_fake_user(),
)
assert resp.request_method == "POST"
assert resp.request_body == {"name": "Ada"}
assert resp.request_params is None
@pytest.mark.asyncio
async def test_tool_test_returns_masked_effective_request_headers(monkeypatch):
import api.routes.tool as tool_route
tool = _http_tool_model(method="POST")
monkeypatch.setattr(
tool_route.db_client, "get_tool_by_uuid", AsyncMock(return_value=tool)
)
monkeypatch.setattr(
tool_route,
"execute_http_tool",
AsyncMock(
return_value={
"status": "success",
"status_code": 200,
"data": {"ok": True},
"request_headers": {
"X-Tenant": "acme",
"Authorization": "****************oken",
},
}
),
)
resp = await call_test_tool_route(
"tu-http", request=ToolTestRequest(), user=_fake_user()
)
assert resp.request_headers == {
"X-Tenant": "acme",
"Authorization": "****************oken",
}
@pytest.mark.asyncio
async def test_tool_test_request_body_includes_resolved_preset_parameters(
monkeypatch,
):
"""The Request preview includes direct preset values alongside LLM values."""
import api.routes.tool as tool_route
tool = _http_tool_model(method="POST")
tool.definition["config"]["preset_parameters"] = [
{
"name": "source",
"type": "string",
"value_template": "{{initial_context.metadata.channel}}",
"required": True,
}
]
monkeypatch.setattr(
tool_route.db_client, "get_tool_by_uuid", AsyncMock(return_value=tool)
)
monkeypatch.setattr(
tool_route,
"execute_http_tool",
AsyncMock(
return_value={"status": "success", "status_code": 200, "data": {"ok": True}}
),
)
resp = await call_test_tool_route(
"tu-http",
request=ToolTestRequest(
llm_params={"name": "Ada"},
preset_params={"source": "web_widget"},
),
user=_fake_user(),
)
assert resp.request_body == {"name": "Ada", "source": "web_widget"}
@pytest.mark.asyncio
async def test_tool_test_no_arguments_post_shows_empty_body(monkeypatch):
import api.routes.tool as tool_route
tool = _http_tool_model(method="POST")
monkeypatch.setattr(
tool_route.db_client, "get_tool_by_uuid", AsyncMock(return_value=tool)
)
monkeypatch.setattr(
tool_route,
"execute_http_tool",
AsyncMock(return_value={"status": "success", "status_code": 200, "data": None}),
)
resp = await call_test_tool_route(
"tu-http", request=ToolTestRequest(), user=_fake_user()
)
# POST with no arguments sends json={} over the wire; preview must show {}
# so callers can distinguish an absent body from an empty one.
assert resp.request_body == {}
assert resp.request_params is None
@pytest.mark.asyncio
@pytest.mark.parametrize(
"status_code,expected_snippet",
[
(400, "HTTP 400 Bad Request"),
(401, "HTTP 401 Unauthorized"),
(403, "HTTP 403 Forbidden"),
(404, "HTTP 404 Not Found"),
(405, "HTTP 405 Method Not Allowed"),
(408, "HTTP 408 Request Timeout"),
(409, "HTTP 409 Conflict"),
(415, "HTTP 415 Unsupported Media Type"),
(422, "HTTP 422 Unprocessable Entity"),
(429, "HTTP 429 Too Many Requests"),
(500, "HTTP 500"),
(503, "HTTP 503"),
],
)
async def test_tool_test_hint_for_status_code(
monkeypatch, status_code, expected_snippet
):
import api.routes.tool as tool_route
tool = _http_tool_model(method="POST")
monkeypatch.setattr(
tool_route.db_client, "get_tool_by_uuid", AsyncMock(return_value=tool)
)
monkeypatch.setattr(
tool_route,
"execute_http_tool",
AsyncMock(
return_value={
"status": "error",
"status_code": status_code,
"error": "boom",
}
),
)
resp = await call_test_tool_route(
"tu-http", request=ToolTestRequest(llm_params={"a": 1}), user=_fake_user()
)
assert resp.hint is not None
assert resp.hint.startswith(expected_snippet)
@pytest.mark.asyncio
async def test_tool_test_no_hint_on_success(monkeypatch):
import api.routes.tool as tool_route
tool = _http_tool_model(method="GET")
monkeypatch.setattr(
tool_route.db_client, "get_tool_by_uuid", AsyncMock(return_value=tool)
)
monkeypatch.setattr(
tool_route,
"execute_http_tool",
AsyncMock(return_value={"status": "success", "status_code": 200, "data": {}}),
)
resp = await call_test_tool_route(
"tu-http", request=ToolTestRequest(), user=_fake_user()
)
assert resp.hint is None
@pytest.mark.asyncio
async def test_tool_test_no_hint_for_uncovered_status_code(monkeypatch):
import api.routes.tool as tool_route
tool = _http_tool_model(method="GET")
monkeypatch.setattr(
tool_route.db_client, "get_tool_by_uuid", AsyncMock(return_value=tool)
)
monkeypatch.setattr(
tool_route,
"execute_http_tool",
AsyncMock(
return_value={"status": "error", "status_code": 418, "error": "teapot"}
),
)
resp = await call_test_tool_route(
"tu-http", request=ToolTestRequest(), user=_fake_user()
)
assert resp.hint is None
def test_tool_test_route_is_registered():
assert any(
route.path == "/tools/{tool_uuid}/test" and "POST" in route.methods
for route in router.routes
)
@pytest.mark.asyncio
async def test_tool_rejects_non_http_api_tool(monkeypatch):
import api.routes.tool as tool_route
monkeypatch.setattr(
tool_route.db_client,
"get_tool_by_uuid",
AsyncMock(return_value=_mcp_tool_model()),
)
with pytest.raises(HTTPException) as ei:
await call_test_tool_route(
"tu-mcp", request=ToolTestRequest(), user=_fake_user()
)
assert ei.value.status_code == 400
@pytest.mark.asyncio
async def test_refresh_success(monkeypatch):
import api.services.tool_management as tool_svc
@ -484,3 +798,41 @@ async def test_refresh_not_found_is_404(monkeypatch):
with pytest.raises(HTTPException) as ei:
await refresh_mcp_tools("nope", user=_fake_user())
assert ei.value.status_code == 404
def test_tool_test_response_has_hint_and_request_fields():
"""ToolTestResponse must carry hint + request_method/url/body/params
so the frontend can show what was sent and why it may have failed."""
resp = ToolTestResponse(
status="error",
status_code=405,
data=None,
error="Method Not Allowed",
duration_ms=12,
hint="HTTP 405 Method Not Allowed — the endpoint rejected the configured method (POST).",
request_method="POST",
request_url="https://example.com/thing",
request_headers={"Authorization": "********oken"},
request_body={"a": 1},
request_params=None,
)
assert resp.hint.startswith("HTTP 405")
assert resp.request_method == "POST"
assert resp.request_url == "https://example.com/thing"
assert resp.request_headers == {"Authorization": "********oken"}
assert resp.request_body == {"a": 1}
assert resp.request_params is None
def test_tool_test_response_request_fields_default_to_none_or_required():
"""hint/request_body/request_params are optional; request_method/url are required."""
resp = ToolTestResponse(
status="success",
duration_ms=5,
request_method="GET",
request_url="https://example.com/thing",
)
assert resp.hint is None
assert resp.request_headers == {}
assert resp.request_body is None
assert resp.request_params is None