apunkt/dograh
1
0
Fork 0
mirror of https://github.com/dograh-hq/dograh.git synced 2026-07-04 10:52:17 +02:00
Code Issues Projects Releases Packages Wiki Activity

feat(webhooks): durable retrying delivery for final webhooks (#478)

Browse source 
* feat(webhooks): durable retrying delivery for final webhooks

Final webhook nodes were fired inline with a single best-effort httpx POST
(run_integrations._execute_webhook_node). On a transient error the failure was
swallowed at three levels, so ARQ never retried and the final call report was
permanently lost -- leaving downstream receivers stuck (e.g. a CRM showing a
call as still "in conversation").

Replace the one-shot POST with a durable, idempotent delivery pipeline modelled
on the campaign retry pattern (persisted row + scheduled_for + bounded attempts):

- New webhook_deliveries table (WebhookDeliveryModel) is the source of truth.
  Payload is rendered once and frozen so retries are deterministic; secrets are
  not stored -- the credential is referenced by uuid and re-resolved at send time.
- run_integrations now persists a delivery row and enqueues deliver_webhook with
  a deterministic ARQ job id instead of sending inline.
- deliver_webhook (new ARQ task) sends the request and:
    * 2xx            -> succeeded
    * transient      -> retry with capped exponential backoff (RequestError /
                        5xx / 408 / 425 / 429), up to max_attempts then dead_letter
    * permanent 4xx  -> dead_letter immediately (no pointless looping)
  It is idempotent: a non-pending delivery is a no-op, so a duplicate enqueue or
  sweeper re-injection can't double-send.
- sweep_webhook_deliveries cron (every 5 min) re-enqueues overdue pending
  deliveries so nothing is lost to a worker restart / Redis flush.
- Stable X-Dograh-Delivery-Id / Workflow-Run-Id / Attempt headers let receivers
  dedupe retried deliveries.
- enqueue_job now forwards ARQ job options (_job_id, _defer_by); failures log
  repr(e) so empty-message errors like ConnectTimeout are diagnosable.

Config via DEFAULT_WEBHOOK_DELIVERY_CONFIG (env-overridable): max_attempts=5,
base_delay=30s, max_delay=600s, timeout=30s.

Tests cover payload rendering, persist+enqueue, success, transient retry,
retryable 5xx, permanent 4xx dead-letter, attempt exhaustion, and idempotency.
Migration verified to apply/rollback against Postgres; table/enum/indexes confirmed.

* fix(webhooks): atomic claim, safe success-recording, sweep paging, migration cleanup

Address review feedback on the webhook delivery pipeline:

- deliver_webhook now atomically claims a delivery (conditional UPDATE that
  leases scheduled_for) before sending, so concurrent ARQ executions can't
  double-send (the prior status=='pending' read was non-atomic).
- Recording success is moved out of the dead-letter try-block: if the receiver
  accepted the payload (2xx) but the success DB-write fails, the row is left
  pending for the sweeper to reconcile instead of being dead-lettered.
- The sweep keyset-paginates by id so a backlog over the page size is fully
  drained, and logs the true re-enqueued total.
- Migration downgrade drops the enum via op.execute(DROP TYPE IF EXISTS ...)
  instead of the deprecated op.get_bind().

* fix(webhooks): idempotent delivery creation and drop secret custom headers

Address the remaining review feedback:

- Add a (workflow_run_id, webhook_node_id) unique constraint and make
  create_webhook_delivery a get-or-create returning (delivery, created). A
  retried run_integrations now reuses the existing row instead of creating and
  sending a duplicate final webhook; only a freshly-created row is enqueued.
- Stop persisting secret-looking custom headers (Authorization, X-API-Key,
  Cookie, ...) in plaintext on the delivery row: they are dropped with a warning
  pointing at the credential store (which is re-resolved securely at send time).
  Non-secret custom headers are unaffected.

* fix(webhooks): harden idempotency key, secret-header match, sweep reclaim id

Address follow-up review feedback:

- webhook_node_id is now NOT NULL so a NULL can't slip past the
  (workflow_run_id, webhook_node_id) unique constraint and create duplicates.
- Secret-header filtering matches normalized markers (auth/token/secret/cookie/
  api-key/...) instead of an exact name list, catching variants like
  X-Custom-Auth-Token while leaving benign headers (e.g. X-Idempotency-Key).
- The sweeper re-enqueues with a reclaim-specific job id (the lease timestamp)
  so reconciling a delivered-but-unrecorded row isn't deduped against the
  original attempt's already-completed ARQ job. The atomic claim still ensures
  at most one send.

* fix(webhooks): scope delivery rows to workflow org

---------

Co-authored-by: Abhishek Kumar <abhishek@a6k.me>
This commit is contained in:
Tararais 2026-07-02 17:14:14 +01:00 • committed by GitHub
parent 3a770a6538
commit fd0d144b08
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
12 changed files with 1360 additions and 141 deletions
Download patch file Download diff file Expand all files Collapse all files

218
api/tasks/run_integrations.py
View file

@ -1,15 +1,15 @@
"""Execute integrations (QA analysis, webhooks) after workflow run completion."""
import random
from datetime import UTC, datetime
from typing import Any, Dict, Optional
import httpx
from loguru import logger
from pipecat.utils.enums import EndTaskReason
from pipecat.utils.run_context import set_current_org_id, set_current_run_id
from pydantic import ValidationError
from api.constants import BACKEND_API_ENDPOINT
from api.constants import BACKEND_API_ENDPOINT, DEFAULT_WEBHOOK_DELIVERY_CONFIG
from api.db import db_client
from api.db.models import WorkflowRunModel
from api.enums import OrganizationConfigurationKey
@ -26,7 +26,7 @@ from api.services.workflow.dto import (
WebhookRFNode,
)
from api.services.workflow.qa import run_per_node_qa_analysis
from api.utils.credential_auth import build_auth_header
from api.tasks.function_names import FunctionNames
from api.utils.recording_artifacts import get_recording_storage_key
from api.utils.template_renderer import render_template
@ -315,13 +315,15 @@ async def run_integrations_post_workflow_run(_ctx, workflow_run_id: int):
webhook_data = webhook_node.data
try:
await _execute_webhook_node(
await _enqueue_webhook_delivery(
webhook_data=webhook_data,
render_context=render_context,
organization_id=organization_id,
workflow_run_id=workflow_run_id,
webhook_node_id=str(node_id),
)
except Exception as e:
logger.warning(f"Failed to execute webhook '{webhook_data.name}': {e}")
logger.warning(f"Failed to enqueue webhook '{webhook_data.name}': {e}")
except Exception as e:
logger.error(f"Error running integrations: {e}", exc_info=True)
@ -387,98 +389,140 @@ def _build_render_context(
return context
async def _execute_webhook_node(
webhook_data: WebhookNodeData,
render_context: Dict[str, Any],
organization_id: int,
) -> bool:
def _build_webhook_payload(
webhook_data: WebhookNodeData, render_context: Dict[str, Any]
) -> Any:
"""Render the webhook payload once, so retries are deterministic.
Always surfaces the call disposition on the outgoing payload, even when the
template author didn't reference it. Fill only if absent so a template that
sets it explicitly keeps its own value.
"""
Execute a single webhook node.
Args:
webhook_data: The validated webhook node data
render_context: Context for template rendering
organization_id: For credential lookup
Returns:
True if successful, False otherwise
"""
webhook_name = webhook_data.name
if not webhook_data.enabled:
logger.debug(f"Webhook '{webhook_name}' is disabled, skipping")
return True
url = webhook_data.endpoint_url
if not url:
logger.warning(f"Webhook '{webhook_name}' has no endpoint URL")
return False
headers = {"Content-Type": "application/json"}
credential_uuid = webhook_data.credential_uuid
if credential_uuid:
credential = await db_client.get_credential_by_uuid(
credential_uuid, organization_id
)
if credential:
auth_header = build_auth_header(credential)
headers.update(auth_header)
logger.debug(f"Applied credential '{credential.name}' to webhook")
else:
logger.warning(
f"Credential {credential_uuid} not found for webhook '{webhook_name}'"
)
for h in webhook_data.custom_headers or []:
if h.key and h.value:
headers[h.key] = h.value
payload = render_template(webhook_data.payload_template or {}, render_context)
# Always surface the call disposition on the outgoing payload, even when the
# template author didn't reference it. Fill only if absent so a template that
# sets it explicitly keeps its own value.
if isinstance(payload, dict):
gathered_context = render_context.get("gathered_context") or {}
payload.setdefault(
"call_disposition", gathered_context.get("call_disposition", "")
)
return payload
# Substrings that mark a header as likely carrying a secret. Matched against the
# normalized key so variants are caught too (e.g. ``X-Custom-Auth-Token``,
# ``My-Api-Key``), not just exact names. Their values are NOT persisted on the
# delivery row (which would store them in plaintext); secrets belong in the
# credential store, re-resolved at send time. Bare "key" is intentionally absent
# to avoid dropping benign headers like ``X-Idempotency-Key``.
_SECRET_HEADER_MARKERS = (
"authorization",
"auth",
"token",
"secret",
"password",
"passwd",
"cookie",
"credential",
"api-key",
"apikey",
"api_key",
"access-key",
)
def _looks_like_secret_header(key: str) -> bool:
normalized = key.strip().lower()
return any(marker in normalized for marker in _SECRET_HEADER_MARKERS)
def _safe_custom_headers(
webhook_data: WebhookNodeData, webhook_name: str
) -> list[dict]:
"""Custom headers to persist, with secret-looking ones dropped.
Persisting arbitrary header values would store credentials (Authorization,
X-API-Key, ...) in plaintext on the delivery row. Drop those and tell the
operator to use a credential instead.
"""
safe = []
for h in webhook_data.custom_headers or []:
if not (h.key and h.value):
continue
if _looks_like_secret_header(h.key):
logger.warning(
f"Webhook '{webhook_name}' custom header '{h.key}' looks like a "
f"secret; it will not be stored or sent. Use a credential instead."
)
continue
safe.append({"key": h.key, "value": h.value})
return safe
async def _enqueue_webhook_delivery(
webhook_data: WebhookNodeData,
render_context: Dict[str, Any],
organization_id: int,
workflow_run_id: int,
webhook_node_id: str,
) -> None:
"""Persist a durable delivery record and enqueue its first send attempt.
The actual HTTP request is performed by the ``deliver_webhook`` task, which
retries transient failures with backoff and dead-letters exhausted/permanent
ones. This replaces the previous one-shot, best-effort inline POST that lost
the webhook entirely on a single network error.
Idempotent on ``(workflow_run_id, webhook_node_id)``: a retried run reuses the
existing delivery row and does not enqueue a second send.
"""
webhook_name = webhook_data.name
if not webhook_data.enabled:
logger.debug(f"Webhook '{webhook_name}' is disabled, skipping")
return
url = webhook_data.endpoint_url
if not url:
logger.warning(f"Webhook '{webhook_name}' has no endpoint URL")
return
payload = _build_webhook_payload(webhook_data, render_context)
# Persist non-secret request definition. The credential is stored by reference
# (uuid) and re-resolved at send time so secrets never land in this row.
custom_headers = _safe_custom_headers(webhook_data, webhook_name)
method = (webhook_data.http_method or "POST").upper()
logger.info(f"Executing webhook '{webhook_name}': {method}")
delivery, created = await db_client.create_webhook_delivery(
workflow_run_id=workflow_run_id,
organization_id=organization_id,
endpoint_url=url,
payload=payload,
max_attempts=DEFAULT_WEBHOOK_DELIVERY_CONFIG["max_attempts"],
http_method=method,
webhook_name=webhook_name,
custom_headers=custom_headers or None,
credential_uuid=webhook_data.credential_uuid,
webhook_node_id=webhook_node_id,
)
try:
async with httpx.AsyncClient() as client:
if method in ("POST", "PUT", "PATCH"):
response = await client.request(
method=method,
url=url,
json=payload,
headers=headers,
timeout=30.0,
)
else: # GET, DELETE
response = await client.request(
method=method,
url=url,
headers=headers,
timeout=30.0,
)
response.raise_for_status()
logger.info(f"Webhook '{webhook_name}' succeeded: {response.status_code}")
return True
except httpx.HTTPStatusError as e:
logger.warning(
f"Webhook '{webhook_name}' failed: {e.response.status_code} - {e.response.text[:200]}"
if not created:
logger.info(
f"Webhook '{webhook_name}' delivery already exists for run "
f"{workflow_run_id} node {webhook_node_id}; not re-enqueuing"
)
return False
except httpx.RequestError as e:
logger.warning(f"Webhook '{webhook_name}' request error: {e}")
return False
except Exception as e:
logger.error(f"Webhook '{webhook_name}' unexpected error: {e}")
return False
return
# Lazy import avoids a circular import (arq imports this module at load time).
from api.tasks.arq import enqueue_job
await enqueue_job(
FunctionNames.DELIVER_WEBHOOK,
delivery.id,
_job_id=f"webhook-delivery-{delivery.id}-0",
)
logger.info(
f"Enqueued webhook '{webhook_name}' delivery {delivery.delivery_uuid} "
f"for run {workflow_run_id}"
)