Two call sites dropped the MPS billing-v2 protocol, so orgs on model
config v2 got 400 "Service Key uses billing v2" from MPS:
- text_chat_runner built PipecatEngine without embeddings_provider
(extracted but never passed), so knowledge-base retrieval fell through
to the plain OpenAI-compatible client instead of DograhEmbeddingService
and sent no correlation_id/mps_billing_version metadata. Also pass
endpoint/api_version for Azure BYOK parity with run_pipeline.
- node_summary built its QA LLM without the run's correlation id, unlike
its sibling call in qa/analysis.py.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix: gate OSS email/password auth endpoints outside local auth mode
The /auth signup/login/me routes were mounted unconditionally, so the
SaaS deployment accepted unauthenticated signups that created oss_*
provider-id users (and auto-provisioned MPS service keys) bypassing
Stack Auth entirely.
Gate them with a router-level dependency that 404s when AUTH_PROVIDER
is not "local", rather than conditionally mounting the router, so the
OpenAPI spec and the clients generated from it stay identical across
deployment modes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: keep current user route available in stack auth
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat: add Helm chart for Kubernetes deployment
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Replace bundled Bitnami subcharts with in-chart manifests on official images
The Bitnami catalog removed all versioned image tags from docker.io/bitnami in
Aug 2025 (old images frozen in bitnamilegacy, maintained catalog now behind a
Broadcom subscription), so the bundled postgresql/redis/minio subcharts no
longer pull. Replace them with plain in-chart manifests built on official
upstream images, keeping the internal/all-in-one path fully self-contained and
free of third-party chart packaging that can disappear:
- internal-postgres.yaml: pgvector/pgvector:pg17 — upstream Postgres plus the
`vector` extension the migrations require. POSTGRES_USER=dograh is the initdb
superuser, so CREATE EXTENSION vector succeeds.
- internal-redis.yaml: redis:7.4-alpine, password-protected, AOF persistence.
- internal-minio.yaml: minio/minio, root creds shared with the app via a single
secret (can't drift); the app auto-creates its bucket.
Service/secret names are unchanged (<rel>-postgresql, <rel>-redisinternal-master,
<rel>-minio) so the app wiring is untouched. Dep passwords are generated once and
persisted across upgrades via lookup. Drop the Chart.yaml dependencies,
Chart.lock, and the `helm dependency` step; the internal manifests gate on the
mode toggles (database.mode=internal, etc.).
Also fixes surfaced by smoke-testing on a live EKS cluster:
- Dockerfile: ship the per-service run_*.sh entrypoints the chart invokes.
- migrate-job: run as a post-install/pre-upgrade hook (the bundled Postgres does
not exist during pre-install) with a wait-for-postgres init container.
- backend env: declare POSTGRES_PASSWORD/REDIS_PASSWORD before the DATABASE_URL/
REDIS_URL that interpolate them (Kubernetes only expands back-references).
- worker liveness probes: pgrep isn't in the slim runtime image; check
/proc/1/cmdline instead (each worker execs its process as PID 1).
- UI: set HOSTNAME=0.0.0.0 so Next.js standalone doesn't bind to the k8s-injected
pod name (which maps to the pod IP only, breaking port-forward/loopback).
Verified end-to-end on EKS 1.36: all pods Ready, migrations applied (pgvector
extension + 27 tables), UI login page and web API served via port-forward.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(webhooks): durable retrying delivery for final webhooks
Final webhook nodes were fired inline with a single best-effort httpx POST
(run_integrations._execute_webhook_node). On a transient error the failure was
swallowed at three levels, so ARQ never retried and the final call report was
permanently lost -- leaving downstream receivers stuck (e.g. a CRM showing a
call as still "in conversation").
Replace the one-shot POST with a durable, idempotent delivery pipeline modelled
on the campaign retry pattern (persisted row + scheduled_for + bounded attempts):
- New webhook_deliveries table (WebhookDeliveryModel) is the source of truth.
Payload is rendered once and frozen so retries are deterministic; secrets are
not stored -- the credential is referenced by uuid and re-resolved at send time.
- run_integrations now persists a delivery row and enqueues deliver_webhook with
a deterministic ARQ job id instead of sending inline.
- deliver_webhook (new ARQ task) sends the request and:
* 2xx -> succeeded
* transient -> retry with capped exponential backoff (RequestError /
5xx / 408 / 425 / 429), up to max_attempts then dead_letter
* permanent 4xx -> dead_letter immediately (no pointless looping)
It is idempotent: a non-pending delivery is a no-op, so a duplicate enqueue or
sweeper re-injection can't double-send.
- sweep_webhook_deliveries cron (every 5 min) re-enqueues overdue pending
deliveries so nothing is lost to a worker restart / Redis flush.
- Stable X-Dograh-Delivery-Id / Workflow-Run-Id / Attempt headers let receivers
dedupe retried deliveries.
- enqueue_job now forwards ARQ job options (_job_id, _defer_by); failures log
repr(e) so empty-message errors like ConnectTimeout are diagnosable.
Config via DEFAULT_WEBHOOK_DELIVERY_CONFIG (env-overridable): max_attempts=5,
base_delay=30s, max_delay=600s, timeout=30s.
Tests cover payload rendering, persist+enqueue, success, transient retry,
retryable 5xx, permanent 4xx dead-letter, attempt exhaustion, and idempotency.
Migration verified to apply/rollback against Postgres; table/enum/indexes confirmed.
* fix(webhooks): atomic claim, safe success-recording, sweep paging, migration cleanup
Address review feedback on the webhook delivery pipeline:
- deliver_webhook now atomically claims a delivery (conditional UPDATE that
leases scheduled_for) before sending, so concurrent ARQ executions can't
double-send (the prior status=='pending' read was non-atomic).
- Recording success is moved out of the dead-letter try-block: if the receiver
accepted the payload (2xx) but the success DB-write fails, the row is left
pending for the sweeper to reconcile instead of being dead-lettered.
- The sweep keyset-paginates by id so a backlog over the page size is fully
drained, and logs the true re-enqueued total.
- Migration downgrade drops the enum via op.execute(DROP TYPE IF EXISTS ...)
instead of the deprecated op.get_bind().
* fix(webhooks): idempotent delivery creation and drop secret custom headers
Address the remaining review feedback:
- Add a (workflow_run_id, webhook_node_id) unique constraint and make
create_webhook_delivery a get-or-create returning (delivery, created). A
retried run_integrations now reuses the existing row instead of creating and
sending a duplicate final webhook; only a freshly-created row is enqueued.
- Stop persisting secret-looking custom headers (Authorization, X-API-Key,
Cookie, ...) in plaintext on the delivery row: they are dropped with a warning
pointing at the credential store (which is re-resolved securely at send time).
Non-secret custom headers are unaffected.
* fix(webhooks): harden idempotency key, secret-header match, sweep reclaim id
Address follow-up review feedback:
- webhook_node_id is now NOT NULL so a NULL can't slip past the
(workflow_run_id, webhook_node_id) unique constraint and create duplicates.
- Secret-header filtering matches normalized markers (auth/token/secret/cookie/
api-key/...) instead of an exact name list, catching variants like
X-Custom-Auth-Token while leaving benign headers (e.g. X-Idempotency-Key).
- The sweeper re-enqueues with a reclaim-specific job id (the lease timestamp)
so reconciling a delivered-but-unrecorded row isn't deduped against the
original attempt's already-completed ARQ job. The atomic claim still ensures
at most one send.
* fix(webhooks): scope delivery rows to workflow org
---------
Co-authored-by: Abhishek Kumar <abhishek@a6k.me>
* chore: drain active calls before rolling updates
* Use provisional VAD interruption strategy
* feat: wire provisional VAD configuration
* chore: refactor user turn strategies
* chore: bump pipecat
* fix: reject misrouted smallwebrtc runs on the telephony websocket
A smallwebrtc (browser/WebRTC) workflow run is established through the WebRTC
signaling endpoint, not the PSTN telephony websocket. When such a run reached
_handle_telephony_websocket it read no "provider" from initial_context and
closed with an opaque "Provider type not found". Detect smallwebrtc runs and
close with a clear reason pointing to the signaling endpoint, without setting
the run to running or invoking a telephony provider. Also store the provider on
smallwebrtc runs at creation so they are self-describing, and make the generic
no-provider close reason include the run id and mode.
Closes#433
* fix: merge workflow run initial context defaults
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Abhishek Kumar <abhishek@a6k.me>
* feat(twilio): add Answering Machine Detection (AMD) support via telephony config
Closes#339
* chore: regenerate OpenAPI spec to fix drift-check
The openapi.json snapshot had drifted from the FastAPI app definition
because main gained new organization endpoints (billing, credits,
context) after this branch was created. Regenerate it with
'python -m scripts.dump_docs_openapi' to bring it back in sync.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat: add provider-level AMD hooks
* fix: handle db error while persisting amd result
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Sabiha Khan <sabihak89@gmail.com>
Co-authored-by: Sabiha Khan <87858386+chewwbaka@users.noreply.github.com>
* fix: disable duplicate trigger nodes in workflow builder
AddNodePanel: disable trigger buttons and show tooltip when a trigger
already exists on the canvas, using bySpecName to identify trigger-
category specs from the live node list.
useWorkflowState: preflight in saveWorkflow rejects saves with multiple
trigger nodes via a sonner toast before the network request is made.
text_chat_session_service: include the original exception message in
TextChatSessionExecutionError so the HTTP 500 detail surfaces the root
cause without DB inspection.
Closes#378
* style: format test_text_chat_session_service.py with ruff
* chore: retrigger CI checks
* fix(workflow): enforce node instance constraints
---------
Co-authored-by: Abhishek Kumar <abhishek@a6k.me>
* fix(qa): tolerate non-dict JSON from QA LLM instead of crashing
parse_llm_json is explicitly designed to return a list when the model emits a
top-level JSON array (it has a dedicated test for that). The QA analyzers then
call parsed.get("tags", ...) directly on the result. When parsed is a list,
that raises AttributeError, which is NOT caught by the surrounding
except (json.JSONDecodeError, ValueError) — so a single stray array response
from the QA model crashed the entire QA analysis run instead of degrading to
empty results.
The live variable-extraction path already guards this exact case with an
isinstance(..., dict) check; mirror it in both QA analysis call sites
(_run_qa_analysis per-node and _run_whole_call_qa_analysis fallback) so a
non-dict parse result coerces to {} and the run produces empty defaults.
Adds a regression test that drives the whole-call analyzer with an array
response and asserts empty results rather than a crash.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(qa): log non-object QA JSON responses
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Abhishek Kumar <abhishek@a6k.me>
validate_trigger_paths used seen_paths.get(trigger_path) and treated a None
result as "path not seen yet". But None is also what node.get("id") returns
for a node without an id, so when the first trigger node sharing a path had no
id, it was stored as None and every later node with the same path was silently
accepted as unique — duplicate trigger paths slipped through validation.
Use a membership test (trigger_path not in seen_paths) so "first occurrence"
and "node_id happens to be None" are no longer conflated. Behavior is
unchanged for nodes that have ids.
Adds a regression test that fails before and passes after.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add model_options to SmallestAITTSConfiguration's voice field so the UI
renders the correct voice list per model — 15 standard voices for
lightning_v3.1 and 6 premium voices (meher, rhea, aviraj, cressida,
willow, maverick) for lightning_v3.1_pro. All 21 voice IDs verified
against the Smallest AI API. The frontend's existing model_options
machinery already handles dropdown filtering and auto-reset on model
change, so no UI changes are needed.
* fix: add language field to CartesiaTTSConfiguration and pass to TTS service
Closes#432
* chore: regenerate OpenAPI spec to fix drift-check
The openapi.json snapshot had drifted from the FastAPI app definition
because main gained new organization endpoints (billing, credits,
context) after this branch was created. Regenerate it with
'python -m scripts.dump_docs_openapi' to bring it back in sync.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore: clarify Cartesia language schema
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Abhishek Kumar <abhishek@a6k.me>
* Validate workflow status filter to prevent 500 on invalid enum value
The /workflow/fetch and /workflow/summary endpoints accepted a free-form
status query param and passed it straight into a query that casts to the
workflow_status PG enum (active/archived). Any other value — e.g. an
external caller passing 'published' (a workflow_definitions version state,
not a workflow status) — failed deep in Postgres as
InvalidTextRepresentationError, surfacing as an unhandled HTTP 500.
Add _validate_status_filter() to reject values outside WorkflowStatus with
a clean 422 before any DB query, for both the single and comma-separated
paths. Add route tests covering invalid, valid-single, comma-separated, and
mixed valid/invalid cases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: add tests
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
