dograh/deploy/helm/dograh
prabhatlepton 58cc9c8b1c
fix(web): honor X-Forwarded-Proto in uvicorn so request.url is https behind a reverse proxy (#515)
* fix(web): honor X-Forwarded-Proto in uvicorn so request.url is https behind a reverse proxy

## Problem

When Dograh runs behind a TLS-terminating reverse proxy (Cloudflare →
Traefik in Kubernetes, nginx in the docker-compose install), the inside
of the cluster/host is plain HTTP. Uvicorn defaults to trusting
`scope["scheme"]` from the socket, so `request.url.scheme` reads `http`
even though the client dialed `https`.

That breaks any code path that hashes or echoes the request URL back to
the caller. Concrete symptom seen in production: **Vobiz inbound webhook
signatures fail with "signature validation failed for vobiz"** because
Vobiz computes HMAC over the URL it dialed (`https://.../inbound/run`)
while Dograh recomputes it as `http://...`.  Log excerpt from the
failing call:

```
WARNING | provider.py | Vobiz webhook signature mismatch.
         Expected: daOpAZPm..., Got: 1+eW/RxE...
WARNING | telephony.py | /inbound/run: signature validation failed for vobiz
```

Twilio, Plivo and any other provider that signs over the callback URL
have the same failure mode when Dograh is deployed behind a proxy.

## Fix

Start uvicorn with `--proxy-headers --forwarded-allow-ips="*"` in
`scripts/run_web.sh`. Uvicorn rewrites `scope["scheme"]` and client
address from `X-Forwarded-Proto` / `X-Forwarded-For` when the request
originates from a trusted upstream — Traefik and Cloudflare set both
correctly, so `request.url.scheme == "https"` inside the app once again
and provider signature checks pass.

Verified end-to-end on a production k3s install (Traefik + Cloudflare
edge → dograh-web pod) — after the change, the very next Vobiz inbound
webhook validated successfully and the call connected past the previous
11-second signature-failure hangup.

* address review: let operators narrow FORWARDED_ALLOW_IPS

Both bot reviewers on #515 flagged `--forwarded-allow-ips="*"` as a
defence-in-depth concern: if uvicorn is directly reachable from an
untrusted network (bypassing the proxy), any client can spoof
`X-Forwarded-Proto` / `X-Forwarded-For`, and uvicorn will rewrite
`request.client` / `request.url` from those attacker-controlled headers.

Fix: consume `FORWARDED_ALLOW_IPS` from the environment (uvicorn already
recognizes this env var; see `deploy/hostinger/docker-compose.yaml:179`
for the existing precedent). Default stays `"*"` so the behavior of the
original fix is preserved for the standard docker-compose / helm layouts
where the app pod is only reachable via the proxy Service. Operators
who terminate uvicorn on a host that's also reachable directly can
narrow it to the proxy CIDR:

  FORWARDED_ALLOW_IPS="10.42.0.0/16" ./scripts/run_web.sh

* address review: declare FORWARDED_ALLOW_IPS in the helm chart, not the script

uvicorn already enables proxy-header handling by default and falls back to
the FORWARDED_ALLOW_IPS env var when --forwarded-allow-ips is absent, so the
CLI flags were redundant and the script-level "*" default hid a
security-relevant trust decision away from operators. Drop the flags, keep
run_web.sh deployment-agnostic, and declare the env var where the other
deployment config lives — web.forwardedAllowIps in values.yaml (default "*",
narrowable to a proxy CIDR) — mirroring how docker-compose already sets it
on the api service.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* simplify run_web.sh comment

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: prabhat pankaj <prabhatiitbhu@gmail.com>
Co-authored-by: Abhishek Kumar <abhishek@a6k.me>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 10:26:45 +05:30
..
examples feat(helm): add HPA for arq-worker + ui, ship a lean k3s prod example (#516) 2026-07-13 14:48:48 +05:30
templates fix(web): honor X-Forwarded-Proto in uvicorn so request.url is https behind a reverse proxy (#515) 2026-07-16 10:26:45 +05:30
.gitignore feat: add Helm chart for Kubernetes deployment (#365) 2026-07-03 12:39:39 +05:30
Chart.yaml feat: add Helm chart for Kubernetes deployment (#365) 2026-07-03 12:39:39 +05:30
README.md feat(helm): add HPA for arq-worker + ui, ship a lean k3s prod example (#516) 2026-07-13 14:48:48 +05:30
values.schema.json feat: add Helm chart for Kubernetes deployment (#365) 2026-07-03 12:39:39 +05:30
values.yaml fix(web): honor X-Forwarded-Proto in uvicorn so request.url is https behind a reverse proxy (#515) 2026-07-16 10:26:45 +05:30

Dograh Helm chart

Deploys Dograh on Kubernetes with decomposed backend workloads (web, ARQ workers, telephony singleton, campaign singleton), Next.js UI, and coturn for WebRTC media relay. Implements the architecture defined in HELM_DEPLOYMENT_PLAN.md at the repo root.

Status

v1, alpha. Validated with helm lint and helm template. Not yet exercised against a live cluster.

Quick start

cd deploy/helm/dograh

# Install with defaults (all internal deps, Gateway API exposure).
# The bundled Postgres/Redis/MinIO are in-chart manifests on official upstream
# images — no `helm dependency` / subchart pull step needed.
helm install dograh . \
  --set secrets.ossJwtSecret="$(openssl rand -hex 32)" \
  --set secrets.turnSecret="$(openssl rand -hex 32)" \
  --set exposure.gatewayApi.gatewayClassName=istio

See examples/values-single-node.yaml, examples/values-managed.yaml, and examples/values-aws.yaml for topology-specific overrides.

Architecture summary

Workload Replicas Strategy Notes
dograh-web 2 (HPA opt) RollingUpdate Long-lived WS, graceful drain
dograh-arq-worker 1 (HPA opt) RollingUpdate Stateless
dograh-ari-manager 1 fixed Recreate Telephony singleton
dograh-campaign-orchestrator 1 fixed Recreate Campaign singleton (in-memory locks)
dograh-ui 2 (HPA opt) RollingUpdate Next.js SSR
dograh-coturn 1 Recreate LoadBalancer Service, port-pinned

HTTP traffic: Gateway API (default) or Ingress (fallback). TURN traffic: dedicated L4 Service of type LoadBalancer.

Decisions log

These are choices the chart made where HELM_DEPLOYMENT_PLAN.md was silent. Each is exposed in values.yaml for operator override.

  • terminationGracePeriodSeconds for web: 600s. Covers a 10-minute call; tune to your call-length distribution.
  • preStop sleep: 15s. Conservative window for the gateway/ingress to observe pod NotReady and stop dispatching new connections.
  • Liveness probes on singletons: exec (pgrep). No HTTP endpoint exists on ari-manager / campaign-orchestrator; process-alive check is the simplest correct signal.
  • HPA on web / workers / ui: CPU(/memory), disabled by default. Templates exist for all three tiers (web-hpa.yaml, arq-worker-hpa.yaml, ui-hpa.yaml) but every autoscaling.<tier> block ships enabled: false — flip on per tier with a knowing eye. CPU/memory is a poor signal for the WS-heavy web tier and a coarse one for the IO-bound ARQ workers (plan: queue-depth / active-call scaling, see TODOs); it is a reasonable signal for the Next.js UI.
  • Singleton replica counts: hard-coded. No replicaCount knob exposed on ari-manager / campaign-orchestrator. Prevents accidental kubectl scale corrupting in-memory dedup state.
  • MinIO browser exposure: shared host, path prefix /voice-audio/. Mirrors current nginx behavior. Operators wanting a separate hostname can override by editing httproute-minio.yaml or ingress.yaml post-install.
  • NetworkPolicy: not in v1. TODO below.
  • ServiceMonitor / Prometheus: not in v1. TODO below.
  • TURN TLS (turns://): not in v1. Original docker-compose exposed port 5349 but never wired certs. Chart scopes v1 to plain TURN.

/tmp audit (review fix #6)

Resolved. End-of-call artifacts (recordings, transcript) are uploaded to object storage directly from the web process (api/services/workflow_run_artifacts.py) before the ARQ completion job is enqueued; the job carries only the workflow run id. No file handoff crosses a pod boundary, so web and arq-worker pods need no shared volume. The remaining /tmp uses (audio_file_cache.py, knowledge_base_processing.py) write and read within a single process.

Open TODOs (deferred from v1)

  • Leader election for singletons. Adopt Kubernetes lease-based leader election so ari-manager / campaign-orchestrator can run HA. Until then, replicas remain hard-coded to 1.
  • Connection-count HPA metric. Expose active WS sessions per pod (Prometheus or KEDA) and replace CPU/memory HPA target.
  • NetworkPolicy. Add default-deny + explicit egress to Postgres, Redis, MinIO/S3, and (for ari-manager) Asterisk.
  • ServiceMonitor. First-class Prometheus integration once observability stack is selected.
  • TURN TLS (turns://). Wire certificate paths through coturn config and document the cert-manager pattern.
  • MinIO public route via separate hostname. Make /voice-audio/ path-prefix the default but allow operators to opt into a dedicated hostname.
  • KEDA for ARQ workers. When a queue-depth or active-calls metric is available, switch ARQ from CPU HPA to KEDA-driven scaling. Keep autoscaling.workers.enabled=false when a KEDA ScaledObject owns the Deployment so the chart doesn't render a competing HPA.

Validation

cd deploy/helm/dograh

helm lint .
helm template test-release . > /tmp/render-default.yaml
helm template test-release . -f examples/values-single-node.yaml > /tmp/render-single.yaml
helm template test-release . -f examples/values-managed.yaml > /tmp/render-managed.yaml
helm template test-release . -f examples/values-aws.yaml > /tmp/render-aws.yaml
helm template test-release . -f examples/values-k3s-prod.yaml > /tmp/render-k3s.yaml

Spot-check expectations:

  • Deployment/<release>-ari-manager has replicas: 1 and strategy.type: Recreate.
  • Deployment/<release>-campaign-orchestrator has replicas: 1 and strategy.type: Recreate.
  • Deployment/<release>-web has terminationGracePeriodSeconds: 600 and a lifecycle.preStop exec hook.
  • Liveness probe on ari-manager / campaign-orchestrator uses exec, not httpGet.

Layout

deploy/helm/dograh/
├── Chart.yaml
├── values.yaml             # heavily commented
├── values.schema.json      # enforces mode enums
├── README.md               # this file
├── examples/
│   ├── values-single-node.yaml
│   ├── values-managed.yaml
│   ├── values-aws.yaml
│   └── values-k3s-prod.yaml
└── templates/
    ├── _helpers.tpl
    ├── NOTES.txt
    ├── serviceaccount.yaml
    ├── configmap.yaml
    ├── secret.yaml
    ├── migrate-job.yaml
    ├── web-deployment.yaml
    ├── web-service.yaml
    ├── web-hpa.yaml
    ├── web-pdb.yaml
    ├── arq-worker-deployment.yaml
    ├── arq-worker-hpa.yaml
    ├── ari-manager-deployment.yaml
    ├── campaign-orchestrator-deployment.yaml
    ├── ui-deployment.yaml
    ├── ui-service.yaml
    ├── ui-hpa.yaml
    ├── ui-pdb.yaml
    ├── coturn-deployment.yaml
    ├── coturn-service.yaml
    ├── coturn-configmap.yaml
    ├── gateway.yaml
    ├── httproute-api.yaml
    ├── httproute-ui.yaml
    ├── httproute-minio.yaml
    └── ingress.yaml