* fix(web): honor X-Forwarded-Proto in uvicorn so request.url is https behind a reverse proxy ## Problem When Dograh runs behind a TLS-terminating reverse proxy (Cloudflare → Traefik in Kubernetes, nginx in the docker-compose install), the inside of the cluster/host is plain HTTP. Uvicorn defaults to trusting `scope["scheme"]` from the socket, so `request.url.scheme` reads `http` even though the client dialed `https`. That breaks any code path that hashes or echoes the request URL back to the caller. Concrete symptom seen in production: **Vobiz inbound webhook signatures fail with "signature validation failed for vobiz"** because Vobiz computes HMAC over the URL it dialed (`https://.../inbound/run`) while Dograh recomputes it as `http://...`. Log excerpt from the failing call: ``` WARNING | provider.py | Vobiz webhook signature mismatch. Expected: daOpAZPm..., Got: 1+eW/RxE... WARNING | telephony.py | /inbound/run: signature validation failed for vobiz ``` Twilio, Plivo and any other provider that signs over the callback URL have the same failure mode when Dograh is deployed behind a proxy. ## Fix Start uvicorn with `--proxy-headers --forwarded-allow-ips="*"` in `scripts/run_web.sh`. Uvicorn rewrites `scope["scheme"]` and client address from `X-Forwarded-Proto` / `X-Forwarded-For` when the request originates from a trusted upstream — Traefik and Cloudflare set both correctly, so `request.url.scheme == "https"` inside the app once again and provider signature checks pass. Verified end-to-end on a production k3s install (Traefik + Cloudflare edge → dograh-web pod) — after the change, the very next Vobiz inbound webhook validated successfully and the call connected past the previous 11-second signature-failure hangup. * address review: let operators narrow FORWARDED_ALLOW_IPS Both bot reviewers on #515 flagged `--forwarded-allow-ips="*"` as a defence-in-depth concern: if uvicorn is directly reachable from an untrusted network (bypassing the proxy), any client can spoof `X-Forwarded-Proto` / `X-Forwarded-For`, and uvicorn will rewrite `request.client` / `request.url` from those attacker-controlled headers. Fix: consume `FORWARDED_ALLOW_IPS` from the environment (uvicorn already recognizes this env var; see `deploy/hostinger/docker-compose.yaml:179` for the existing precedent). Default stays `"*"` so the behavior of the original fix is preserved for the standard docker-compose / helm layouts where the app pod is only reachable via the proxy Service. Operators who terminate uvicorn on a host that's also reachable directly can narrow it to the proxy CIDR: FORWARDED_ALLOW_IPS="10.42.0.0/16" ./scripts/run_web.sh * address review: declare FORWARDED_ALLOW_IPS in the helm chart, not the script uvicorn already enables proxy-header handling by default and falls back to the FORWARDED_ALLOW_IPS env var when --forwarded-allow-ips is absent, so the CLI flags were redundant and the script-level "*" default hid a security-relevant trust decision away from operators. Drop the flags, keep run_web.sh deployment-agnostic, and declare the env var where the other deployment config lives — web.forwardedAllowIps in values.yaml (default "*", narrowable to a proxy CIDR) — mirroring how docker-compose already sets it on the api service. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * simplify run_web.sh comment Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: prabhat pankaj <prabhatiitbhu@gmail.com> Co-authored-by: Abhishek Kumar <abhishek@a6k.me> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| examples | ||
| templates | ||
| .gitignore | ||
| Chart.yaml | ||
| README.md | ||
| values.schema.json | ||
| values.yaml | ||
Dograh Helm chart
Deploys Dograh on Kubernetes with decomposed backend workloads (web,
ARQ workers, telephony singleton, campaign singleton), Next.js UI, and
coturn for WebRTC media relay. Implements the architecture defined in
HELM_DEPLOYMENT_PLAN.md at the repo root.
Status
v1, alpha. Validated with helm lint and helm template. Not yet
exercised against a live cluster.
Quick start
cd deploy/helm/dograh
# Install with defaults (all internal deps, Gateway API exposure).
# The bundled Postgres/Redis/MinIO are in-chart manifests on official upstream
# images — no `helm dependency` / subchart pull step needed.
helm install dograh . \
--set secrets.ossJwtSecret="$(openssl rand -hex 32)" \
--set secrets.turnSecret="$(openssl rand -hex 32)" \
--set exposure.gatewayApi.gatewayClassName=istio
See examples/values-single-node.yaml, examples/values-managed.yaml,
and examples/values-aws.yaml for topology-specific overrides.
Architecture summary
| Workload | Replicas | Strategy | Notes |
|---|---|---|---|
dograh-web |
2 (HPA opt) | RollingUpdate | Long-lived WS, graceful drain |
dograh-arq-worker |
1 (HPA opt) | RollingUpdate | Stateless |
dograh-ari-manager |
1 fixed | Recreate | Telephony singleton |
dograh-campaign-orchestrator |
1 fixed | Recreate | Campaign singleton (in-memory locks) |
dograh-ui |
2 (HPA opt) | RollingUpdate | Next.js SSR |
dograh-coturn |
1 | Recreate | LoadBalancer Service, port-pinned |
HTTP traffic: Gateway API (default) or Ingress (fallback).
TURN traffic: dedicated L4 Service of type LoadBalancer.
Decisions log
These are choices the chart made where HELM_DEPLOYMENT_PLAN.md was
silent. Each is exposed in values.yaml for operator override.
- terminationGracePeriodSeconds for web: 600s. Covers a 10-minute call; tune to your call-length distribution.
- preStop sleep: 15s. Conservative window for the gateway/ingress to observe pod NotReady and stop dispatching new connections.
- Liveness probes on singletons:
exec(pgrep). No HTTP endpoint exists on ari-manager / campaign-orchestrator; process-alive check is the simplest correct signal. - HPA on web / workers / ui: CPU(/memory), disabled by default.
Templates exist for all three tiers (
web-hpa.yaml,arq-worker-hpa.yaml,ui-hpa.yaml) but everyautoscaling.<tier>block shipsenabled: false— flip on per tier with a knowing eye. CPU/memory is a poor signal for the WS-heavy web tier and a coarse one for the IO-bound ARQ workers (plan: queue-depth / active-call scaling, see TODOs); it is a reasonable signal for the Next.js UI. - Singleton replica counts: hard-coded. No
replicaCountknob exposed on ari-manager / campaign-orchestrator. Prevents accidentalkubectl scalecorrupting in-memory dedup state. - MinIO browser exposure: shared host, path prefix
/voice-audio/. Mirrors current nginx behavior. Operators wanting a separate hostname can override by editinghttproute-minio.yamloringress.yamlpost-install. - NetworkPolicy: not in v1. TODO below.
- ServiceMonitor / Prometheus: not in v1. TODO below.
- TURN TLS (turns://): not in v1. Original docker-compose exposed port 5349 but never wired certs. Chart scopes v1 to plain TURN.
/tmp audit (review fix #6)
Resolved. End-of-call artifacts (recordings, transcript) are uploaded to
object storage directly from the web process
(api/services/workflow_run_artifacts.py) before the ARQ completion job
is enqueued; the job carries only the workflow run id. No file handoff
crosses a pod boundary, so web and arq-worker pods need no shared
volume. The remaining /tmp uses (audio_file_cache.py,
knowledge_base_processing.py) write and read within a single process.
Open TODOs (deferred from v1)
- Leader election for singletons. Adopt Kubernetes lease-based
leader election so
ari-manager/campaign-orchestratorcan run HA. Until then, replicas remain hard-coded to 1. - Connection-count HPA metric. Expose active WS sessions per pod (Prometheus or KEDA) and replace CPU/memory HPA target.
- NetworkPolicy. Add default-deny + explicit egress to Postgres, Redis, MinIO/S3, and (for ari-manager) Asterisk.
- ServiceMonitor. First-class Prometheus integration once observability stack is selected.
- TURN TLS (turns://). Wire certificate paths through coturn config and document the cert-manager pattern.
- MinIO public route via separate hostname. Make
/voice-audio/path-prefix the default but allow operators to opt into a dedicated hostname. - KEDA for ARQ workers. When a queue-depth or active-calls metric
is available, switch ARQ from CPU HPA to KEDA-driven scaling. Keep
autoscaling.workers.enabled=falsewhen a KEDA ScaledObject owns the Deployment so the chart doesn't render a competing HPA.
Validation
cd deploy/helm/dograh
helm lint .
helm template test-release . > /tmp/render-default.yaml
helm template test-release . -f examples/values-single-node.yaml > /tmp/render-single.yaml
helm template test-release . -f examples/values-managed.yaml > /tmp/render-managed.yaml
helm template test-release . -f examples/values-aws.yaml > /tmp/render-aws.yaml
helm template test-release . -f examples/values-k3s-prod.yaml > /tmp/render-k3s.yaml
Spot-check expectations:
Deployment/<release>-ari-managerhasreplicas: 1andstrategy.type: Recreate.Deployment/<release>-campaign-orchestratorhasreplicas: 1andstrategy.type: Recreate.Deployment/<release>-webhasterminationGracePeriodSeconds: 600and alifecycle.preStopexec hook.- Liveness probe on ari-manager / campaign-orchestrator uses
exec, nothttpGet.
Layout
deploy/helm/dograh/
├── Chart.yaml
├── values.yaml # heavily commented
├── values.schema.json # enforces mode enums
├── README.md # this file
├── examples/
│ ├── values-single-node.yaml
│ ├── values-managed.yaml
│ ├── values-aws.yaml
│ └── values-k3s-prod.yaml
└── templates/
├── _helpers.tpl
├── NOTES.txt
├── serviceaccount.yaml
├── configmap.yaml
├── secret.yaml
├── migrate-job.yaml
├── web-deployment.yaml
├── web-service.yaml
├── web-hpa.yaml
├── web-pdb.yaml
├── arq-worker-deployment.yaml
├── arq-worker-hpa.yaml
├── ari-manager-deployment.yaml
├── campaign-orchestrator-deployment.yaml
├── ui-deployment.yaml
├── ui-service.yaml
├── ui-hpa.yaml
├── ui-pdb.yaml
├── coturn-deployment.yaml
├── coturn-service.yaml
├── coturn-configmap.yaml
├── gateway.yaml
├── httproute-api.yaml
├── httproute-ui.yaml
├── httproute-minio.yaml
└── ingress.yaml