* feat(helm): add HPA for arq-worker + ui, ship a lean k3s prod example
## Problem
The chart's autoscaling story only covers the `web` tier — one
`web-hpa.yaml` template gated by `autoscaling.web.enabled`. Operators
scaling the `arq-worker` (background jobs) or `ui` (Next.js SSR) tiers
have to write their own HPA manifests out-of-band or fork the chart.
Turning the existing memory-utilization target on for freshly-installed
workloads also silently breaks: idle Python at the chart's default
`128Mi` (workers) / `256Mi` (ui) memory request already sits above
`80%`, so HPA scales every tier to `maxReplicas` on cold start with no
traffic. On a tight node this cascades into "insufficient CPU" and
blocks new-workload scheduling.
## Fix
**New HPA templates** — `templates/arq-worker-hpa.yaml` and
`templates/ui-hpa.yaml`, both mirroring the existing
`templates/web-hpa.yaml` shape (autoscaling/v2, resource metrics,
gated on `.Values.autoscaling.<tier>.enabled`).
**Extended `values.yaml`**:
- `autoscaling.workers` and `autoscaling.ui` blocks with sane defaults
(`enabled: true`, `minReplicas: 1`, `maxReplicas: 5`,
`targetCPUUtilizationPercentage: 70`).
- `targetMemoryUtilizationPercentage: null` on both tiers by default,
with an inline comment explaining why memory-utilization HPA is a
broken signal at the chart's default request sizes.
- Header comment reworked to (a) document the `metrics-server`
requirement, (b) note that HPA takes ownership of Deployment
`replicas` after first sync, (c) call out that CPU is a poor signal
for the web tier (long-lived WebSockets), and (d) note that CPU is
a fine signal for workers and ui.
**Example**: `examples/values-k3s-prod.yaml` — a single-node k3s
production override that exercises the new HPA blocks and demonstrates
the paired safety changes (memory targets nulled, sized resource
requests, migration job CPU sized for a tight node). Ship-ready
starting point for the operator flow: hosted-AI only (no local
models), all state on the node's local-path StorageClass, invite-only
signup, TLS terminated at a shared Cloudflare Origin cert.
## Behavior
Fresh install with defaults:
- Workers scale 1 → 5 on CPU 70% target only. No memory-based
scale-up storm on cold start.
- UI scales 1 → 5 on CPU 70% target only.
- Web autoscaling stays `enabled: false` by default (unchanged) —
operators opt in per the existing README warning.
Operators who want memory-based HPA back can:
1. Bump `workers.resources.requests.memory` (~256Mi) or
`ui.resources.requests.memory` (~384Mi).
2. Set `autoscaling.<tier>.targetMemoryUtilizationPercentage: 80`.
* address review: omit replicas when HPA on, suppress empty-metrics HPA, docs
Fixes raised on #516:
- **Worker/UI Replicas Reset On Upgrade** — arq-worker-deployment.yaml and
ui-deployment.yaml now wrap `replicas:` in `{{- if not .Values.autoscaling.<tier>.enabled }}`,
mirroring the existing web-deployment guard. With HPA on, Helm no longer
reapplies the static replicaCount on upgrade and briefly shrink an
HPA-scaled pool.
- **Empty Metrics Render Invalid HPA** — arq-worker-hpa.yaml and ui-hpa.yaml
now short-circuit the whole HPA object when both CPU and memory targets
are null. Previously the template emitted `spec.metrics:` with no items
(rejected by the k8s API server).
- **`enableSignup: false` removed from examples/values-k3s-prod.yaml** — that
knob depends on #514 which hasn't landed; unwiring it here avoids
suggesting a lockdown that isn't in effect until the sibling PR merges.
- **Header comment mismatch** — `# HPA: 1 → 5 on CPU 70% / memory 80%` claimed
memory was on while every tier had `targetMemoryUtilizationPercentage: null`.
Updated to "CPU 70% only (memory HPA opt-in)".
- **Wrong default in comment** — `values.yaml` said workers default is `128Mi`;
actual is `256Mi`. Fixed.
- **UI comment said "idle Python"** — UI is Next.js/Node.js. Corrected on the
UI HPA memory comment and the per-tier comments in values-k3s-prod.yaml
(web: FastAPI, workers: Python/ARQ, ui: Node.js).
All lints pass; verified with `helm template`:
- Defaults render both HPAs and Deployments without static `replicas:`.
- `--set autoscaling.workers.targetCPUUtilizationPercentage=null --set autoscaling.workers.targetMemoryUtilizationPercentage=null`
renders only the Deployment (HPA suppressed).
- `--set autoscaling.workers.enabled=false` renders the Deployment with
static `replicas:` restored.
* address review: align Deployment replicas gate with HPA render gate
Follow-up on #516: my earlier fix guarded `spec.replicas` on only
`autoscaling.<tier>.enabled`, but the HPA-empty-metrics guard I added
suppresses the HPA object when both metric targets are null while
`enabled: true`. That combination produced a Deployment with neither
a `spec.replicas` value nor an HPA owner — a k8s Deployment defaults
to `replicas: 1` in that case, but the chart no longer expresses intent.
Fix: the Deployment `replicas` gate now mirrors the HPA render gate
exactly. Rendered outcomes verified with `helm template`:
| autoscaling.<tier> | HPA rendered? | Deployment replicas? |
|-------------------------------|---------------|----------------------|
| enabled: true, target set | yes | omitted (HPA owns) |
| enabled: true, both null | no | static (kept) |
| enabled: false | no | static (kept) |
* fix(helm): default worker/ui autoscaling off; ui HPA floor of 2
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(helm): align web replicas/HPA gate with worker/ui pattern
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(helm): document worker/ui HPAs in README; polish k3s example
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: prabhat pankaj <prabhatiitbhu@gmail.com>
Co-authored-by: Abhishek Kumar <abhishek@a6k.me>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(auth): gate OSS signup behind ENABLE_SIGNUP flag
## Problem
The `POST /api/v1/auth/signup` endpoint is unconditionally exposed on
every OSS install. Operators running an invite-only deployment (private
customer instances, staging environments, internal-only tenants) have
no way to disable public account creation without patching the codebase.
The UI also shows the "Sign up" link on `/auth/login` regardless of
whether signup is available, so a locked-down deployment leaves broken
navigation on the login page.
## Fix
Introduce a single `ENABLE_SIGNUP` env var (default `true` — no behavior
change for existing installs) that controls signup end-to-end:
- **Backend** — `api/constants.ENABLE_SIGNUP` is read at module load.
The signup handler returns 403 when it's false. Also exposed on
`GET /api/v1/health` as `signup_enabled: bool` so the UI can mirror
the operator's choice at runtime instead of at bundle-build time.
- **UI** — `getSignupEnabled()` in `lib/auth/config.ts` proxies the
health field, `/api/config/auth` surfaces it to the browser, the
login page conditionally renders the "Sign up" link via a one-shot
`fetch("/api/config/auth")` in `useEffect`, and the middleware
redirects `/auth/signup` → `/auth/login` when disabled (fires before
Next.js can serve the statically-prerendered signup page).
- **Helm** — `config.enableSignup` (default `true`) is rendered into
the ConfigMap as `ENABLE_SIGNUP` so operators can flip it via
`--set config.enableSignup=false` at install/upgrade time.
Fallbacks default to `signupEnabled: true` in every layer so a fresh
install "just works" and matches the backend default.
* address review: rollout on ConfigMap change, cache TTL, no signup-link flash
Four review points on #514:
**P1 — ConfigMap Change Skips Rollout** (`configmap.yaml`). `helm upgrade
--set config.enableSignup=false` updated the ConfigMap but did NOT roll
the api pods, so running processes kept the ENABLE_SIGNUP env from
startup and continued serving the old signup behavior — including
divergence between replicas mid-upgrade.
Fix: add the standard `checksum/config` pod-template annotation on the
four backend Deployments that `envFrom` the ConfigMap (`web`,
`arq-worker`, `ari-manager`, `campaign-orchestrator`). Verified with
`helm template`: all four Deployments share the same checksum on any
given render, and flipping `config.enableSignup` changes the checksum
uniformly so kubectl sees a pod-template diff and rolls all four.
**P1 — Signup Flag Stays Cached (server)** (`ui/src/lib/auth/config.ts`).
Module-scoped cache had no TTL. `revalidate: 300` was passed on the
underlying `fetch()` but the in-memory short-circuit above ran first, so
the value never refreshed until the UI pod restarted.
Fix: add `AUTH_CONFIG_TTL_MS = 5 * 60 * 1000` (matching the fetch
revalidate hint) so the module cache and the Next fetch cache stay in
sync. Backend flag flips propagate within 5 minutes without a pod
restart.
**P1 — Middleware Redirect Uses Stale State** (`ui/src/middleware.ts`).
Same shape as above — a separate module cache with no expiry could keep
redirecting `/auth/signup → /auth/login` after signup was re-enabled, or
keep serving the statically-prerendered signup page after lockdown.
Fix: same `SERVER_CONFIG_TTL_MS = 5 * 60 * 1000` TTL on the middleware
cache.
**P2 — Signup link flash on login page** (`ui/src/app/auth/login/page.tsx`).
Initial `signupEnabled` state was `null`, so `{signupEnabled && ...}`
hid the link on first paint and it popped in after the fetch resolved
— a CLS on every login-page load on stock installs where signup is
enabled.
Fix: initialise the state to `true` (matches the backend default). The
fetch still overrides to `false` when the operator has actually
disabled signup, so the lockdown UI behavior is unchanged; only the
happy-path flash is gone.
* simplify signup flag: drop TTL caches and middleware redirect
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* resolve signup flag server-side to avoid signup link flicker
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: prabhat pankaj <prabhatiitbhu@gmail.com>
Co-authored-by: Abhishek Kumar <abhishek@a6k.me>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat: add Helm chart for Kubernetes deployment
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Replace bundled Bitnami subcharts with in-chart manifests on official images
The Bitnami catalog removed all versioned image tags from docker.io/bitnami in
Aug 2025 (old images frozen in bitnamilegacy, maintained catalog now behind a
Broadcom subscription), so the bundled postgresql/redis/minio subcharts no
longer pull. Replace them with plain in-chart manifests built on official
upstream images, keeping the internal/all-in-one path fully self-contained and
free of third-party chart packaging that can disappear:
- internal-postgres.yaml: pgvector/pgvector:pg17 — upstream Postgres plus the
`vector` extension the migrations require. POSTGRES_USER=dograh is the initdb
superuser, so CREATE EXTENSION vector succeeds.
- internal-redis.yaml: redis:7.4-alpine, password-protected, AOF persistence.
- internal-minio.yaml: minio/minio, root creds shared with the app via a single
secret (can't drift); the app auto-creates its bucket.
Service/secret names are unchanged (<rel>-postgresql, <rel>-redisinternal-master,
<rel>-minio) so the app wiring is untouched. Dep passwords are generated once and
persisted across upgrades via lookup. Drop the Chart.yaml dependencies,
Chart.lock, and the `helm dependency` step; the internal manifests gate on the
mode toggles (database.mode=internal, etc.).
Also fixes surfaced by smoke-testing on a live EKS cluster:
- Dockerfile: ship the per-service run_*.sh entrypoints the chart invokes.
- migrate-job: run as a post-install/pre-upgrade hook (the bundled Postgres does
not exist during pre-install) with a wait-for-postgres init container.
- backend env: declare POSTGRES_PASSWORD/REDIS_PASSWORD before the DATABASE_URL/
REDIS_URL that interpolate them (Kubernetes only expands back-references).
- worker liveness probes: pgrep isn't in the slim runtime image; check
/proc/1/cmdline instead (each worker execs its process as PID 1).
- UI: set HOSTNAME=0.0.0.0 so Next.js standalone doesn't bind to the k8s-injected
pod name (which maps to the pod IP only, breaking port-forward/loopback).
Verified end-to-end on EKS 1.36: all pods Ready, migrations applied (pgvector
extension + 27 tables), UI login page and web API served via port-forward.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
