Redis-backed session cache for cross-replica model affinity (#879)

Browse source

* add pluggable session cache with Redis backend

* add Redis session affinity demos (Docker Compose and Kubernetes)

* address PR review feedback on session cache

* document Redis session cache backend for model affinity

* sync rendered config reference with session_cache addition

* add tenant-scoped Redis session cache keys and remove dead log_affinity_hit

- Add tenant_header to SessionCacheConfig; when set, cache keys are scoped
  as plano:affinity:{tenant_id}:{session_id} for multi-tenant isolation
- Thread tenant_id through RouterService, routing_service, and llm handlers
- Use Cow<'_, str> in session_key to avoid allocation when no tenant is set
- Remove unused log_affinity_hit (logging was already inlined at call sites)

* remove session_affinity_redis and session_affinity_redis_k8s demos

...

This commit is contained in:

Musa 2026-04-13 19:30:47 -07:00 • committed by GitHub

parent 128059e7c1

commit 980faef6be

No known key found for this signature in database

GPG key ID: B5690EEEBB952194

15 changed files with 1538 additions and 729 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download patch file Download diff file Expand all files Collapse all files

1757

crates/Cargo.lock generated

View file

File diff suppressed because it is too large Load diff