Model affinity for consistent model selection in agentic loops (#827)

demos/llm_routing/model_routing_service/README.md

@@ -106,6 +106,63 @@ Response:
 
 The response contains the model list — your client should try `models[0]` first and fall back to `models[1]` on 429 or 5xx errors.
 
+## Session Pinning
+
+Send an `X-Model-Affinity` header to pin the routing decision for a session. Once a model is selected, all subsequent requests with the same session ID return the same model without re-running routing.
+
+```bash
+# First call — runs routing, caches result
+curl http://localhost:12000/routing/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "X-Model-Affinity: my-session-123" \
+  -d '{
+    "model": "gpt-4o-mini",
+    "messages": [{"role": "user", "content": "Write a Python function for binary search"}]
+  }'
+```
+
+Response (first call):
+```json
+{
+    "model": "anthropic/claude-sonnet-4-20250514",
+    "route": "code_generation",
+    "trace_id": "c16d1096c1af4a17abb48fb182918a88",
+    "session_id": "my-session-123",
+    "pinned": false
+}
+```
+
+```bash
+# Second call — same session, returns cached result
+curl http://localhost:12000/routing/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "X-Model-Affinity: my-session-123" \
+  -d '{
+    "model": "gpt-4o-mini",
+    "messages": [{"role": "user", "content": "Now explain merge sort"}]
+  }'
+```
+
+Response (pinned):
+```json
+{
+    "model": "anthropic/claude-sonnet-4-20250514",
+    "route": "code_generation",
+    "trace_id": "a1b2c3d4e5f6...",
+    "session_id": "my-session-123",
+    "pinned": true
+}
+```
+
+Session TTL and max cache size are configurable in `config.yaml`:
+```yaml
+routing:
+  session_ttl_seconds: 600      # default: 600 (10 minutes)
+  session_max_entries: 10000    # default: 10000
+```
+
+Without the `X-Model-Affinity` header, routing runs fresh every time (no breaking change).
+
 ## Kubernetes Deployment (Self-hosted Arch-Router on GPU)
 
 To run Arch-Router in-cluster using vLLM instead of the default hosted endpoint:
@@ -167,3 +224,68 @@ kubectl create configmap plano-config \
   --dry-run=client -o yaml | kubectl apply -f -
 kubectl rollout restart deployment/plano
 ```
+
+
+## Demo Output
+
+```
+=== Model Routing Service Demo ===
+
+--- 1. Code generation query (OpenAI format) ---
+{
+    "models": ["anthropic/claude-sonnet-4-20250514", "openai/gpt-4o"],
+    "route": "code_generation",
+    "trace_id": "c16d1096c1af4a17abb48fb182918a88"
+}
+
+--- 2. Complex reasoning query (OpenAI format) ---
+{
+    "models": ["openai/gpt-4o", "openai/gpt-4o-mini"],
+    "route": "complex_reasoning",
+    "trace_id": "30795e228aff4d7696f082ed01b75ad4"
+}
+
+--- 3. Simple query - no routing match (OpenAI format) ---
+{
+    "models": ["none"],
+    "route": null,
+    "trace_id": "ae0b6c3b220d499fb5298ac63f4eac0e"
+}
+
+--- 4. Code generation query (Anthropic format) ---
+{
+    "models": ["anthropic/claude-sonnet-4-20250514", "openai/gpt-4o"],
+    "route": "code_generation",
+    "trace_id": "26be822bbdf14a3ba19fe198e55ea4a9"
+}
+
+--- 7. Session pinning - first call (fresh routing decision) ---
+{
+    "models": ["anthropic/claude-sonnet-4-20250514", "openai/gpt-4o"],
+    "route": "code_generation",
+    "trace_id": "f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6",
+    "session_id": "demo-session-001",
+    "pinned": false
+}
+
+--- 8. Session pinning - second call (same session, pinned) ---
+    Notice: same model returned with "pinned": true, routing was skipped
+{
+    "model": "anthropic/claude-sonnet-4-20250514",
+    "route": "code_generation",
+    "trace_id": "a9b8c7d6e5f4a3b2c1d0e9f8a7b6c5d4",
+    "session_id": "demo-session-001",
+    "pinned": true
+}
+
+--- 9. Different session gets its own fresh routing ---
+{
+    "models": ["openai/gpt-4o", "openai/gpt-4o-mini"],
+    "route": "complex_reasoning",
+    "trace_id": "1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d",
+    "session_id": "demo-session-002",
+    "pinned": false
+}
+
+=== Demo Complete ===
+```