apunkt/plano
1
0
Fork 0
mirror of https://github.com/katanemo/plano.git synced 2026-05-08 23:32:43 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

rename session pinning to model affinity with x-model-affinity header

Browse source
This commit is contained in:
Adil Hafeez 2026-04-08 15:23:53 -07:00
parent 5789694d2f
commit da9792c2dd
14 changed files with 468 additions and 371 deletions
Download patch file Download diff file Expand all files Collapse all files

8
demos/llm_routing/model_routing_service/README.md
View file

@ -108,13 +108,13 @@ The response contains the model list — your client should try `models[0]` firs
## Session Pinning
Send an `X-Routing-Session-Id` header to pin the routing decision for a session. Once a model is selected, all subsequent requests with the same session ID return the same model without re-running routing.
Send an `X-Model-Affinity` header to pin the routing decision for a session. Once a model is selected, all subsequent requests with the same session ID return the same model without re-running routing.
```bash
# First call — runs routing, caches result
curl http://localhost:12000/routing/v1/chat/completions \
-H "Content-Type: application/json" \
-H "X-Routing-Session-Id: my-session-123" \
-H "X-Model-Affinity: my-session-123" \
-d '{
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "Write a Python function for binary search"}]
@ -136,7 +136,7 @@ Response (first call):
# Second call — same session, returns cached result
curl http://localhost:12000/routing/v1/chat/completions \
-H "Content-Type: application/json" \
-H "X-Routing-Session-Id: my-session-123" \
-H "X-Model-Affinity: my-session-123" \
-d '{
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "Now explain merge sort"}]
@ -161,7 +161,7 @@ routing:
session_max_entries: 10000 # default: 10000
```
Without the `X-Routing-Session-Id` header, routing runs fresh every time (no breaking change).
Without the `X-Model-Affinity` header, routing runs fresh every time (no breaking change).
## Kubernetes Deployment (Self-hosted Arch-Router on GPU)