apunkt/plano
1
0
Fork 0
mirror of https://github.com/katanemo/plano.git synced 2026-05-10 00:02:43 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

support session pinning for consistent model selection in routing (#813)

Browse source
This commit is contained in:
Adil Hafeez 2026-03-13 17:32:32 -07:00
parent 785bf7e021
commit 46a5bfd82d
No known key found for this signature in database
GPG key ID: 9B18EF7691369645
8 changed files with 406 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

85
demos/llm_routing/model_routing_service/README.md
View file

@ -55,6 +55,63 @@ Response:
The response tells you which model would handle this request and which route was matched, without actually making the LLM call.
## Session Pinning
Send an `X-Session-Id` header to pin the routing decision for a session. Once a model is selected, all subsequent requests with the same session ID return the same model without re-running routing.
```bash
# First call — runs routing, caches result
curl http://localhost:12000/routing/v1/chat/completions \
-H "Content-Type: application/json" \
-H "X-Session-Id: my-session-123" \
-d '{
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "Write a Python function for binary search"}]
}'
```
Response (first call):
```json
{
"model": "anthropic/claude-sonnet-4-20250514",
"route": "code_generation",
"trace_id": "c16d1096c1af4a17abb48fb182918a88",
"session_id": "my-session-123",
"pinned": false
}
```
```bash
# Second call — same session, returns cached result
curl http://localhost:12000/routing/v1/chat/completions \
-H "Content-Type: application/json" \
-H "X-Session-Id: my-session-123" \
-d '{
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "Now explain merge sort"}]
}'
```
Response (pinned):
```json
{
"model": "anthropic/claude-sonnet-4-20250514",
"route": "code_generation",
"trace_id": "a1b2c3d4e5f6...",
"session_id": "my-session-123",
"pinned": true
}
```
Session TTL and max cache size are configurable in `config.yaml`:
```yaml
routing:
session_ttl_seconds: 600 # default: 600 (10 minutes)
session_max_entries: 10000 # default: 10000
```
Without the `X-Session-Id` header, routing runs fresh every time (no breaking change).
## Demo Output
```
@ -88,5 +145,33 @@ The response tells you which model would handle this request and which route was
"trace_id": "26be822bbdf14a3ba19fe198e55ea4a9"
}
--- 7. Session pinning - first call (fresh routing decision) ---
{
"model": "anthropic/claude-sonnet-4-20250514",
"route": "code_generation",
"trace_id": "f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6",
"session_id": "demo-session-001",
"pinned": false
}
--- 8. Session pinning - second call (same session, pinned) ---
Notice: same model returned with "pinned": true, routing was skipped
{
"model": "anthropic/claude-sonnet-4-20250514",
"route": "code_generation",
"trace_id": "a9b8c7d6e5f4a3b2c1d0e9f8a7b6c5d4",
"session_id": "demo-session-001",
"pinned": true
}
--- 9. Different session gets its own fresh routing ---
{
"model": "openai/gpt-4o",
"route": "complex_reasoning",
"trace_id": "1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d",
"session_id": "demo-session-002",
"pinned": false
}
=== Demo Complete ===
```

43
demos/llm_routing/model_routing_service/demo.sh
View file

@ -117,4 +117,47 @@ curl -s "$PLANO_URL/routing/v1/messages" \
}' | python3 -m json.tool
echo ""
# --- Example 7: Session pinning - first call (fresh routing) ---
echo "--- 7. Session pinning - first call (fresh routing decision) ---"
echo ""
curl -s "$PLANO_URL/routing/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "X-Session-Id: demo-session-001" \
-d '{
"model": "gpt-4o-mini",
"messages": [
{"role": "user", "content": "Write a Python function that implements binary search on a sorted array"}
]
}' | python3 -m json.tool
echo ""
# --- Example 8: Session pinning - second call (pinned result) ---
echo "--- 8. Session pinning - second call (same session, pinned) ---"
echo " Notice: same model returned with \"pinned\": true, routing was skipped"
echo ""
curl -s "$PLANO_URL/routing/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "X-Session-Id: demo-session-001" \
-d '{
"model": "gpt-4o-mini",
"messages": [
{"role": "user", "content": "Now explain how merge sort works and when to prefer it over quicksort"}
]
}' | python3 -m json.tool
echo ""
# --- Example 9: Different session gets fresh routing ---
echo "--- 9. Different session gets its own fresh routing ---"
echo ""
curl -s "$PLANO_URL/routing/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "X-Session-Id: demo-session-002" \
-d '{
"model": "gpt-4o-mini",
"messages": [
{"role": "user", "content": "Explain the trade-offs between microservices and monolithic architectures"}
]
}' | python3 -m json.tool
echo ""
echo "=== Demo Complete ==="