apunkt/plano
1
0
Fork 0
mirror of https://github.com/katanemo/plano.git synced 2026-05-27 14:17:15 +02:00
Code Issues Projects Releases Packages Wiki Activity

model routing: cost/latency ranking with ranked fallback list (#849)

Browse source
This commit is contained in:
Adil Hafeez 2026-03-30 13:46:52 -07:00 • committed by GitHub
parent 3a531ce22a
commit e5751d6b13
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
23 changed files with 1524 additions and 317 deletions
Download patch file Download diff file Expand all files Collapse all files

245
docs/routing-api.md Normal file
View file

@ -0,0 +1,245 @@
# Plano Routing API — Request & Response Format
## Overview
Plano intercepts LLM requests and routes them to the best available model based on semantic intent and live cost/latency data. The developer sends a standard OpenAI-compatible request with an optional `routing_preferences` field. Plano returns an ordered list of candidate models; the client uses the first and falls back to the next on 429 or 5xx errors.
---
## Request Format
Standard OpenAI chat completion body. The only addition is the optional `routing_preferences` field, which is stripped before the request is forwarded upstream.
```json
POST /v1/chat/completions
{
"model": "openai/gpt-4o-mini",
"messages": [
{"role": "user", "content": "write a sorting algorithm in Python"}
],
"routing_preferences": [
{
"name": "code generation",
"description": "generating new code snippets",
"models": ["anthropic/claude-sonnet-4-20250514", "openai/gpt-4o", "openai/gpt-4o-mini"],
"selection_policy": {"prefer": "fastest"}
},
{
"name": "general questions",
"description": "casual conversation and simple queries",
"models": ["openai/gpt-4o-mini"],
"selection_policy": {"prefer": "cheapest"}
}
]
}
```
### `routing_preferences` fields
| Field | Type | Required | Description |
|---|---|---|---|
| `name` | string | yes | Route identifier. Must match the LLM router's route classification. |
| `description` | string | yes | Natural language description used by the router to match user intent. |
| `models` | string[] | yes | Ordered candidate pool. At least one entry required. Must be declared in `model_providers`. |
| `selection_policy.prefer` | enum | yes | How to rank models: `cheapest`, `fastest`, or `none`. |
### `selection_policy.prefer` values
| Value | Behavior |
|---|---|
| `cheapest` | Sort by ascending cost from the metrics endpoint. Models with no data appended last. |
| `fastest` | Sort by ascending latency from the metrics endpoint. Models with no data appended last. |
| `none` | Return models in the order they were defined — no reordering. |
### Notes
- `routing_preferences` is **optional**. If omitted, the config-defined preferences are used.
- If provided in the request body, it **overrides** the config for that single request only.
- `model` is still required and is used as the fallback if no route is matched.
---
## Response Format
```json
{
"models": [
"anthropic/claude-sonnet-4-20250514",
"openai/gpt-4o",
"openai/gpt-4o-mini"
],
"route": "code generation",
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}
```
### Fields
| Field | Type | Description |
|---|---|---|
| `models` | string[] | Ranked model list. Use `models[0]` as primary; retry with `models[1]` on 429/5xx, and so on. |
| `route` | string \| null | Name of the matched route. `null` if no route matched — client should use the original request `model`. |
| `trace_id` | string | Trace ID for distributed tracing and observability. |
---
## Client Usage Pattern
```python
response = plano.routing_decision(request)
models = response["models"]
for model in models:
try:
result = call_llm(model, messages)
break # success — stop trying
except (RateLimitError, ServerError):
continue # try next model in the ranked list
```
---
## Configuration (set by platform/ops team)
Requires `version: v0.4.0` or above. Models listed under `routing_preferences` must be declared in `model_providers`.
```yaml
version: v0.4.0
model_providers:
- model: anthropic/claude-sonnet-4-20250514
access_key: $ANTHROPIC_API_KEY
- model: openai/gpt-4o
access_key: $OPENAI_API_KEY
- model: openai/gpt-4o-mini
access_key: $OPENAI_API_KEY
default: true
routing_preferences:
- name: code generation
description: generating new code snippets or boilerplate
models:
- anthropic/claude-sonnet-4-20250514
- openai/gpt-4o
selection_policy:
prefer: fastest
- name: general questions
description: casual conversation and simple queries
models:
- openai/gpt-4o-mini
- openai/gpt-4o
selection_policy:
prefer: cheapest
# Optional: live cost and latency data sources (max one per type)
model_metrics_sources:
# Option A: DigitalOcean public pricing (no auth required)
- type: digitalocean_pricing
refresh_interval: 3600
# Option B: custom cost endpoint (mutually exclusive with digitalocean_pricing)
# - type: cost_metrics
# url: https://internal-cost-api/models
# refresh_interval: 300 # seconds; omit for fetch-once on startup
# auth:
# type: bearer
# token: $COST_API_TOKEN
- type: prometheus_metrics
url: https://internal-prometheus/
query: histogram_quantile(0.95, sum by (model_name, le) (rate(model_latency_seconds_bucket[5m])))
refresh_interval: 60
```
### Startup validation
Plano validates metric source configuration at startup and exits with a clear error if:
| Condition | Error |
|---|---|
| `prefer: cheapest` with no cost source | `prefer: cheapest requires a cost data source — add cost_metrics or digitalocean_pricing` |
| `prefer: fastest` with no `prometheus_metrics` | `prefer: fastest requires a prometheus_metrics source` |
| Two `cost_metrics` entries | `only one cost_metrics source is allowed` |
| Two `prometheus_metrics` entries | `only one prometheus_metrics source is allowed` |
| Two `digitalocean_pricing` entries | `only one digitalocean_pricing source is allowed` |
| `cost_metrics` and `digitalocean_pricing` both present | `cannot both be configured — use one or the other` |
If a model listed in `routing_preferences` has no matching entry in the fetched pricing or latency data, Plano logs a `WARN` at startup — the model is still included but ranked last. The same warning is also emitted per routing request when a model has no data in cache at decision time (relevant for inline `routing_preferences` overrides that reference models not covered by the configured metrics sources).
### cost_metrics endpoint
Plano GETs `url` on startup (and on each `refresh_interval`). Expected response — a JSON object mapping model name to an object with `input_per_million` and `output_per_million` fields:
```json
{
"anthropic/claude-sonnet-4-20250514": {
"input_per_million": 3.0,
"output_per_million": 15.0
},
"openai/gpt-4o": {
"input_per_million": 5.0,
"output_per_million": 20.0
},
"openai/gpt-4o-mini": {
"input_per_million": 0.15,
"output_per_million": 0.6
}
}
```
- `auth.type: bearer` adds `Authorization: Bearer <token>` to the request
- Plano combines the two fields as `input_per_million + output_per_million` to produce a single cost scalar used for ranking
- Only relative order matters — the unit (e.g. USD per million tokens) is consistent so ranking is correct
### digitalocean_pricing source
Fetches public model pricing from the DigitalOcean Gen-AI catalog. No authentication required.
```yaml
model_metrics_sources:
- type: digitalocean_pricing
refresh_interval: 3600 # re-fetch every hour; omit to fetch once on startup
model_aliases:
openai-gpt-4o: openai/gpt-4o
openai-gpt-4o-mini: openai/gpt-4o-mini
anthropic-claude-sonnet-4: anthropic/claude-sonnet-4-20250514
```
DO catalog entries are stored by their `model_id` field (e.g. `openai-gpt-4o`). The cost scalar is `input_price_per_million + output_price_per_million`.
**`model_aliases`** — optional. Maps DO `model_id` values to the model names used in `routing_preferences`. Without aliases, cost data is stored under the DO model_id (e.g. `openai-gpt-4o`), which won't match models configured as `openai/gpt-4o`. Aliases let you bridge the naming gap without changing your routing config.
**Constraints:**
- `cost_metrics` and `digitalocean_pricing` cannot both be configured — use one or the other.
- Only one `digitalocean_pricing` entry is allowed.
### prometheus_metrics endpoint
Plano queries `{url}/api/v1/query?query={query}` on startup and each `refresh_interval`. The PromQL expression must return an instant vector with a `model_name` label:
```json
{
"status": "success",
"data": {
"resultType": "vector",
"result": [
{"metric": {"model_name": "anthropic/claude-sonnet-4-20250514"}, "value": [1234567890, "120.5"]},
{"metric": {"model_name": "openai/gpt-4o"}, "value": [1234567890, "200.3"]}
]
}
}
```
- The PromQL query is responsible for computing the percentile (e.g. `histogram_quantile(0.95, ...)`)
- Latency units are arbitrary — only relative order matters
- Models missing from the result are appended at the end of the ranked list
---
## Version Requirements
| Version | Top-level `routing_preferences` |
|---|---|
| `< v0.4.0` | Not allowed — startup error if present |
| `v0.4.0+` | Supported (required for model routing) |

21
docs/source/resources/includes/plano_config_full_reference.yaml
View file

@ -36,35 +36,20 @@ model_providers:
# can select the best model for each request based on intent. Requires the
# Arch-Router model (or equivalent) to be configured in overrides.llm_routing_model.
# Each preference has a name (short label) and a description (used for intent matching).
- model: openai/gpt-4o
name: gpt-4o-coding # Optional friendly name to distinguish multiple entries for same model
access_key: $OPENAI_API_KEY
- model: groq/llama-3.3-70b-versatile
access_key: $GROQ_API_KEY
routing_preferences:
- name: code generation
description: generating new code snippets, functions, or boilerplate based on user prompts or requirements
- name: code review
description: reviewing, analyzing, and suggesting improvements to existing code
- model: anthropic/claude-sonnet-4-0
name: claude-sonnet-reasoning
access_key: $ANTHROPIC_API_KEY
routing_preferences:
- name: reasoning
description: complex multi-step reasoning, math, logic puzzles, and analytical tasks
# passthrough_auth: forwards the client's Authorization header upstream instead of
# using the configured access_key. Useful for LiteLLM or similar proxy setups.
- model: openai/gpt-4o-litellm
base_url: https://litellm.example.com
passthrough_auth: true
# provider_interface: specifies the API format when the provider doesn't match
# the default inferred from the model name. Supported: openai, claude, gemini,
# mistral, groq, deepseek, plano
- model: groq/llama-3.3-70b-versatile
access_key: $GROQ_API_KEY
provider_interface: groq
# Custom/self-hosted endpoint with explicit http_host override
- model: openai/llama-3.3-70b
base_url: https://api.custom-provider.com
@ -179,7 +164,7 @@ overrides:
# Trim conversation history to fit within the model's context window
optimize_context_window: true
# Use Plano's agent orchestrator for multi-agent request routing
use_agent_orchestrator: true
use_agent_orchestrator: false
# Connect timeout for upstream provider clusters (e.g., "5s", "10s"). Default: "5s"
upstream_connect_timeout: 10s
# Path to the trusted CA bundle for upstream TLS verification

101
docs/source/resources/includes/plano_config_full_reference_rendered.yaml
View file

@ -8,6 +8,7 @@ endpoints:
connect_timeout: 0.005s
endpoint: 127.0.0.1
port: 80
protocol: http
flight_agent:
endpoint: localhost
port: 10520
@ -19,6 +20,11 @@ endpoints:
mistral_local:
endpoint: 127.0.0.1
port: 8001
secure_service:
endpoint: api.example.com
http_host: api.example.com
port: 443
protocol: https
weather_agent:
endpoint: localhost
port: 10510
@ -38,6 +44,9 @@ listeners:
router: plano_orchestrator_v1
type: agent
- address: 0.0.0.0
input_filters:
- input_guards
max_retries: 3
model_providers:
- access_key: $OPENAI_API_KEY
default: true
@ -56,6 +65,16 @@ listeners:
model: ministral-3b-latest
name: mistral/ministral-3b-latest
provider_interface: mistral
- access_key: $GROQ_API_KEY
model: llama-3.3-70b-versatile
name: groq/llama-3.3-70b-versatile
provider_interface: groq
routing_preferences:
- description: generating new code snippets, functions, or boilerplate based on
user prompts or requirements
name: code generation
- description: reviewing, analyzing, and suggesting improvements to existing code
name: code review
- base_url: https://litellm.example.com
cluster_name: openai_litellm.example.com
endpoint: litellm.example.com
@ -65,8 +84,21 @@ listeners:
port: 443
protocol: https
provider_interface: openai
- access_key: $CUSTOM_API_KEY
base_url: https://api.custom-provider.com
cluster_name: openai_api.custom-provider.com
endpoint: api.custom-provider.com
http_host: api.custom-provider.com
model: llama-3.3-70b
name: openai/llama-3.3-70b
port: 443
protocol: https
provider_interface: openai
name: model_1
output_filters:
- input_guards
port: 12000
timeout: 30s
type: model
- address: 0.0.0.0
name: prompt_function_listener
@ -95,6 +127,16 @@ model_providers:
model: ministral-3b-latest
name: mistral/ministral-3b-latest
provider_interface: mistral
- access_key: $GROQ_API_KEY
model: llama-3.3-70b-versatile
name: groq/llama-3.3-70b-versatile
provider_interface: groq
routing_preferences:
- description: generating new code snippets, functions, or boilerplate based on
user prompts or requirements
name: code generation
- description: reviewing, analyzing, and suggesting improvements to existing code
name: code review
- base_url: https://litellm.example.com
cluster_name: openai_litellm.example.com
endpoint: litellm.example.com
@ -104,6 +146,20 @@ model_providers:
port: 443
protocol: https
provider_interface: openai
- access_key: $CUSTOM_API_KEY
base_url: https://api.custom-provider.com
cluster_name: openai_api.custom-provider.com
endpoint: api.custom-provider.com
http_host: api.custom-provider.com
model: llama-3.3-70b
name: openai/llama-3.3-70b
port: 443
protocol: https
provider_interface: openai
- internal: true
model: Arch-Router
name: arch-router
provider_interface: plano
- internal: true
model: Arch-Function
name: arch-function
@ -112,8 +168,22 @@ model_providers:
model: Plano-Orchestrator
name: plano/orchestrator
provider_interface: plano
overrides:
agent_orchestration_model: Plano-Orchestrator
llm_routing_model: Arch-Router
optimize_context_window: true
prompt_target_intent_matching_threshold: 0.7
upstream_connect_timeout: 10s
upstream_tls_ca_path: /etc/ssl/certs/ca-certificates.crt
use_agent_orchestrator: false
prompt_guards:
input_guards:
jailbreak:
on_exception:
message: I'm sorry, I can't help with that request.
prompt_targets:
- description: Get current weather at a location.
- auto_llm_dispatch_on_response: true
description: Get current weather at a location.
endpoint:
http_method: POST
name: app_server
@ -129,7 +199,36 @@ prompt_targets:
name: days
required: true
type: int
system_prompt: You are a weather expert. Provide accurate and concise weather information.
ratelimits:
- limit:
tokens: 100000
unit: hour
model: openai/gpt-4o
selector:
key: x-user-id
value: '*'
- limit:
tokens: 500000
unit: day
model: openai/gpt-4o-mini
selector:
key: x-org-id
value: acme-corp
state_storage:
type: memory
system_prompt: 'You are a helpful assistant. Always respond concisely and accurately.
'
tracing:
opentracing_grpc_endpoint: http://localhost:4317
random_sampling: 100
span_attributes:
header_prefixes:
- x-user-
- x-org-
static:
environment: production
service.team: platform
trace_arch_internal: false
version: v0.3.0