Implement RetryOrchestrator as the top-level coordinator that:
- Manages the full retry lifecycle per request
- Integrates backoff, error detection, provider selection
- Handles request deduplication via content hashing
- Supports both same-provider retry and cross-provider failover
- Emits structured attempt records for observability
Signed-off-by: Troy Mitchell <i@troy-y.org>
Implement ProviderSelector that determines the next provider
for retry attempts based on:
- Failover provider list with priority ordering
- Latency-blocked provider filtering
- Retry-After header honoring
- Round-robin and priority-based selection strategies
Signed-off-by: Troy Mitchell <i@troy-y.org>
Add three state management components:
- LatencyBlockStateManager: tracks providers blocked due to
high latency with configurable block duration and scope
- LatencyTriggerCounter: counts consecutive latency threshold
breaches before triggering provider blocking
- RetryAfterStateManager: honors Retry-After headers with
per-provider/model/endpoint blocking scope
Signed-off-by: Troy Mitchell <i@troy-y.org>
Implement RetryErrorResponseBuilder that constructs structured
JSON error responses when all retry attempts are exhausted,
including per-attempt error details and provider information.
Signed-off-by: Troy Mitchell <i@troy-y.org>
Add the retry module with core type definitions including:
- RequestContext, RequestSignature for request deduplication
- RetryExhaustedError, AllProvidersExhaustedError for error handling
- AttemptError, AttemptErrorType for attempt tracking
- ValidationError, ValidationWarning for config validation
- Helper functions for provider extraction and hashing
Wire up pub mod retry in lib.rs.
Signed-off-by: Troy Mitchell <i@troy-y.org>
Add retry policy configuration types to support automatic retry and
failover for LLM requests:
- RetryPolicy: top-level config with fallback_models, default_strategy,
default_max_attempts, and per-status-code overrides
- BackoffConfig: exponential backoff with base_ms, max_ms, jitter, and
scope (per-model, per-provider, or global)
- RetryAfterConfig: Retry-After header handling with block scope and
duration limits
- HighLatencyConfig: latency-based blocking with threshold, measurement
type, and trigger conditions
- LatencyTriggerConfig: min_triggers and trigger_window for debouncing
- RetryStrategy enum: same_model, same_provider, different_provider
- StatusCodeEntry: flexible status code matching (single, range, list)
Also add retry_policy field to GatewayConfig with Default impl.
Signed-off-by: Troy Mitchell <i@troy-y.org>
The Python CLI (#902) and the JSON schema both allow `vercel` and
`openrouter` as `provider_interface`, and `hermesllm::ProviderId` knows
how to dispatch them — but `crates/common::LlmProviderType` was never
extended to deserialize them. As a result, `planoai up` with no user
config (which synthesizes both providers via `cli/planoai/defaults.py`)
caused brightstaff to crash on startup with:
unknown variant `vercel`, expected one of `anthropic`, ..., `digitalocean`
Add the missing enum variants and Display arms, plus a regression test
that asserts both round-trip through serde and resolve through
`to_provider_id()` (the exact path that previously panicked at parse).
* fix(routing): auto-migrate v0.3.0 inline routing_preferences to v0.4.0 top-level
Lift inline routing_preferences under each model_provider into the
top-level routing_preferences list with merged models[] and bump
version to v0.4.0, with a deprecation warning. Existing v0.3.0
demo configs (Claude Code, Codex, preference_based_routing, etc.)
keep working unchanged. Schema flags the inline shape as deprecated
but still accepts it. Docs and skills updated to canonical top-level
multi-model form.
* test(common): bump reference config assertion to v0.4.0
The rendered reference config was bumped to v0.4.0 when its inline
routing_preferences were lifted to the top level; align the
configuration deserialization test with that change.
* fix(config_generator): bump version to v0.4.0 up front in migration
Move the v0.3.0 -> v0.4.0 version bump to the top of
migrate_inline_routing_preferences so it runs unconditionally,
including for configs that already declare top-level
routing_preferences at v0.3.0. Previously the bump only fired
when inline migration produced entries, leaving top-level v0.3.0
configs rejected by brightstaff's v0.4.0 gate. Tests updated to
cover the new behavior and to confirm we never downgrade newer
versions.
* fix(config_generator): gate routing_preferences migration on version < v0.4.0
Short-circuit the migration when the config already declares v0.4.0
or newer. Anything at v0.4.0+ is assumed to be on the canonical
top-level shape and is passed through untouched, including stray
inline preferences (which are the author's bug to fix). Only v0.3.0
and older configs are rewritten and bumped.
Publish docker image (latest) / build-arm64 (push) Has been cancelled
Publish docker image (latest) / build-amd64 (push) Has been cancelled
Build and Deploy Documentation / build (push) Has been cancelled
CI / security-scan (push) Has been cancelled
CI / test-prompt-gateway (push) Has been cancelled
CI / test-model-alias-routing (push) Has been cancelled
CI / test-responses-api-with-state (push) Has been cancelled
CI / e2e-plano-tests (3.10) (push) Has been cancelled
CI / e2e-plano-tests (3.11) (push) Has been cancelled
CI / e2e-plano-tests (3.12) (push) Has been cancelled
CI / e2e-plano-tests (3.13) (push) Has been cancelled
CI / e2e-plano-tests (3.14) (push) Has been cancelled
CI / e2e-demo-preference (push) Has been cancelled
CI / e2e-demo-currency (push) Has been cancelled
Publish docker image (latest) / create-manifest (push) Has been cancelled
* add pluggable session cache with Redis backend
* add Redis session affinity demos (Docker Compose and Kubernetes)
* address PR review feedback on session cache
* document Redis session cache backend for model affinity
* sync rendered config reference with session_cache addition
* add tenant-scoped Redis session cache keys and remove dead log_affinity_hit
- Add tenant_header to SessionCacheConfig; when set, cache keys are scoped
as plano:affinity:{tenant_id}:{session_id} for multi-tenant isolation
- Thread tenant_id through RouterService, routing_service, and llm handlers
- Use Cow<'_, str> in session_key to avoid allocation when no tenant is set
- Remove unused log_affinity_hit (logging was already inlined at call sites)
* remove session_affinity_redis and session_affinity_redis_k8s demos
CI / test-prompt-gateway (push) Has been cancelled
CI / test-model-alias-routing (push) Has been cancelled
CI / test-responses-api-with-state (push) Has been cancelled
CI / e2e-plano-tests (3.10) (push) Has been cancelled
CI / e2e-plano-tests (3.11) (push) Has been cancelled
CI / e2e-plano-tests (3.12) (push) Has been cancelled
CI / e2e-plano-tests (3.13) (push) Has been cancelled
CI / e2e-plano-tests (3.14) (push) Has been cancelled
CI / e2e-demo-preference (push) Has been cancelled
CI / e2e-demo-currency (push) Has been cancelled
Publish docker image (latest) / build-arm64 (push) Has been cancelled
Publish docker image (latest) / build-amd64 (push) Has been cancelled
Publish docker image (latest) / create-manifest (push) Has been cancelled
Build and Deploy Documentation / build (push) Has been cancelled
* feat(provider): add xiaomi as first-class provider
* feat(demos): add xiaomi mimo integration demo
* refactor(demos): remove Xiaomi MiMo integration demo and update documentation
* updating model list and adding the xiamoi models
---------
Co-authored-by: Salman Paracha <salmanparacha@MacBook-Pro-389.local>
* support configurable orchestrator model via orchestration config section
* add self-hosting docs and demo for Plano-Orchestrator
* list all Plano-Orchestrator model variants in docs
* use overrides for custom routing and orchestration model
* update docs
* update orchestrator model name
* rename arch provider to plano, use llm_routing_model and agent_orchestration_model
* regenerate rendered config reference
* cleaning up plano cli commands
* adding support for wildcard model providers
* fixing compile errors
* fixing bugs related to default model provider, provider hint and duplicates in the model provider list
* fixed cargo fmt issues
* updating tests to always include the model id
* using default for the prompt_gateway path
* fixed the model name, as gpt-5-mini-2025-08-07 wasn't in the config
* making sure that all aliases and models match the config
* fixed the config generator to allow for base_url providers LLMs to include wildcard models
* re-ran the models list utility and added a shell script to run it
* updating docs to mention wildcard model providers
* updated provider_models.json to yaml, added that file to our docs for reference
* updating the build docs to use the new root-based build
---------
Co-authored-by: Salman Paracha <salmanparacha@MacBook-Pro-342.local>
* agents framework demo
* more changes
* add more changes
* pending changes
* fix tests
* fix more
* rebase with main and better handle error from mcp
* add trace for filters
* add test for client error, server error and for mcp error
* update schema validate code and rename kind => type in agent_filter
* fix agent description and pre-commit
* fix tests
* add provider specific request parsing in agents chat
* fix precommit and tests
* cleanup demo
* update readme
* fix pre-commit
* refactor tracing
* fix fmt
* fix: handle MessageContent enum in responses API conversion
- Update request.rs to handle new MessageContent enum structure from main
- MessageContent can now be Text(String) or Items(Vec<InputContent>)
- Handle new InputItem variants (ItemReference, FunctionCallOutput)
- Fixes compilation error after merging latest main (#632)
* address pr feedback
* fix span
* fix build
* update openai version
* first commit with tests to enable state mamangement via memory
* fixed logs to follow the conversational flow a bit better
* added support for supabase
* added the state_storage_v1_responses flag, and use that to store state appropriately
* cleaned up logs and fixed issue with connectivity for llm gateway in weather forecast demo
* fixed mixed inputs from openai v1/responses api (#632)
* fixed mixed inputs from openai v1/responses api
* removing tracing from model-alias-rouing
* handling additional input types from openairs
---------
Co-authored-by: Salman Paracha <salmanparacha@MacBook-Pro-342.local>
* resolving PR comments
---------
Co-authored-by: Salman Paracha <salmanparacha@MacBook-Pro-342.local>
* adding canonical tracing support via bright-staff
* improved formatting for tools in the traces
* removing anthropic from the currency exchange demo
* using Envoy to transport traces, not calling OTEL directly
* moving otel collcetor cluster outside tracing if/else
* minor fixes to not write to the OTEL collector if tracing is disabled
* fixed PR comments and added more trace attributes
* more fixes based on PR comments
* more clean up based on PR comments
---------
Co-authored-by: Salman Paracha <salmanparacha@MacBook-Pro-342.local>
* making first commit. still need to work on streaming respones
* making first commit. still need to work on streaming respones
* stream buffer implementation with tests
* adding grok API keys to workflow
* fixed changes based on code review
* adding support for bedrock models
* fixed issues with translation to claude code
---------
Co-authored-by: Salman Paracha <salmanparacha@MacBook-Pro-342.local>
* adding function_calling functionality via rust
* fixed rendered YAML file
* removed model_server from envoy.template and forwarding traffic to bright_staff
* fixed bugs in function_calling.rs that were breaking tests. All good now
* updating e2e test to clean up disk usage
* removing Arch* models to be used as a default model if one is not specified
* if the user sets arch-function base_url we should honor it
* fixing demos as we needed to pin to a particular version of huggingface_hub else the chatbot ui wouldn't build
* adding a constant for Arch-Function model name
* fixing some edge cases with calls made to Arch-Function
* fixed JSON parsing issues in function_calling.rs
* fixed bug where the raw response from Arch-Function was re-encoded
* removed debug from supervisord.conf
* commenting out disk cleanup
* adding back disk space
---------
Co-authored-by: Salman Paracha <salmanparacha@MacBook-Pro-288.local>
Co-authored-by: Salman Paracha <salmanparacha@MacBook-Pro-342.local>
* adding support for base_url
* updated docs
* fixed tests for config generator
* making fixes based on PR comments
---------
Co-authored-by: Salman Paracha <salmanparacha@MacBook-Pro-288.local>
* first commit to get Bedrock Converse API working. Next commit support for streaming and binary frames
* adding translation from BedrockBinaryFrameDecoder to AnthropicMessagesEvent
* Claude Code works with Amazon Bedrock
* added tests for openai streaming from bedrock
* PR comments fixed
* adding support for bedrock in docs as supported provider
* cargo fmt
* revertted to chatgpt models for claude code routing
---------
Co-authored-by: Salman Paracha <salmanparacha@MacBook-Pro-288.local>
Co-authored-by: Adil Hafeez <adil.hafeez@gmail.com>
