mirror of
https://github.com/katanemo/plano.git
synced 2026-07-23 16:51:04 +02:00
844f08bda7
* feat(routing): cache-aware routing and zero-config prompt caching
Reconcile prompt caching with intelligent routing: zero-config implicit
session affinity, cache_control preservation/injection across transforms,
cache-adjusted routing economics, and a response-driven cache-hit feedback loop.
* refactor(caching): model-aware cache markers, drop cache-aware routing
* feat(routing): session stickiness cache-regret cost gate
* refactor(routing): expose per-model input and cached rates
* fix(docker): patch libssh2 CVEs by upgrading after package install
* feat: record counterfactual route on vetoed session switches
Add opt-in session_stickiness.record_counterfactual that emits the
plano.switch.counterfactual_route OTEL attribute when the cache-regret
gate retains the previous model, capturing the route it would have taken
had the switch been allowed. Telemetry only; the candidate is never
dispatched. Includes schema, demo config, and a getting-started guide.
* refactor: group prompt-caching and session-stickiness logic into cohesive modules
Extract the two concerns interleaved in the LLM handler into dedicated
sibling modules for readability: prompt_caching.rs (cache-marker injection)
and session_stickiness.rs (session key/prefix-hash resolution, pin lookup,
cache-regret gate, and pin planning). handlers/llm/mod.rs is now a thin
orchestrator that calls them at clear phase boundaries. Behavior-preserving.
* refactor(routing): unify session routing into a cumulative switch budget
Replace observed_cache_hit + per-request threshold with time/provider-TTL
warmth and a per-session switch budget in a single session_router::route()
used by both the full-proxy and /routing decision paths.
* refactor(routing): move switch budget to routing.routing_budget and rename telemetry
* refactor(routing): drop negative-cost budget refund from routing_budget
* refactor(routing): express routing_budget as a percentage overhead cap
Replace the absolute seed_usd pool with max_overhead_pct (whole-number
percent) measured against the session's running never-switch baseline:
allow a paid switch only while cumulative switch spend stays within
max_overhead_pct% of what staying on the anchor would have cost. Track
baseline_usd and switch_spend_usd on the binding (monotonic, no refunds).
* refactor(observability): align routing-budget telemetry with the overhead cap
Rename the switch-decision reasons (within_budget/over_budget -> within_cap/
over_cap) and the session span attributes to match the percentage overhead-cap
model: budget_remaining_in_usd -> overhead_pct + switch_spend_in_usd +
baseline_in_usd, and switch threshold_in_usd -> overhead_ceiling_in_usd.
* feat(observability): emit per-request and cumulative session cost
Price each turn from the catalog rates and surface it as llm.usage.{input,
output,total}_cost_usd on the (llm) span, then accumulate into a conversation-
level session_cost_usd on the binding and emit plano.session.total_cost_in_usd
on the routing span. Cache-creation tokens are priced at the plain input rate,
and the OpenAI vs Anthropic prompt-token convention (cached folded in vs
reported separately) is captured at parse time so the cost math is correct for
both. Rates are resolved request-side since the response path is synchronous.
* refactor(routing): price the never-switch baseline against the session default model
Distinguish the session's default_model (the model it started on, i.e. what it
would have cost by never switching) from anchor_model (the model that handled
the latest request, which the session is warm on). The never-switch baseline now
grows at the default model's cached rate rather than the drifting anchor's, so
the overhead-cap denominator stays true to "% above never-switching"; switch cost
is still measured against the current anchor. Also rename estimate_context_tokens
-> actual_context_tokens (and est_context_tokens -> context_tokens) since the
router prefers the provider's real prompt-token count when the session is warm.
* feat(routing): track bounded route history and price returns to still-warm models
Record a bounded (LRU, capped) per-model visit history on the session binding
and use it to sharpen the switch-cost estimate: a return to a model still within
its cache window re-reads only the tokens appended since its last visit (at its
cached rate) instead of the whole context at the uncached rate, so an A->B->A
switch is no longer over-charged as a full re-ingest. History is carried through
the response-side refresh (refining the anchor's entry from real usage) and
cleared on prefix drift. Adds the plano.switch.candidate_warm_tokens span
attribute.
* refactor(routing): tidy test names and rename pin events to binding events
Shorten run-on test names and replace stale "pin" vocabulary in the session
binding metric (brightstaff_session_pin_events_total -> _binding_events_total),
dropping lifecycle event labels that are no longer emitted.
* feat(routing): price switch cost against uncached rates when caching is off
The overhead-cap gate previously always priced the anchor at its cached rate,
assuming a warm provider cache. When prompt caching is disabled there is no
warm cache to lose, so both the switch cost and the never-switch baseline are
now priced at the plain uncached input rate:
switch_cost = context_tokens x (candidate_uncached - anchor_uncached) / 1M.
* fix(brightstaff): bound pricing-feed fetch with timeouts and warn when session state runs without a tenant header
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(routing): validate the session overhead cap end-to-end across a multi-turn session
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(demos): fix stale pinning semantics and add routing_budget demo script
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Adil Hafeez <adil.hafeez@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
1167 lines
51 KiB
YAML
1167 lines
51 KiB
YAML
admin:
|
|
address:
|
|
socket_address: { address: 0.0.0.0, port_value: 9901 }
|
|
|
|
stats_config:
|
|
histogram_bucket_settings:
|
|
match:
|
|
exact: "wasmcustom.time_to_first_token"
|
|
buckets:
|
|
- 100
|
|
- 500
|
|
- 800
|
|
- 1000
|
|
- 1200
|
|
- 1400
|
|
- 1600
|
|
- 1800
|
|
- 2000
|
|
- 2200
|
|
- 2400
|
|
- 3000
|
|
- 3500
|
|
- 4000
|
|
- 4500
|
|
- 5000
|
|
- 6000
|
|
- 10000
|
|
- 60000
|
|
- 180000
|
|
static_resources:
|
|
listeners:
|
|
- name: ingress_traffic
|
|
address:
|
|
socket_address:
|
|
address: {{ prompt_gateway_listener.address }}
|
|
port_value: {{ prompt_gateway_listener.port }}
|
|
traffic_direction: INBOUND
|
|
filter_chains:
|
|
- filters:
|
|
- name: envoy.filters.network.http_connection_manager
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
|
|
{% if "random_sampling" in plano_tracing and plano_tracing["random_sampling"] > 0 %}
|
|
generate_request_id: true
|
|
tracing:
|
|
provider:
|
|
name: envoy.tracers.opentelemetry
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig
|
|
grpc_service:
|
|
envoy_grpc:
|
|
cluster_name: opentelemetry_collector
|
|
timeout: 0.250s
|
|
service_name: plano(inbound)
|
|
random_sampling:
|
|
value: {{ plano_tracing.random_sampling }}
|
|
operation: "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
|
|
{% endif %}
|
|
stat_prefix: plano(inbound)
|
|
codec_type: AUTO
|
|
scheme_header_transformation:
|
|
scheme_to_overwrite: https
|
|
access_log:
|
|
- name: envoy.access_loggers.file
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
|
|
path: "/var/log/access_ingress.log"
|
|
format: |
|
|
[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" "%UPSTREAM_CLUSTER%"
|
|
route_config:
|
|
name: local_routes
|
|
virtual_hosts:
|
|
- name: local_service
|
|
domains:
|
|
- "*"
|
|
routes:
|
|
- match:
|
|
prefix: "/"
|
|
route:
|
|
auto_host_rewrite: true
|
|
cluster: arch_prompt_gateway_listener
|
|
timeout: {{ prompt_gateway_listener.timeout }}
|
|
http_filters:
|
|
- name: envoy.filters.http.router
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
|
|
|
|
- name: ingress_traffic_prompt
|
|
address:
|
|
socket_address:
|
|
address: 0.0.0.0
|
|
port_value: 10001
|
|
traffic_direction: INBOUND
|
|
filter_chains:
|
|
- filters:
|
|
- name: envoy.filters.network.http_connection_manager
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
|
|
stat_prefix: ingress_traffic
|
|
codec_type: AUTO
|
|
scheme_header_transformation:
|
|
scheme_to_overwrite: https
|
|
access_log:
|
|
- name: envoy.access_loggers.file
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
|
|
path: "/var/log/access_ingress_prompt.log"
|
|
format: |
|
|
[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" "%UPSTREAM_CLUSTER%"
|
|
route_config:
|
|
name: local_routes
|
|
virtual_hosts:
|
|
- name: local_service
|
|
domains:
|
|
- "*"
|
|
routes:
|
|
{% for provider in plano_model_providers %}
|
|
# if endpoint is set then use custom cluster for upstream llm
|
|
{% if provider.endpoint %}
|
|
{% set llm_cluster_name = provider.cluster_name %}
|
|
{% else %}
|
|
{% set llm_cluster_name = provider.provider_interface %}
|
|
{% endif %}
|
|
- match:
|
|
prefix: "/"
|
|
headers:
|
|
- name: "x-arch-llm-provider"
|
|
string_match:
|
|
exact: {{ llm_cluster_name }}
|
|
route:
|
|
auto_host_rewrite: true
|
|
cluster: {{ llm_cluster_name }}
|
|
timeout: 300s
|
|
{% endfor %}
|
|
|
|
{% if agent_orchestrator %}
|
|
- match:
|
|
prefix: "/"
|
|
headers:
|
|
- name: "x-arch-llm-provider"
|
|
string_match:
|
|
exact: {{ agent_orchestrator }}
|
|
route:
|
|
auto_host_rewrite: true
|
|
cluster: {{ agent_orchestrator }}
|
|
timeout: 300s
|
|
{% endif %}
|
|
http_filters:
|
|
- name: envoy.filters.http.compressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor
|
|
compressor_library:
|
|
name: compress
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.compression.gzip.compressor.v3.Gzip
|
|
memory_level: 3
|
|
window_bits: 10
|
|
- name: envoy.filters.http.wasm_prompt
|
|
typed_config:
|
|
"@type": type.googleapis.com/udpa.type.v1.TypedStruct
|
|
type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
|
|
value:
|
|
config:
|
|
name: "http_config"
|
|
root_id: prompt_gateway
|
|
configuration:
|
|
"@type": "type.googleapis.com/google.protobuf.StringValue"
|
|
value: |
|
|
{{ plano_config | indent(32) }}
|
|
vm_config:
|
|
runtime: "envoy.wasm.runtime.v8"
|
|
code:
|
|
local:
|
|
filename: "/etc/envoy/proxy-wasm-plugins/prompt_gateway.wasm"
|
|
- name: envoy.filters.http.wasm_llm
|
|
typed_config:
|
|
"@type": type.googleapis.com/udpa.type.v1.TypedStruct
|
|
type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
|
|
value:
|
|
config:
|
|
name: "http_config"
|
|
root_id: llm_gateway
|
|
configuration:
|
|
"@type": "type.googleapis.com/google.protobuf.StringValue"
|
|
value: |
|
|
{{ plano_llm_config | indent(32) }}
|
|
vm_config:
|
|
runtime: "envoy.wasm.runtime.v8"
|
|
code:
|
|
local:
|
|
filename: "/etc/envoy/proxy-wasm-plugins/llm_gateway.wasm"
|
|
- name: envoy.filters.http.decompressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.decompressor.v3.Decompressor
|
|
decompressor_library:
|
|
name: decompress
|
|
typed_config:
|
|
"@type": "type.googleapis.com/envoy.extensions.compression.gzip.decompressor.v3.Gzip"
|
|
# Must be >= the window_bits of whatever compressed the response
|
|
# (upstreams use 15; our compressor uses 10). A smaller inflate window
|
|
# makes zlib silently stop emitting data mid-stream, truncating SSE
|
|
# responses after the first few KB. 15 handles any compliant gzip.
|
|
window_bits: 15
|
|
chunk_size: 8192
|
|
# If this ratio is set too low, then body data will not be decompressed completely.
|
|
max_inflate_ratio: 1000
|
|
- name: envoy.filters.http.router
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
|
|
|
|
- name: outbound_api_traffic
|
|
address:
|
|
socket_address:
|
|
address: 0.0.0.0
|
|
port_value: 11000
|
|
traffic_direction: OUTBOUND
|
|
filter_chains:
|
|
- filters:
|
|
- name: envoy.filters.network.http_connection_manager
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
|
|
# {% if "random_sampling" in plano_tracing and plano_tracing["random_sampling"] > 0 %}
|
|
# generate_request_id: true
|
|
# tracing:
|
|
# provider:
|
|
# name: envoy.tracers.opentelemetry
|
|
# typed_config:
|
|
# "@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig
|
|
# grpc_service:
|
|
# envoy_grpc:
|
|
# cluster_name: opentelemetry_collector
|
|
# timeout: 0.250s
|
|
# service_name: tools
|
|
# random_sampling:
|
|
# value: {{ plano_tracing.random_sampling }}
|
|
# {% endif %}
|
|
stat_prefix: outbound_api_traffic
|
|
codec_type: AUTO
|
|
scheme_header_transformation:
|
|
scheme_to_overwrite: https
|
|
access_log:
|
|
- name: envoy.access_loggers.file
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
|
|
path: "/var/log/access_internal.log"
|
|
format: |
|
|
[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" "%UPSTREAM_CLUSTER%"
|
|
route_config:
|
|
name: local_routes
|
|
virtual_hosts:
|
|
- name: local_service
|
|
domains:
|
|
- "*"
|
|
routes:
|
|
- match:
|
|
prefix: "/"
|
|
headers:
|
|
- name: "x-arch-upstream"
|
|
string_match:
|
|
exact: bright_staff
|
|
route:
|
|
auto_host_rewrite: true
|
|
cluster: bright_staff
|
|
timeout: 300s
|
|
{% for cluster_name, cluster in plano_clusters.items() %}
|
|
- match:
|
|
prefix: "/"
|
|
headers:
|
|
- name: "x-arch-upstream"
|
|
string_match:
|
|
exact: {{ cluster_name }}
|
|
route:
|
|
auto_host_rewrite: true
|
|
cluster: {{ cluster_name }}
|
|
timeout: 300s
|
|
{% if cluster.prefix_affinity %}
|
|
# Feeds the cluster's RING_HASH lb_policy: requests with the
|
|
# same prompt-prefix hash stick to the same replica.
|
|
hash_policy:
|
|
- header:
|
|
header_name: x-plano-prefix-hash
|
|
{% endif %}
|
|
{% endfor %}
|
|
http_filters:
|
|
- name: envoy.filters.http.router
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
|
|
|
|
{% for listener in listeners %}
|
|
|
|
{% if listener.agents %}
|
|
|
|
# agent listeners
|
|
- name: {{ listener.name | replace(" ", "_") }}
|
|
address:
|
|
socket_address:
|
|
address: 0.0.0.0
|
|
port_value: {{ listener.port }}
|
|
filter_chains:
|
|
- filters:
|
|
- name: envoy.filters.network.http_connection_manager
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
|
|
{% if "random_sampling" in plano_tracing and plano_tracing["random_sampling"] > 0 %}
|
|
generate_request_id: true
|
|
tracing:
|
|
provider:
|
|
name: envoy.tracers.opentelemetry
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig
|
|
grpc_service:
|
|
envoy_grpc:
|
|
cluster_name: opentelemetry_collector
|
|
timeout: 0.250s
|
|
service_name: plano(inbound)
|
|
random_sampling:
|
|
value: {{ plano_tracing.random_sampling }}
|
|
operation: "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
|
|
{% endif %}
|
|
stat_prefix: {{ listener.name | replace(" ", "_") }}_traffic
|
|
codec_type: AUTO
|
|
scheme_header_transformation:
|
|
scheme_to_overwrite: https
|
|
access_log:
|
|
- name: envoy.access_loggers.file
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
|
|
path: "/var/log/access_agent.log"
|
|
route_config:
|
|
name: local_routes
|
|
request_headers_to_add:
|
|
- header:
|
|
key: "x-arch-agent-listener-name"
|
|
value: "{{ listener.name }}"
|
|
virtual_hosts:
|
|
- name: local_service
|
|
domains:
|
|
- "*"
|
|
routes:
|
|
- match:
|
|
prefix: "/healthz"
|
|
direct_response:
|
|
status: 200
|
|
- match:
|
|
prefix: "/"
|
|
route:
|
|
auto_host_rewrite: true
|
|
prefix_rewrite: "/agents/"
|
|
cluster: bright_staff
|
|
timeout: {{ listener.timeout | default('30s') }}
|
|
http_filters:
|
|
- name: envoy.filters.http.compressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor
|
|
compressor_library:
|
|
name: envoy.compression.brotli.compressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.compression.brotli.compressor.v3.Brotli
|
|
- name: envoy.filters.http.compressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor
|
|
compressor_library:
|
|
name: compress
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.compression.gzip.compressor.v3.Gzip
|
|
memory_level: 3
|
|
window_bits: 10
|
|
- name: envoy.filters.http.decompressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.decompressor.v3.Decompressor
|
|
decompressor_library:
|
|
name: decompress
|
|
typed_config:
|
|
"@type": "type.googleapis.com/envoy.extensions.compression.gzip.decompressor.v3.Gzip"
|
|
# Must be >= the window_bits of whatever compressed the response
|
|
# (upstreams use 15; our compressor uses 10). A smaller inflate window
|
|
# makes zlib silently stop emitting data mid-stream, truncating SSE
|
|
# responses after the first few KB. 15 handles any compliant gzip.
|
|
window_bits: 15
|
|
chunk_size: 8192
|
|
# If this ratio is set too low, then body data will not be decompressed completely.
|
|
max_inflate_ratio: 1000
|
|
- name: envoy.filters.http.decompressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.decompressor.v3.Decompressor
|
|
decompressor_library:
|
|
name: envoy.compression.brotli.decompressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.compression.brotli.decompressor.v3.Brotli
|
|
- name: envoy.filters.http.router
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
|
|
|
|
{% endif %}
|
|
{% endfor %}
|
|
|
|
- name: egress_traffic
|
|
address:
|
|
socket_address:
|
|
address: {{ llm_gateway_listener.address }}
|
|
port_value: {{ llm_gateway_listener.port }}
|
|
traffic_direction: OUTBOUND
|
|
filter_chains:
|
|
- filters:
|
|
- name: envoy.filters.network.http_connection_manager
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
|
|
stat_prefix: egress_traffic
|
|
codec_type: AUTO
|
|
scheme_header_transformation:
|
|
scheme_to_overwrite: https
|
|
access_log:
|
|
- name: envoy.access_loggers.file
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
|
|
path: "/var/log/access_llm.log"
|
|
format: |
|
|
[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" "%UPSTREAM_CLUSTER%" attempts=%UPSTREAM_REQUEST_ATTEMPT_COUNT%
|
|
route_config:
|
|
name: local_routes
|
|
virtual_hosts:
|
|
- name: local_service
|
|
domains:
|
|
- "*"
|
|
routes:
|
|
- match:
|
|
prefix: "/healthz"
|
|
direct_response:
|
|
status: 200
|
|
- match:
|
|
prefix: "/"
|
|
route:
|
|
auto_host_rewrite: true
|
|
cluster: bright_staff
|
|
timeout: {{ llm_gateway_listener.timeout }}
|
|
http_filters:
|
|
- name: envoy.filters.http.compressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor
|
|
compressor_library:
|
|
name: envoy.compression.brotli.compressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.compression.brotli.compressor.v3.Brotli
|
|
- name: envoy.filters.http.compressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor
|
|
compressor_library:
|
|
name: compress
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.compression.gzip.compressor.v3.Gzip
|
|
memory_level: 3
|
|
window_bits: 10
|
|
- name: envoy.filters.http.decompressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.decompressor.v3.Decompressor
|
|
decompressor_library:
|
|
name: decompress
|
|
typed_config:
|
|
"@type": "type.googleapis.com/envoy.extensions.compression.gzip.decompressor.v3.Gzip"
|
|
# Must be >= the window_bits of whatever compressed the response
|
|
# (upstreams use 15; our compressor uses 10). A smaller inflate window
|
|
# makes zlib silently stop emitting data mid-stream, truncating SSE
|
|
# responses after the first few KB. 15 handles any compliant gzip.
|
|
window_bits: 15
|
|
chunk_size: 8192
|
|
# If this ratio is set too low, then body data will not be decompressed completely.
|
|
max_inflate_ratio: 1000
|
|
- name: envoy.filters.http.decompressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.decompressor.v3.Decompressor
|
|
decompressor_library:
|
|
name: envoy.compression.brotli.decompressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.compression.brotli.decompressor.v3.Brotli
|
|
- name: envoy.filters.http.router
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
|
|
|
|
- name: egress_traffic_llm
|
|
address:
|
|
socket_address:
|
|
address: 0.0.0.0
|
|
port_value: 12001
|
|
filter_chains:
|
|
- filters:
|
|
- name: envoy.filters.network.http_connection_manager
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
|
|
{% if "random_sampling" in plano_tracing and plano_tracing["random_sampling"] > 0 %}
|
|
generate_request_id: true
|
|
tracing:
|
|
provider:
|
|
name: envoy.tracers.opentelemetry
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig
|
|
grpc_service:
|
|
envoy_grpc:
|
|
cluster_name: opentelemetry_collector
|
|
timeout: 0.250s
|
|
service_name: plano(outbound)
|
|
random_sampling:
|
|
value: {{ plano_tracing.random_sampling }}
|
|
operation: "%REQ(:METHOD)% %REQ(:AUTHORITY)%%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
|
|
{% endif %}
|
|
stat_prefix: egress_traffic
|
|
codec_type: AUTO
|
|
scheme_header_transformation:
|
|
scheme_to_overwrite: https
|
|
access_log:
|
|
- name: envoy.access_loggers.file
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
|
|
path: "/var/log/access_llm.log"
|
|
format: |
|
|
[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" "%UPSTREAM_CLUSTER%" attempts=%UPSTREAM_REQUEST_ATTEMPT_COUNT%
|
|
route_config:
|
|
name: local_routes
|
|
virtual_hosts:
|
|
- name: local_service
|
|
domains:
|
|
- "*"
|
|
routes:
|
|
{% for provider in plano_model_providers %}
|
|
# if endpoint is set then use custom cluster for upstream llm
|
|
{% if provider.endpoint %}
|
|
{% set llm_cluster_name = provider.cluster_name %}
|
|
{% else %}
|
|
{% set llm_cluster_name = provider.provider_interface %}
|
|
{% endif %}
|
|
- match:
|
|
prefix: "/"
|
|
headers:
|
|
- name: "x-arch-llm-provider"
|
|
string_match:
|
|
exact: {{ llm_cluster_name }}
|
|
route:
|
|
auto_host_rewrite: true
|
|
cluster: {{ llm_cluster_name }}
|
|
timeout: 300s
|
|
{% if llm_gateway_listener.max_retries %}
|
|
retry_policy:
|
|
retry_on: "5xx,connect-failure,refused-stream,reset,retriable-status-codes"
|
|
num_retries: {{ llm_gateway_listener.max_retries }}
|
|
per_try_timeout: 30s
|
|
retriable_status_codes: [429, 500, 502, 503, 504]
|
|
retry_back_off:
|
|
base_interval: 0.5s
|
|
max_interval: 5s
|
|
{% endif %}
|
|
{% endfor %}
|
|
- match:
|
|
prefix: "/"
|
|
direct_response:
|
|
status: 400
|
|
body:
|
|
inline_string: "x-arch-llm-provider header not set, llm gateway cannot perform routing\n"
|
|
http_filters:
|
|
- name: envoy.filters.http.compressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor
|
|
compressor_library:
|
|
name: envoy.compression.brotli.compressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.compression.brotli.compressor.v3.Brotli
|
|
chunk_size: 8192
|
|
- name: envoy.filters.http.compressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor
|
|
compressor_library:
|
|
name: compress
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.compression.gzip.compressor.v3.Gzip
|
|
memory_level: 3
|
|
window_bits: 10
|
|
- name: envoy.filters.http.wasm
|
|
typed_config:
|
|
"@type": type.googleapis.com/udpa.type.v1.TypedStruct
|
|
type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
|
|
value:
|
|
config:
|
|
name: "http_config"
|
|
root_id: llm_gateway
|
|
configuration:
|
|
"@type": "type.googleapis.com/google.protobuf.StringValue"
|
|
value: |
|
|
{{ plano_llm_config | indent(32) }}
|
|
vm_config:
|
|
runtime: "envoy.wasm.runtime.v8"
|
|
code:
|
|
local:
|
|
filename: "/etc/envoy/proxy-wasm-plugins/llm_gateway.wasm"
|
|
- name: envoy.filters.http.decompressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.decompressor.v3.Decompressor
|
|
decompressor_library:
|
|
name: decompress
|
|
typed_config:
|
|
"@type": "type.googleapis.com/envoy.extensions.compression.gzip.decompressor.v3.Gzip"
|
|
# Must be >= the window_bits of whatever compressed the response
|
|
# (upstreams use 15; our compressor uses 10; Envoy's default here is
|
|
# only 12). A smaller inflate window makes zlib silently stop emitting
|
|
# data mid-stream, truncating SSE responses after the first few KB.
|
|
# 15 handles any compliant gzip.
|
|
window_bits: 15
|
|
chunk_size: 8192
|
|
# If this ratio is set too low, then body data will not be decompressed completely.
|
|
max_inflate_ratio: 1000
|
|
- name: envoy.filters.http.decompressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.decompressor.v3.Decompressor
|
|
decompressor_library:
|
|
name: envoy.compression.brotli.decompressor
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.compression.brotli.decompressor.v3.Brotli
|
|
chunk_size: 8192
|
|
- name: envoy.filters.http.router
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
|
|
|
|
clusters:
|
|
|
|
- name: plano
|
|
connect_timeout: {{ upstream_connect_timeout | default('5s') }}
|
|
type: LOGICAL_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
load_assignment:
|
|
cluster_name: plano
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: archfc.katanemo.dev
|
|
port_value: 443
|
|
hostname: "archfc.katanemo.dev"
|
|
transport_socket:
|
|
name: envoy.transport_sockets.tls
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
|
|
sni: archfc.katanemo.dev
|
|
common_tls_context:
|
|
tls_params:
|
|
tls_minimum_protocol_version: TLSv1_2
|
|
tls_maximum_protocol_version: TLSv1_3
|
|
validation_context:
|
|
trusted_ca:
|
|
filename: {{ upstream_tls_ca_path | default('/etc/ssl/certs/ca-certificates.crt') }}
|
|
|
|
- name: anthropic
|
|
connect_timeout: {{ upstream_connect_timeout | default('5s') }}
|
|
type: LOGICAL_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
load_assignment:
|
|
cluster_name: anthropic
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: api.anthropic.com
|
|
port_value: 443
|
|
hostname: "api.anthropic.com"
|
|
transport_socket:
|
|
name: envoy.transport_sockets.tls
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
|
|
sni: api.anthropic.com
|
|
common_tls_context:
|
|
tls_params:
|
|
tls_minimum_protocol_version: TLSv1_2
|
|
tls_maximum_protocol_version: TLSv1_3
|
|
validation_context:
|
|
trusted_ca:
|
|
filename: {{ upstream_tls_ca_path | default('/etc/ssl/certs/ca-certificates.crt') }}
|
|
|
|
- name: deepseek
|
|
connect_timeout: {{ upstream_connect_timeout | default('5s') }}
|
|
type: LOGICAL_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
load_assignment:
|
|
cluster_name: deepseek
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: api.deepseek.com
|
|
port_value: 443
|
|
hostname: "api.deepseek.com"
|
|
transport_socket:
|
|
name: envoy.transport_sockets.tls
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
|
|
sni: api.deepseek.com
|
|
common_tls_context:
|
|
tls_params:
|
|
tls_minimum_protocol_version: TLSv1_2
|
|
tls_maximum_protocol_version: TLSv1_3
|
|
validation_context:
|
|
trusted_ca:
|
|
filename: {{ upstream_tls_ca_path | default('/etc/ssl/certs/ca-certificates.crt') }}
|
|
|
|
- name: xai
|
|
connect_timeout: {{ upstream_connect_timeout | default('5s') }}
|
|
type: LOGICAL_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
load_assignment:
|
|
cluster_name: xai
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: api.x.ai
|
|
port_value: 443
|
|
hostname: "api.x.ai"
|
|
transport_socket:
|
|
name: envoy.transport_sockets.tls
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
|
|
sni: api.x.ai
|
|
common_tls_context:
|
|
tls_params:
|
|
tls_minimum_protocol_version: TLSv1_2
|
|
tls_maximum_protocol_version: TLSv1_3
|
|
validation_context:
|
|
trusted_ca:
|
|
filename: {{ upstream_tls_ca_path | default('/etc/ssl/certs/ca-certificates.crt') }}
|
|
|
|
- name: moonshotai
|
|
connect_timeout: {{ upstream_connect_timeout | default('5s') }}
|
|
type: LOGICAL_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
load_assignment:
|
|
cluster_name: moonshotai
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: api.moonshot.ai
|
|
port_value: 443
|
|
hostname: "api.moonshot.ai"
|
|
transport_socket:
|
|
name: envoy.transport_sockets.tls
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
|
|
sni: api.moonshot.ai
|
|
common_tls_context:
|
|
tls_params:
|
|
tls_minimum_protocol_version: TLSv1_2
|
|
tls_maximum_protocol_version: TLSv1_3
|
|
validation_context:
|
|
trusted_ca:
|
|
filename: {{ upstream_tls_ca_path | default('/etc/ssl/certs/ca-certificates.crt') }}
|
|
|
|
- name: zhipu
|
|
connect_timeout: {{ upstream_connect_timeout | default('5s') }}
|
|
type: LOGICAL_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
load_assignment:
|
|
cluster_name: zhipu
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: open.bigmodel.cn
|
|
port_value: 443
|
|
hostname: "open.bigmodel.cn"
|
|
transport_socket:
|
|
name: envoy.transport_sockets.tls
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
|
|
sni: open.bigmodel.cn
|
|
common_tls_context:
|
|
tls_params:
|
|
tls_minimum_protocol_version: TLSv1_2
|
|
tls_maximum_protocol_version: TLSv1_3
|
|
validation_context:
|
|
trusted_ca:
|
|
filename: {{ upstream_tls_ca_path | default('/etc/ssl/certs/ca-certificates.crt') }}
|
|
|
|
- name: together_ai
|
|
connect_timeout: {{ upstream_connect_timeout | default('5s') }}
|
|
type: LOGICAL_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
load_assignment:
|
|
cluster_name: xai
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: api.together.xyz
|
|
port_value: 443
|
|
hostname: "api.together.xyz"
|
|
transport_socket:
|
|
name: envoy.transport_sockets.tls
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
|
|
sni: api.together.xyz
|
|
common_tls_context:
|
|
tls_params:
|
|
tls_minimum_protocol_version: TLSv1_2
|
|
tls_maximum_protocol_version: TLSv1_3
|
|
validation_context:
|
|
trusted_ca:
|
|
filename: {{ upstream_tls_ca_path | default('/etc/ssl/certs/ca-certificates.crt') }}
|
|
|
|
- name: gemini
|
|
connect_timeout: {{ upstream_connect_timeout | default('5s') }}
|
|
type: LOGICAL_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
load_assignment:
|
|
cluster_name: gemini
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: generativelanguage.googleapis.com
|
|
port_value: 443
|
|
hostname: "generativelanguage.googleapis.com"
|
|
transport_socket:
|
|
name: envoy.transport_sockets.tls
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
|
|
sni: generativelanguage.googleapis.com
|
|
common_tls_context:
|
|
tls_params:
|
|
tls_minimum_protocol_version: TLSv1_2
|
|
tls_maximum_protocol_version: TLSv1_3
|
|
validation_context:
|
|
trusted_ca:
|
|
filename: {{ upstream_tls_ca_path | default('/etc/ssl/certs/ca-certificates.crt') }}
|
|
|
|
- name: groq
|
|
connect_timeout: {{ upstream_connect_timeout | default('5s') }}
|
|
type: LOGICAL_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
load_assignment:
|
|
cluster_name: groq
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: api.groq.com
|
|
port_value: 443
|
|
hostname: "api.groq.com"
|
|
transport_socket:
|
|
name: envoy.transport_sockets.tls
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
|
|
sni: api.groq.com
|
|
common_tls_context:
|
|
tls_params:
|
|
tls_minimum_protocol_version: TLSv1_2
|
|
tls_maximum_protocol_version: TLSv1_3
|
|
validation_context:
|
|
trusted_ca:
|
|
filename: {{ upstream_tls_ca_path | default('/etc/ssl/certs/ca-certificates.crt') }}
|
|
|
|
- name: mistral
|
|
connect_timeout: {{ upstream_connect_timeout | default('5s') }}
|
|
type: LOGICAL_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
load_assignment:
|
|
cluster_name: mistral
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: api.mistral.ai
|
|
port_value: 443
|
|
hostname: "api.mistral.ai"
|
|
transport_socket:
|
|
name: envoy.transport_sockets.tls
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
|
|
sni: api.mistral.ai
|
|
common_tls_context:
|
|
tls_params:
|
|
tls_minimum_protocol_version: TLSv1_2
|
|
tls_maximum_protocol_version: TLSv1_3
|
|
validation_context:
|
|
trusted_ca:
|
|
filename: {{ upstream_tls_ca_path | default('/etc/ssl/certs/ca-certificates.crt') }}
|
|
|
|
- name: openai
|
|
connect_timeout: {{ upstream_connect_timeout | default('5s') }}
|
|
type: LOGICAL_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
load_assignment:
|
|
cluster_name: openai
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: api.openai.com
|
|
port_value: 443
|
|
hostname: "api.openai.com"
|
|
transport_socket:
|
|
name: envoy.transport_sockets.tls
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
|
|
sni: api.openai.com
|
|
common_tls_context:
|
|
tls_params:
|
|
tls_minimum_protocol_version: TLSv1_2
|
|
tls_maximum_protocol_version: TLSv1_3
|
|
validation_context:
|
|
trusted_ca:
|
|
filename: {{ upstream_tls_ca_path | default('/etc/ssl/certs/ca-certificates.crt') }}
|
|
- name: digitalocean
|
|
connect_timeout: {{ upstream_connect_timeout | default('5s') }}
|
|
type: LOGICAL_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
load_assignment:
|
|
cluster_name: digitalocean
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: inference.do-ai.run
|
|
port_value: 443
|
|
hostname: "inference.do-ai.run"
|
|
transport_socket:
|
|
name: envoy.transport_sockets.tls
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
|
|
sni: inference.do-ai.run
|
|
common_tls_context:
|
|
tls_params:
|
|
tls_minimum_protocol_version: TLSv1_2
|
|
tls_maximum_protocol_version: TLSv1_3
|
|
validation_context:
|
|
trusted_ca:
|
|
filename: {{ upstream_tls_ca_path | default('/etc/ssl/certs/ca-certificates.crt') }}
|
|
- name: xiaomi
|
|
connect_timeout: {{ upstream_connect_timeout | default('5s') }}
|
|
type: LOGICAL_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
load_assignment:
|
|
cluster_name: xiaomi
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: api.xiaomimimo.com
|
|
port_value: 443
|
|
hostname: "api.xiaomimimo.com"
|
|
transport_socket:
|
|
name: envoy.transport_sockets.tls
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
|
|
sni: api.xiaomimimo.com
|
|
common_tls_context:
|
|
tls_params:
|
|
tls_minimum_protocol_version: TLSv1_2
|
|
tls_maximum_protocol_version: TLSv1_3
|
|
validation_context:
|
|
trusted_ca:
|
|
filename: {{ upstream_tls_ca_path | default('/etc/ssl/certs/ca-certificates.crt') }}
|
|
- name: mistral_7b_instruct
|
|
connect_timeout: 0.5s
|
|
type: STRICT_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
load_assignment:
|
|
cluster_name: mistral_7b_instruct
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: mistral_7b_instruct
|
|
port_value: 10001
|
|
hostname: "mistral_7b_instruct"
|
|
{% for cluster_name, cluster in plano_clusters.items() %}
|
|
- name: {{ cluster_name }}
|
|
{% if cluster.connect_timeout -%}
|
|
connect_timeout: {{ cluster.connect_timeout }}
|
|
{% else -%}
|
|
connect_timeout: {{ upstream_connect_timeout | default('5s') }}
|
|
{% endif -%}
|
|
{% if cluster.prefix_affinity -%}
|
|
# KV-aware replica stickiness: resolve every replica address and consistent-hash
|
|
# requests (by x-plano-prefix-hash, see the route-level hash_policy) so the same
|
|
# prompt prefix lands on the replica holding its warm KV cache.
|
|
type: STRICT_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: RING_HASH
|
|
{% else -%}
|
|
type: LOGICAL_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
{% endif -%}
|
|
load_assignment:
|
|
cluster_name: {{ cluster_name }}
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: {{ cluster.endpoint }}
|
|
port_value: {{ cluster.port }}
|
|
{% if cluster.http_host %}
|
|
hostname: {{ cluster.http_host }}
|
|
{% else %}
|
|
hostname: {{ cluster.endpoint }}
|
|
{% endif %}
|
|
{% if cluster.protocol == "https" %}
|
|
transport_socket:
|
|
name: envoy.transport_sockets.tls
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
|
|
sni: {{ cluster.endpoint }}
|
|
common_tls_context:
|
|
tls_params:
|
|
tls_minimum_protocol_version: TLSv1_2
|
|
tls_maximum_protocol_version: TLSv1_3
|
|
validation_context:
|
|
trusted_ca:
|
|
filename: {{ upstream_tls_ca_path | default('/etc/ssl/certs/ca-certificates.crt') }}
|
|
{% endif %}
|
|
{% endfor %}
|
|
|
|
{% for local_llm_provider in local_llms %}
|
|
- name: {{ local_llm_provider.cluster_name }}
|
|
connect_timeout: {{ upstream_connect_timeout | default('5s') }}
|
|
type: LOGICAL_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
load_assignment:
|
|
cluster_name: {{ local_llm_provider.cluster_name }}
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: {{ local_llm_provider.endpoint }}
|
|
port_value: {{ local_llm_provider.port }}
|
|
{% if local_llm_provider.http_host %}
|
|
hostname: {{ local_llm_provider.http_host }}
|
|
{% else %}
|
|
hostname: {{ local_llm_provider.endpoint }}
|
|
{% endif %}
|
|
{% if local_llm_provider.protocol == "https" %}
|
|
transport_socket:
|
|
name: envoy.transport_sockets.tls
|
|
typed_config:
|
|
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
|
|
sni: {{ local_llm_provider.endpoint }}
|
|
common_tls_context:
|
|
tls_params:
|
|
tls_minimum_protocol_version: TLSv1_2
|
|
tls_maximum_protocol_version: TLSv1_3
|
|
validation_context:
|
|
trusted_ca:
|
|
filename: {{ upstream_tls_ca_path | default('/etc/ssl/certs/ca-certificates.crt') }}
|
|
{% endif %}
|
|
|
|
{% endfor %}
|
|
- name: arch_internal
|
|
connect_timeout: 0.5s
|
|
type: LOGICAL_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
load_assignment:
|
|
cluster_name: arch_internal
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: 0.0.0.0
|
|
port_value: 11000
|
|
hostname: arch_internal
|
|
|
|
- name: bright_staff
|
|
connect_timeout: 0.5s
|
|
type: LOGICAL_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
load_assignment:
|
|
cluster_name: bright_staff
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: 0.0.0.0
|
|
port_value: 9091
|
|
hostname: localhost
|
|
|
|
- name: arch_prompt_gateway_listener
|
|
connect_timeout: 0.5s
|
|
type: LOGICAL_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
load_assignment:
|
|
cluster_name: arch_prompt_gateway_listener
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: 0.0.0.0
|
|
port_value: 10001
|
|
hostname: arch_prompt_gateway_listener
|
|
|
|
- name: arch_listener_llm
|
|
connect_timeout: 0.5s
|
|
type: LOGICAL_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
load_assignment:
|
|
cluster_name: arch_listener_llm
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
address: 0.0.0.0
|
|
port_value: 12001
|
|
hostname: arch_listener_llm
|
|
|
|
{% if "random_sampling" in plano_tracing and plano_tracing["random_sampling"] > 0 %}
|
|
- name: opentelemetry_collector
|
|
type: STRICT_DNS
|
|
dns_lookup_family: V4_ONLY
|
|
lb_policy: ROUND_ROBIN
|
|
typed_extension_protocol_options:
|
|
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
|
|
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
|
|
explicit_http_config:
|
|
http2_protocol_options: {}
|
|
load_assignment:
|
|
cluster_name: opentelemetry_collector
|
|
endpoints:
|
|
- lb_endpoints:
|
|
- endpoint:
|
|
address:
|
|
socket_address:
|
|
{% set _otel_endpoint = plano_tracing.opentracing_grpc_endpoint | default('host.docker.internal:4317') | replace("http://", "") | replace("https://", "") %}
|
|
address: {{ _otel_endpoint.split(":") | first }}
|
|
port_value: {{ _otel_endpoint.split(":") | last }}
|
|
{% endif %}
|