apunkt/plano
1
0
Fork 0
mirror of https://github.com/katanemo/plano.git synced 2026-04-26 17:26:26 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

move custom tracer to llm filter (#267)

Browse source
This commit is contained in:
Adil Hafeez 2024-11-15 10:44:01 -08:00 • committed by GitHub
parent 1d229cba8f
commit d3c17c7abd
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
22 changed files with 335 additions and 133 deletions
Download patch file Download diff file Expand all files Collapse all files

19
demos/llm_routing/README.md Normal file
View file

@ -0,0 +1,19 @@
# LLM Routing
This demo shows how you can arch gateway to manage keys and route to appropricate LLM.
# Starting the demo
1. Please make sure the [pre-requisites](https://github.com/katanemo/arch/?tab=readme-ov-file#prerequisites) are installed correctly
1. Start Arch
```sh
sh run_demo.sh
```
1. Navigate to http://localhost:18080/
# Observability
Arch gateway publishes stats endpoint at http://localhost:19901/stats. In this demo we are using prometheus to pull stats from arch and we are using grafana to visalize the stats in dashboard. To see grafana dashboard follow instructions below,
1. Navigate to http://localhost:3000/ to open grafana UI (use admin/grafana as credentials)
1. From grafana left nav click on dashboards and select "Intelligent Gateway Overview" to view arch gateway stats
# Selecting different LLM
You can pick different LLM based on header `x-arch-llm-provider-hint` to override default LLM.