mirror of
https://github.com/katanemo/plano.git
synced 2026-04-26 01:06:25 +02:00
20 lines
933 B
Markdown
20 lines
933 B
Markdown
|
# LLM Routing
|
||
|
|
This demo shows how you can arch gateway to manage keys and route to appropricate LLM.
|
||
|
|
|
||
|
|
# Starting the demo
|
||
|
|
1. Please make sure the [pre-requisites](https://github.com/katanemo/arch/?tab=readme-ov-file#prerequisites) are installed correctly
|
||
|
|
1. Start Arch
|
||
|
|
```sh
|
||
|
|
sh run_demo.sh
|
||
|
|
```
|
||
|
|
1. Navigate to http://localhost:18080/
|
||
|
|
|
||
|
|
# Observability
|
||
|
|
Arch gateway publishes stats endpoint at http://localhost:19901/stats. In this demo we are using prometheus to pull stats from arch and we are using grafana to visalize the stats in dashboard. To see grafana dashboard follow instructions below,
|
||
|
|
|
||
|
|
1. Navigate to http://localhost:3000/ to open grafana UI (use admin/grafana as credentials)
|
||
|
|
1. From grafana left nav click on dashboards and select "Intelligent Gateway Overview" to view arch gateway stats
|
||
|
|
|
||
|
|
# Selecting different LLM
|
||
|
|
You can pick different LLM based on header `x-arch-llm-provider-hint` to override default LLM.
|