mirror of
https://github.com/katanemo/plano.git
synced 2026-07-11 16:12:13 +02:00
16a86c0f3b
We replaced provider with provider_interface to make it more clear to developers about provider api/backend being used. During that upgrade we removed support for mistral in provider to encourage developers to start using provider_interface. But this demo was not updated to use provider_interface as it was using mistral. This code change fixes it by replacing provider with provider_interface. Signed-off-by: Adil Hafeez <adil.hafeez@gmail.com>
27 lines
1.2 KiB
Markdown
27 lines
1.2 KiB
Markdown
# LLM Routing
|
|
This demo shows how you can arch gateway to manage keys and route to appropricate LLM.
|
|
|
|
# Starting the demo
|
|
1. Please make sure the [pre-requisites](https://github.com/katanemo/arch/?tab=readme-ov-file#prerequisites) are installed correctly
|
|
1. Start Arch
|
|
```sh
|
|
sh run_demo.sh
|
|
```
|
|
1. Navigate to http://localhost:18080/
|
|
|
|
# Observability
|
|
Arch gateway publishes stats endpoint at http://localhost:19901/stats. In this demo we are using prometheus to pull stats from arch and we are using grafana to visualize the stats in dashboard. To see grafana dashboard follow instructions below,
|
|
|
|
1. Navigate to http://localhost:3000/ to open grafana UI (use admin/grafana as credentials)
|
|
1. From grafana left nav click on dashboards and select "Intelligent Gateway Overview" to view arch gateway stats
|
|
|
|
# Selecting different LLM
|
|
You can pick different LLM based on header `x-arch-llm-provider-hint` to override default LLM. Or you can use chatbot UI to override LLM.
|
|
|
|
Here is a preview of chat UI,
|
|
|
|
|
|
|
|
And you can also see tracing using jaeger UI that we start by heading over to http://localhost:16686/
|
|
|
|
|