mirror of https://github.com/katanemo/plano.git synced 2026-06-17 15:25:17 +02:00

Adil Hafeez 16a86c0f3b Fix llm_routing provider element We replaced provider with provider_interface to make it more clear to developers about provider api/backend being used. During that upgrade we removed support for mistral in provider to encourage developers to start using provider_interface. But this demo was not updated to use provider_interface as it was using mistral. This code change fixes it by replacing provider with provider_interface. Signed-off-by: Adil Hafeez <adil.hafeez@gmail.com>		2025-01-24 15:02:37 -08:00
..
arch_config.yaml	Fix llm_routing provider element	2025-01-24 15:02:37 -08:00
docker-compose.yaml	move custom tracer to llm filter (#267 )	2024-11-15 10:44:01 -08:00
README.md	Fix llm_routing provider element	2025-01-24 15:02:37 -08:00
run_demo.sh	Add run_demo.sh for llm_routing demo (#381 )	2025-01-24 14:41:51 -08:00

README.md

LLM Routing

This demo shows how you can arch gateway to manage keys and route to appropricate LLM.

Starting the demo

Please make sure the pre-requisites are installed correctly
Start Arch
```
sh run_demo.sh
```
Navigate to http://localhost:18080/

Observability

Arch gateway publishes stats endpoint at http://localhost:19901/stats. In this demo we are using prometheus to pull stats from arch and we are using grafana to visualize the stats in dashboard. To see grafana dashboard follow instructions below,

Navigate to http://localhost:3000/ to open grafana UI (use admin/grafana as credentials)
From grafana left nav click on dashboards and select "Intelligent Gateway Overview" to view arch gateway stats

Selecting different LLM

You can pick different LLM based on header x-arch-llm-provider-hint to override default LLM. Or you can use chatbot UI to override LLM.

Here is a preview of chat UI,

And you can also see tracing using jaeger UI that we start by heading over to http://localhost:16686/