apunkt/plano
1
0
Fork 0
mirror of https://github.com/katanemo/plano.git synced 2026-04-25 16:56:24 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 3
plano/demos/llm_routing/README.md
Adil Hafeez d3c17c7abd
move custom tracer to llm filter (#267)
2024-11-15 10:44:01 -08:00

933 B
Raw Blame History

LLM Routing

This demo shows how you can arch gateway to manage keys and route to appropricate LLM.

Starting the demo

  1. Please make sure the pre-requisites are installed correctly
  2. Start Arch 
    sh run_demo.sh
  3. Navigate to http://localhost:18080/

Observability

Arch gateway publishes stats endpoint at http://localhost:19901/stats. In this demo we are using prometheus to pull stats from arch and we are using grafana to visalize the stats in dashboard. To see grafana dashboard follow instructions below,

  1. Navigate to http://localhost:3000/ to open grafana UI (use admin/grafana as credentials)
  2. From grafana left nav click on dashboards and select "Intelligent Gateway Overview" to view arch gateway stats

Selecting different LLM

You can pick different LLM based on header x-arch-llm-provider-hint to override default LLM.