plano/demos/llm_routing

LLM Routing

This demo shows how you can arch gateway to manage keys and route to appropricate LLM.

Starting the demo

1. Please make sure the pre-requisites are installed correctly
2. Start Arch

   sh run_demo.sh
   

3. Navigate to http://localhost:18080/

Observability

Arch gateway publishes stats endpoint at http://localhost:19901/stats. In this demo we are using prometheus to pull stats from arch and we are using grafana to visualize the stats in dashboard. To see grafana dashboard follow instructions below,

1. Navigate to http://localhost:3000/ to open grafana UI (use admin/grafana as credentials)
2. From grafana left nav click on dashboards and select "Intelligent Gateway Overview" to view arch gateway stats

Selecting different LLM

You can pick different LLM based on header x-arch-llm-provider-hint to override default LLM. Or you can use chatbot UI to override LLM.

Here is a preview of chat UI,

And you can also see tracing using jaeger UI that we start by heading over to http://localhost:16686/