plano/demos/weather-forecast-local-llm

Weather forecasting

This demo shows how you can use intelligent prompt gateway to provide realtime weather forecast using Mistral LLM locally hosted using llama.cpp as LLM Hosting Service.

Startig the demo

1. Ensure that submodule is up to date

   git submodule sync --recursive
   

2. Download mistral 7b model using following shell command

   sh download_mistral_7b.sh
   

3. Start services

   docker compose up
   

4. Navigate to http://localhost:18080/
5. You can type in queries like "how is the weather in Seattle"
   1. You can also ask follow up questions like "show me sunny days"
6. To see metrics navigate to "http://localhost:3000/" (use admin/grafana for login)
   1. Open up dahsboard named "Intelligent Gateway Overview"
   2. On this dashboard you can see reuqest latency and number of requests