plano/demos/function_calling

Function calling

This demo shows how you can use intelligent prompt gateway to do function calling. This demo assumes you are using ollama running natively. If you want to run ollama running inside docker then please update ollama endpoint in docker-compose file.

Startig the demo

1. Ensure that submodule is up to date

   git submodule sync --recursive
   

2. Create .env file and set OpenAI key using env var OPENAI_API_KEY
3. Start services

   docker compose up
   

4. Download Bolt-FC model. This demo assumes we have downloaded Bolt-Function-Calling-1B:Q4_K_M to local folder.
5. If running ollama natively run

   ollama serve
   

6. Create model file in ollama repository

   ollama create Bolt-Function-Calling-1B:Q4_K_M -f Bolt-FC-1B-Q4_K_M.model_file
   

7. Navigate to http://localhost:18080/
8. You can type in queries like "how is the weather in Seattle"
   • You can also ask follow up questions like "show me sunny days"
9. To see metrics navigate to "http://localhost:3000/" (use admin/grafana for login)
   • Open up dahsboard named "Intelligent Gateway Overview"
   • On this dashboard you can see reuqest latency and number of requests

Here is sample interaction,