apunkt/plano
1
0
Fork 0
mirror of https://github.com/katanemo/plano.git synced 2026-04-25 08:46:24 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

Add function calling support using bolt-fc-1b (#35)

Browse source
This commit is contained in:
Adil Hafeez 2024-09-10 14:24:46 -07:00 • committed by GitHub
parent fdfad87347
commit 7b5203a2ce
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
39 changed files with 1763 additions and 416 deletions
Download patch file Download diff file Expand all files Collapse all files

24
demos/function-calling/README.md Normal file
View file

@ -0,0 +1,24 @@
# Function calling
This demo shows how you can use intelligent prompt gateway to do function calling. This demo assumes you are using ollama running natively. If you want to run ollama running inside docker then please update ollama endpoint in docker-compose file.
# Startig the demo
1. Ensure that submodule is up to date
```sh
git submodule sync --recursive
```
1. Create `.env` file and set OpenAI key using env var `OPENAI_API_KEY`
1. Start services
```sh
docker compose up
```
1. Download Bolt-FC model. This demo assumes we have downloaded `Bolt-Function-Calling-1B:Q4_K_M` to local folder
2. Create model file in ollama repository
```sh
ollama create Bolt-Function-Calling-1B:Q4_K_M -f Bolt-FC-1B-Q4_K_M.model_file
```
3. Navigate to http://localhost:18080/
4. You can type in queries like "how is the weather in Seattle"
1. You can also ask follow up questions like "show me sunny days"
5. To see metrics navigate to "http://localhost:3000/" (use admin/grafana for login)
1. Open up dahsboard named "Intelligent Gateway Overview"
2. On this dashboard you can see reuqest latency and number of requests