apunkt/plano
1
0
Fork 0
mirror of https://github.com/katanemo/plano.git synced 2026-04-27 09:46:28 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
plano/demos/use_cases/preference_based_routing
History
Salman Paracha 2941392ed1
Adding support for wildcard models in the model_providers config (#696) 
* cleaning up plano cli commands

* adding support for wildcard model providers

* fixing compile errors

* fixing bugs related to default model provider, provider hint and duplicates in the model provider list

* fixed cargo fmt issues

* updating tests to always include the model id

* using default for the prompt_gateway path

* fixed the model name, as gpt-5-mini-2025-08-07 wasn't in the config

* making sure that all aliases and models match the config

* fixed the config generator to allow for base_url providers LLMs to include wildcard models

* re-ran the models list utility and added a shell script to run it

* updating docs to mention wildcard model providers

* updated provider_models.json to yaml, added that file to our docs for reference

* updating the build docs to use the new root-based build

---------

Co-authored-by: Salman Paracha <salmanparacha@MacBook-Pro-342.local>
2026-01-28 17:47:33 -08:00
..
hurl_tests Adding support for wildcard models in the model_providers config (#696) 2026-01-28 17:47:33 -08:00
arch_config_local.yaml better model names (#517) 2025-07-11 16:42:16 -07:00
config.yaml rename cli to plano (#647) 2025-12-23 18:37:58 -08:00
docker-compose.yaml rename cli to plano (#647) 2025-12-23 18:37:58 -08:00
README.md release 0.4.3 (#701) 2026-01-18 00:07:46 -08:00
test_router_endpoint.rest Adding support for wildcard models in the model_providers config (#696) 2026-01-28 17:47:33 -08:00

README.md

Usage based LLM Routing

This demo shows how you can use user preferences to route user prompts to appropriate llm. See config.yaml for details on how you can define user preferences.

How to start the demo

Make sure your machine is up to date with latest version of plano. And you have activated the virtual environment.

  1. start the openwebui
(venv) $ cd demos/use_cases/preference_based_routing
(venv) $ docker compose up -d
  1. start plano in the foreground
(venv) $ planoai up --service plano --foreground
# Or if installed with uv: uvx planoai up --service plano --foreground
2025-05-30 18:00:09,953 - planoai.main - INFO - Starting plano cli version: 0.4.3
2025-05-30 18:00:09,953 - planoai.main - INFO - Validating /Users/adilhafeez/src/intelligent-prompt-gateway/demos/use_cases/preference_based_routing/config.yaml
2025-05-30 18:00:10,422 - cli.core - INFO - Starting arch gateway, image name: plano, tag: katanemo/plano:0.4.3
2025-05-30 18:00:10,662 - cli.core - INFO - plano status: running, health status: starting
2025-05-30 18:00:11,712 - cli.core - INFO - plano status: running, health status: starting
2025-05-30 18:00:12,761 - cli.core - INFO - plano is running and is healthy!
...
  1. open openwebui http://localhost:8080/

Testing out preference based routing

We have defined two routes 1. code generation and 2. code understanding

For code generation query LLM that is better suited for code generation wil handle the request,

If you look at the logs you'd see that code generation llm was selected,

...
2025-05-31T01:02:19.382716Z  INFO brightstaff::router::llm_router: router response: {'route': 'code_generation'}, response time: 203ms
...
image

Now if you ask for query related to code understanding you'd see llm that is better suited to handle code understanding in handled,

...
2025-05-31T01:06:33.555680Z  INFO brightstaff::router::llm_router: router response: {'route': 'code_understanding'}, response time: 327ms
...
image