apunkt/plano
1
0
Fork 0
mirror of https://github.com/katanemo/plano.git synced 2026-07-02 15:51:02 +02:00
Code Issues Projects Releases Packages Wiki Activity
plano/demos/llm_routing/preference_based_routing
History
Adil Hafeez 5a4f8cb321
Standardize demo configs and CLI templates to v0.3.0 
- Update all demo config files to version v0.3.0
- Rename llm_providers → model_providers across all configs
- Modernize listener format from object to array style
  (e.g. listeners: [{type: model, name:, port:}])
- Replace chatbot_ui with AnythingLLM in 6 docker-compose files
- Remove prometheus/grafana services from llm_gateway and
  preference_based_routing docker-compose files
- Add tracing: random_sampling: 100 to configs that lacked it
- Update 3 CLI init templates (coding_agent_routing,
  preference_aware_routing, conversational_state_v1_responses)
- Fix test_init.py assertion for model_providers key

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 01:43:59 -08:00
..
hurl_tests Clean up dead demos and restructure directory layout 2026-02-15 01:43:31 -08:00
config.yaml Standardize demo configs and CLI templates to v0.3.0 2026-02-15 01:43:59 -08:00
docker-compose.yaml Standardize demo configs and CLI templates to v0.3.0 2026-02-15 01:43:59 -08:00
plano_config_local.yaml Standardize demo configs and CLI templates to v0.3.0 2026-02-15 01:43:59 -08:00
README.md Clean up dead demos and restructure directory layout 2026-02-15 01:43:31 -08:00
test_router_endpoint.rest Clean up dead demos and restructure directory layout 2026-02-15 01:43:31 -08:00

README.md

Usage based LLM Routing

This demo shows how you can use user preferences to route user prompts to appropriate llm. See config.yaml for details on how you can define user preferences.

How to start the demo

Make sure your machine is up to date with latest version of plano. And you have activated the virtual environment.

  1. start anythingllm
(venv) $ cd demos/use_cases/preference_based_routing
(venv) $ docker compose up -d
  1. start plano in the foreground
(venv) $ planoai up --service plano --foreground
# Or if installed with uv: uvx planoai up --service plano --foreground
2025-05-30 18:00:09,953 - planoai.main - INFO - Starting plano cli version: 0.4.6
2025-05-30 18:00:09,953 - planoai.main - INFO - Validating /Users/adilhafeez/src/intelligent-prompt-gateway/demos/use_cases/preference_based_routing/config.yaml
2025-05-30 18:00:10,422 - cli.core - INFO - Starting plano gateway, image name: plano, tag: katanemo/plano:0.4.6
2025-05-30 18:00:10,662 - cli.core - INFO - plano status: running, health status: starting
2025-05-30 18:00:11,712 - cli.core - INFO - plano status: running, health status: starting
2025-05-30 18:00:12,761 - cli.core - INFO - plano is running and is healthy!
...
  1. open AnythingLLM http://localhost:3001/

Testing out preference based routing

We have defined two routes 1. code generation and 2. code understanding

For code generation query LLM that is better suited for code generation wil handle the request,

If you look at the logs you'd see that code generation llm was selected,

...
2025-05-31T01:02:19.382716Z  INFO brightstaff::router::llm_router: router response: {'route': 'code_generation'}, response time: 203ms
...
image

Now if you ask for query related to code understanding you'd see llm that is better suited to handle code understanding in handled,

...
2025-05-31T01:06:33.555680Z  INFO brightstaff::router::llm_router: router response: {'route': 'code_understanding'}, response time: 327ms
...
image