feat: enhance load balancing
Yes a WRR describes this quite good. However, only if the model is already loaded.
I guess what I am describing could look like this:
if all(tracking_usage(ep) == 0 for ep in loaded_and_fr…
feat: enhance load balancing
Firstly thank you for looking into this. I can imagine that an ideal load balancing for different endpoints is pretty tricky.
Maybe prioritization should only occur if all endpoints are…
issue: api/version not found
feat: enhance load balancing