In addition to this nomyo also sometimes tries to use models that are not promoted by the endpoint: `[generate_proxy] upstream error from (http://192.168.0.1:8000/v1, qwen2.5-coder:1.5b-base)…
You are right, with the request opt-in caching is working as expected. I initially thought by reading the docs that this was an additional possibility to activate caching, but as a guard this…
Yes I am using custom CA, in that regard I didn't notice any issues and I guess you can close this issue ;)
Hi @alpha-nerd, I just tested the rc3 version. I didn't see any issues. However, the caching doesn't seem to take effect. Currently, I am using exact cache match:
cache_enabled:…
I tested the new release and couldn't find any difference to the 0.9.1 version, which is good I guess :D
Hi @alpha-nerd, just returned from a sick-leave. I can't promise, but I will try to test it this week ;)
That is all right, just wanted to make sure that future tagged releases will stay available ;)
Thank you for the quick response. This explains the behavior, can this be fixed in a new release?
Yes a WRR describes this quite good. However, only if the model is already loaded.
I guess what I am describing could look like this:
if all(tracking_usage(ep) == 0 for ep in loaded_and_fr…