RC-3 CA testing

alpha-nerd commented

2026-06-07 11:36:20 +02:00

Owner

@JTHesse the new rc-3 introduces a cache for clients to be reused rather than created for each single request.
The previous behaviour introduced a full init with the complete OS truststore for CA which is a heavy lift causing a performance ceiling.
The cache enables now 7x more req/s with concurrent users and cuts latency to 1/8 making the router in the same machine much more performant and responsive.
All tests pass, but as we are not using custom CA - and I believe you do - may I kindly ask once more for a real-world testing on your end.

docker pull bitfreedom.net/nomyo-ai/nomyo-router:0.9.9-rc3

highly appreciated.
if not possible, just close the issue.

@JTHesse the new rc-3 introduces a cache for clients to be reused rather than created for each single request. The previous behaviour introduced a full init with the complete OS truststore for CA which is a heavy lift causing a performance ceiling. The cache enables now 7x more req/s with concurrent users and cuts latency to 1/8 making the router in the same machine much more performant and responsive. All tests pass, but as we are not using custom CA - and I believe you do - may I kindly ask once more for a real-world testing on your end. `docker pull bitfreedom.net/nomyo-ai/nomyo-router:0.9.9-rc3` highly appreciated. if not possible, just close the issue.

alpha-nerd added the

help wanted

label 2026-06-07 11:36:20 +02:00

alpha-nerd self-assigned this 2026-06-07 11:36:32 +02:00

alpha-nerd commented

2026-06-07 11:42:11 +02:00

Author

Owner

reference: 3cd530586c

reference: 3cd530586c

JTHesse commented

2026-06-10 10:51:07 +02:00

Hi @alpha-nerd, I just tested the rc3 version.
I didn't see any issues. However, the caching doesn't seem to take effect.
Currently, I am using exact cache match:

cache_enabled: true
cache_backend: sqlite    # persists across restarts
cache_similarity: 1.0   # exact match only, no sentence-transformers needed
cache_ttl: 3600

Which is also acknowledged in the log: [cache] Exact-match cache ready (backend=sqlite)
But running the same /api/chat request twice doesn't trigger caching:

INFO:     Application startup complete.
[cache] Exact-match cache ready (backend=sqlite)
INFO:     Uvicorn running on http://0.0.0.0:12434 (Press CTRL+C to quit)
INFO:     192.168.4.9:36954 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:36954 - "GET /api/config HTTP/1.1" 200 OK
INFO:     192.168.4.9:33636 - "GET /api/ps_details HTTP/1.1" 200 OK
INFO:     192.168.4.9:33636 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:35576 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:57096 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:50286 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:39464 - "GET /api/ps_details HTTP/1.1" 200 OK
INFO:     192.168.4.9:39464 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:49350 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:43078 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:38042 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:51264 - "POST /api/chat HTTP/1.1" 200 OK
INFO:     192.168.4.9:35042 - "GET /api/ps_details HTTP/1.1" 200 OK
INFO:     192.168.4.9:35042 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:42246 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:42246 - "POST /api/chat HTTP/1.1" 200 OK

Hi @alpha-nerd, I just tested the rc3 version. I didn't see any issues. However, the caching doesn't seem to take effect. Currently, I am using exact cache match: ``` cache_enabled: true cache_backend: sqlite # persists across restarts cache_similarity: 1.0 # exact match only, no sentence-transformers needed cache_ttl: 3600 ``` Which is also acknowledged in the log: `[cache] Exact-match cache ready (backend=sqlite)` But running the same /api/chat request twice doesn't trigger caching: ``` INFO: Application startup complete. [cache] Exact-match cache ready (backend=sqlite) INFO: Uvicorn running on http://0.0.0.0:12434 (Press CTRL+C to quit) INFO: 192.168.4.9:36954 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:36954 - "GET /api/config HTTP/1.1" 200 OK INFO: 192.168.4.9:33636 - "GET /api/ps_details HTTP/1.1" 200 OK INFO: 192.168.4.9:33636 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:35576 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:57096 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:50286 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:39464 - "GET /api/ps_details HTTP/1.1" 200 OK INFO: 192.168.4.9:39464 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:49350 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:43078 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:38042 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:51264 - "POST /api/chat HTTP/1.1" 200 OK INFO: 192.168.4.9:35042 - "GET /api/ps_details HTTP/1.1" 200 OK INFO: 192.168.4.9:35042 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:42246 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:42246 - "POST /api/chat HTTP/1.1" 200 OK ```

alpha-nerd commented

2026-06-10 11:38:39 +02:00

Author

Owner

This seems to be a different issue,
the cache in this issue refers to a httpx client cache, that was refactored especially in light of TLS Certificate Authorities.
The code until now was re-creating a http-connection for each backend api request, which adds a lot of latency when using CAs, because the re-init of the connection will load the full OS CA truststore every single time.

This refactor now, establishes a connection pool for backend requests, I only couldn't test with a local CA.
Thought you were using this featue. Do you?

P.S.: opened #112 for the semantic-cache related issue

This seems to be a different issue, the cache in this issue refers to a httpx client cache, that was refactored especially in light of TLS Certificate Authorities. The code until now was re-creating a http-connection for each backend api request, which adds a lot of latency when using CAs, because the re-init of the connection will load the full OS CA truststore every single time. This refactor now, establishes a connection pool for backend requests, I only couldn't test with a local CA. Thought you were using this featue. Do you? P.S.: opened #112 for the semantic-cache related issue

JTHesse commented

2026-06-10 13:51:06 +02:00

Yes I am using custom CA, in that regard I didn't notice any issues and I guess you can close this issue ;)

👍 1

alpha-nerd commented

2026-06-10 15:47:22 +02:00

Author

Owner

Thanks for the confirmation

alpha-nerd closed this issue

2026-06-10 15:47:25 +02:00