Test semantic caching with the given setup #112

New issue

Closed

opened 2026-06-10 11:39:20 +02:00 by alpha-nerd · 2 comments

alpha-nerd commented

2026-06-10 11:39:20 +02:00

Owner

Hi @alpha-nerd, I just tested the rc3 version.
I didn't see any issues. However, the caching doesn't seem to take effect.
Currently, I am using exact cache match:

cache_enabled: true
cache_backend: sqlite    # persists across restarts
cache_similarity: 1.0   # exact match only, no sentence-transformers needed
cache_ttl: 3600

Which is also acknowledged in the log: [cache] Exact-match cache ready (backend=sqlite)
But running the same /api/chat request twice doesn't trigger caching:

INFO:     Application startup complete.
[cache] Exact-match cache ready (backend=sqlite)
INFO:     Uvicorn running on http://0.0.0.0:12434 (Press CTRL+C to quit)
INFO:     192.168.4.9:36954 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:36954 - "GET /api/config HTTP/1.1" 200 OK
INFO:     192.168.4.9:33636 - "GET /api/ps_details HTTP/1.1" 200 OK
INFO:     192.168.4.9:33636 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:35576 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:57096 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:50286 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:39464 - "GET /api/ps_details HTTP/1.1" 200 OK
INFO:     192.168.4.9:39464 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:49350 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:43078 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:38042 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:51264 - "POST /api/chat HTTP/1.1" 200 OK
INFO:     192.168.4.9:35042 - "GET /api/ps_details HTTP/1.1" 200 OK
INFO:     192.168.4.9:35042 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:42246 - "GET /api/affinity_stats HTTP/1.1" 200 OK
INFO:     192.168.4.9:42246 - "POST /api/chat HTTP/1.1" 200 OK

Originally posted by @JTHesse in #106 (comment)

Hi @alpha-nerd, I just tested the rc3 version. I didn't see any issues. However, the caching doesn't seem to take effect. Currently, I am using exact cache match: ``` cache_enabled: true cache_backend: sqlite # persists across restarts cache_similarity: 1.0 # exact match only, no sentence-transformers needed cache_ttl: 3600 ``` Which is also acknowledged in the log: `[cache] Exact-match cache ready (backend=sqlite)` But running the same /api/chat request twice doesn't trigger caching: ``` INFO: Application startup complete. [cache] Exact-match cache ready (backend=sqlite) INFO: Uvicorn running on http://0.0.0.0:12434 (Press CTRL+C to quit) INFO: 192.168.4.9:36954 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:36954 - "GET /api/config HTTP/1.1" 200 OK INFO: 192.168.4.9:33636 - "GET /api/ps_details HTTP/1.1" 200 OK INFO: 192.168.4.9:33636 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:35576 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:57096 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:50286 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:39464 - "GET /api/ps_details HTTP/1.1" 200 OK INFO: 192.168.4.9:39464 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:49350 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:43078 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:38042 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:51264 - "POST /api/chat HTTP/1.1" 200 OK INFO: 192.168.4.9:35042 - "GET /api/ps_details HTTP/1.1" 200 OK INFO: 192.168.4.9:35042 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:42246 - "GET /api/affinity_stats HTTP/1.1" 200 OK INFO: 192.168.4.9:42246 - "POST /api/chat HTTP/1.1" 200 OK ``` _Originally posted by @JTHesse in https://bitfreedom.net/code/nomyo-ai/nomyo-router/issues/106#issuecomment-6299_

alpha-nerd referenced this issue

2026-06-10 11:40:49 +02:00

RC-3 CA testing #106

alpha-nerd commented

2026-06-10 11:49:41 +02:00

Author

Owner

@JTHesse
one question:

did you use the per request opt-in?

this is an additional guard for the cache feature as it potentially can leak privacy sensitive data to foreign users.

i.e. your system uses a memory, that memorizes users and injects automatically user info ->

user A prompt: What is the capital of france?
system injects metadata
llm reponse: The capital of france is Paris, A!

if user X now does the very same prompt, the router will not forward the prompt to the llm, but respond from cache with "The capital of france is Paris, A!", which leaked the data of user A to user N.

Hence why, to make sure this is understood it requires both configuration and per-request opt-in.
Safe to use, where you only have general user prompts ;)

@JTHesse one question: - did you use the per [request opt-in](https://bitfreedom.net/code/nomyo-ai/nomyo-router/src/branch/main/doc/semantic-cache.md#per-request-opt-in)? this is an additional guard for the cache feature as it potentially can leak privacy sensitive data to foreign users. i.e. your system uses a memory, that memorizes users and injects automatically user info -> user A prompt: What is the capital of france? system injects metadata llm reponse: The capital of france is Paris, A! if user X now does the very same prompt, the router will not forward the prompt to the llm, but respond from cache with "The capital of france is Paris, A!", which leaked the data of user A to user N. Hence why, to make sure this is understood it requires both configuration and per-request opt-in. Safe to use, where you only have general user prompts ;)

JTHesse commented

2026-06-10 14:07:32 +02:00

You are right, with the request opt-in caching is working as expected.
I initially thought by reading the docs that this was an additional possibility to activate caching, but as a guard this makes total sense ;) Thanks for clarifying.

You are right, with the request opt-in caching is working as expected. I initially thought by reading the docs that this was an additional possibility to activate caching, but as a guard this makes total sense ;) Thanks for clarifying.

alpha-nerd closed this issue

2026-06-10 15:47:57 +02:00