nomyo-ai/async-semantic-llm-cache
2
0
Fork 0
Code Issues Pull requests Projects Releases 2 Packages Wiki Activity Actions

doc: correction

Browse source
This commit is contained in:
Alpha Nerd 2026-03-06 16:00:19 +01:00
parent b33bb415dd
commit 3d8e5044d6
2 changed files with 31 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

54
README.md
View file

@ -2,7 +2,6 @@
**Async semantic caching for LLM API calls — reduce costs with one decorator.** **Async semantic caching for LLM API calls — reduce costs with one decorator.**
[![PyPI](https://img.shields.io/pypi/v/semantic-llm-cache)](https://pypi.org/project/semantic-llm-cache/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Python](https://img.shields.io/pypi/pyversions/semantic-llm-cache)](https://pypi.org/project/semantic-llm-cache/) [![Python](https://img.shields.io/pypi/pyversions/semantic-llm-cache)](https://pypi.org/project/semantic-llm-cache/)
@ -21,16 +20,17 @@ LLM API calls are expensive and slow. In production applications, **20-40% of pr
## What changed from the original ## What changed from the original
| Area | Original | This fork | | Area | Original | This fork |
| -------------------- | ------------------------- | ------------------------------------------------------------------- | | ---------------------- | --------------------------- | --------------------------------------------------------------------- |
| Backends | sync (`sqlite3`, `redis`) | async (`aiosqlite`, `redis.asyncio`) | | Backends | sync (`sqlite3`, `redis`) | async (`aiosqlite`, `redis.asyncio`) |
| `@cache` decorator | sync only | auto-detects async/sync | | `@cache` decorator | sync only | auto-detects async/sync |
| `EmbeddingCache` | sync `encode()` | adds `async aencode()` via `asyncio.to_thread` | | `EmbeddingCache` | sync`encode()` | adds`async aencode()` via `asyncio.to_thread` |
| `CacheContext` | sync only | supports both `with` and `async with` | | `CacheContext` | sync only | supports both`with` and `async with` |
| `CachedLLM` | `chat()` | adds `achat()` | | `CachedLLM` | `chat()` | adds`achat()` |
| Utility functions | sync | `clear_cache`, `invalidate`, `warm_cache`, `export_cache` all async | | Utility functions | sync | `clear_cache`, `invalidate`, `warm_cache`, `export_cache` all async |
| `StorageBackend` ABC | sync abstract methods | all abstract methods are `async def` | | `StorageBackend` ABC | sync abstract methods | all abstract methods are`async def` |
| Min Python | 3.9 | 3.10 (uses `X \| Y` union syntax) | | Min Python | 3.9 | 3.10 (uses`X | Y` union syntax) |
## Installation ## Installation
@ -198,14 +198,15 @@ async def my_llm_function(prompt: str) -> str:
### Parameters ### Parameters
| Parameter | Type | Default | Description |
| ------------ | ------------- | ----------- | --------------------------------------------------------- | | Parameter | Type | Default | Description |
| `similarity` | `float` | `1.0` | Cosine similarity threshold (1.0 = exact, 0.9 = semantic) | | -------------- | -------------- | ------------- | ----------------------------------------------------------- |
| `ttl` | `int \| None` | `3600` | Time-to-live in seconds (None = never expires) | | `similarity` | `float` | `1.0` | Cosine similarity threshold (1.0 = exact, 0.9 = semantic) |
| `backend` | `Backend` | `None` | Storage backend (None = in-memory) | | `ttl` | `int | None` | `3600` | Time-to-live in seconds (None = never expires) |
| `namespace` | `str` | `"default"` | Isolate different use cases | | `backend` | `Backend` | `None` | Storage backend (None = in-memory) |
| `enabled` | `bool` | `True` | Enable/disable caching | | `namespace` | `str` | `"default"` | Isolate different use cases |
| `key_func` | `Callable` | `None` | Custom cache key function | | `enabled` | `bool` | `True` | Enable/disable caching |
| `key_func` | `Callable` | `None` | Custom cache key function |
### Utility Functions ### Utility Functions
@ -221,19 +222,21 @@ from semantic_llm_cache.stats import (
## Backends ## Backends
| Backend | Description | I/O |
| --------------- | ------------------------------------ | ------------------------- | | Backend | Description | I/O |
| ----------------- | -------------------------------------- | ---------------------------- |
| `MemoryBackend` | In-memory LRU (default) | none — runs in event loop | | `MemoryBackend` | In-memory LRU (default) | none — runs in event loop |
| `SQLiteBackend` | Persistent, file-based (`aiosqlite`) | async non-blocking | | `SQLiteBackend` | Persistent, file-based (`aiosqlite`) | async non-blocking |
| `RedisBackend` | Distributed (`redis.asyncio`) | async non-blocking | | `RedisBackend` | Distributed (`redis.asyncio`) | async non-blocking |
## Embedding Providers ## Embedding Providers
| Provider | Quality | Notes |
| ----------------------------- | ---------------------------- | --------------------------- | | Provider | Quality | Notes |
| `DummyEmbeddingProvider` | hash-only, no semantic match | zero deps, default | | ------------------------------- | ------------------------------ | ---------------------------- |
| `SentenceTransformerProvider` | high (local model) | requires `[semantic]` extra | | `DummyEmbeddingProvider` | hash-only, no semantic match | zero deps, default |
| `OpenAIEmbeddingProvider` | high (API) | requires `[openai]` extra | | `SentenceTransformerProvider` | high (local model) | requires`[semantic]` extra |
| `OpenAIEmbeddingProvider` | high (API) | requires`[openai]` extra |
Embedding inference is offloaded via `asyncio.to_thread` — model loading is blocking and should be done at application startup, not on first request. Embedding inference is offloaded via `asyncio.to_thread` — model loading is blocking and should be done at application startup, not on first request.
@ -250,8 +253,9 @@ embedding = await embedding_cache.aencode("my prompt")
## Performance ## Performance
| Metric | Value | | Metric | Value |
| -------------------------- | ---------------------------------------- | | ---------------------------- | ------------------------------------------ |
| Cache hit latency | <10ms | | Cache hit latency | <10ms |
| Embedding overhead on miss | ~50ms (sentence-transformers, offloaded) | | Embedding overhead on miss | ~50ms (sentence-transformers, offloaded) |
| Typical hit rate | 25-40% | | Typical hit rate | 25-40% |

1
pyproject.toml
View file

@ -20,6 +20,7 @@ keywords = [
"openai", "openai",
"anthropic", "anthropic",
"ollama", "ollama",
"llama.cpp",
"prompt", "prompt",
"optimization", "optimization",
"cost-reduction", "cost-reduction",