mirror of
https://github.com/asg017/sqlite-vec.git
synced 2026-04-25 08:46:49 +02:00
70 lines
2 KiB
Markdown
70 lines
2 KiB
Markdown
|
# bench-delete: Recall degradation after random deletion
|
||
|
|
|
||
|
|
Measures how KNN recall changes after deleting a random percentage of rows
|
||
|
|
from different index types (flat, rescore, DiskANN).
|
||
|
|
|
||
|
|
## Quick start
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Ensure dataset exists
|
||
|
|
make -C ../datasets/cohere1m
|
||
|
|
|
||
|
|
# Ensure extension is built
|
||
|
|
make -C ../.. loadable
|
||
|
|
|
||
|
|
# Quick smoke test
|
||
|
|
make smoke
|
||
|
|
|
||
|
|
# Full benchmark at 10k vectors
|
||
|
|
make bench-10k
|
||
|
|
```
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
```bash
|
||
|
|
python bench_delete.py --subset-size 10000 --delete-pct 10,25,50,75 \
|
||
|
|
"flat:type=vec0-flat,variant=float" \
|
||
|
|
"diskann-R72:type=diskann,R=72,L=128,quantizer=binary" \
|
||
|
|
"rescore-bit:type=rescore,quantizer=bit,oversample=8"
|
||
|
|
```
|
||
|
|
|
||
|
|
## What it measures
|
||
|
|
|
||
|
|
For each config and delete percentage:
|
||
|
|
|
||
|
|
| Metric | Description |
|
||
|
|
|--------|-------------|
|
||
|
|
| **recall** | KNN recall@k after deletion (ground truth recomputed over surviving rows) |
|
||
|
|
| **delta** | Recall change vs 0% baseline |
|
||
|
|
| **query latency** | Mean/median query time after deletion |
|
||
|
|
| **db_size_mb** | DB file size before VACUUM |
|
||
|
|
| **vacuum_size_mb** | DB file size after VACUUM (space reclaimed) |
|
||
|
|
| **delete_time_s** | Wall time for the DELETE operations |
|
||
|
|
|
||
|
|
## How it works
|
||
|
|
|
||
|
|
1. Build index with N vectors (one copy per config)
|
||
|
|
2. Measure recall at k=10 (pre-delete baseline)
|
||
|
|
3. For each delete %:
|
||
|
|
- Copy the master DB
|
||
|
|
- Delete a random selection of rows (deterministic seed)
|
||
|
|
- Measure recall (ground truth recomputed over surviving rows only)
|
||
|
|
- VACUUM and measure size savings
|
||
|
|
4. Print comparison table
|
||
|
|
|
||
|
|
## Expected behavior
|
||
|
|
|
||
|
|
- **Flat index**: Recall should be 1.0 at all delete percentages (brute-force is always exact)
|
||
|
|
- **Rescore**: Recall should stay close to baseline (quantized scan + rescore is robust)
|
||
|
|
- **DiskANN**: Recall may degrade at high delete % due to graph fragmentation (dangling edges, broken connectivity)
|
||
|
|
|
||
|
|
## Results DB
|
||
|
|
|
||
|
|
Results are stored in `runs/<dataset>/<subset_size>/delete_results.db`:
|
||
|
|
|
||
|
|
```sql
|
||
|
|
SELECT config_name, delete_pct, recall, vacuum_size_mb
|
||
|
|
FROM delete_runs
|
||
|
|
ORDER BY config_name, delete_pct;
|
||
|
|
```
|