apunkt/sqlite-vec
1
0
Fork 0
mirror of https://github.com/asg017/sqlite-vec.git synced 2026-04-25 08:46:49 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Add FTS5-style command column and runtime oversample for rescore

Browse source 
Replace the old INSERT INTO t(rowid) VALUES('command') hack with a
proper hidden command column named after the table (FTS5 pattern):

  INSERT INTO t(t) VALUES ('oversample=16')

The command column is the first hidden column (before distance and k)
to reserve ability for future table-valued function argument use.

Schema: CREATE TABLE x(rowid, <cols>, "<table>" hidden, distance hidden, k hidden)

For backwards compat, pre-v0.1.10 tables (detected via _info shadow
table version) skip the command column to avoid name conflicts with
user columns that may share the table's name. Verified with legacy
fixture DB generated by sqlite-vec v0.1.6.

Changes:
- Add hidden command column to sqlite3_declare_vtab for new tables
- Version-gate via _info shadow table for existing tables
- Validate at CREATE time that no column name matches table name
- Add rescore_handle_command() with oversample=N support
- rescore_knn() prefers runtime oversample_search over CREATE default
- Remove old rowid-based command dispatch
- Migrate all DiskANN/IVF/fuzz tests and benchmarks to new syntax
- Add legacy DB fixture (v0.1.6) and 9 backwards-compat tests

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alex Garcia 2026-03-31 22:39:18 -07:00
parent b7fc459be4
commit 6e2c4c6bab
21 changed files with 512 additions and 105 deletions
Download patch file Download diff file Expand all files Collapse all files

25
sqlite-vec-rescore.c
View file

@ -351,7 +351,9 @@ static int rescore_knn(vec0_vtab *p, vec0_cursor *pCur,
(void)pCur;
(void)aMetadataIn;
int rc = SQLITE_OK;
int oversample = vector_column->rescore.oversample;
int oversample = vector_column->rescore.oversample_search > 0
? vector_column->rescore.oversample_search
: vector_column->rescore.oversample;
i64 k_oversample = k * oversample;
if (k_oversample > 4096)
k_oversample = 4096;
@ -640,6 +642,27 @@ cleanup:
return rc;
}
/**
* Handle FTS5-style command dispatch for rescore parameters.
* Returns SQLITE_OK if handled, SQLITE_EMPTY if not a rescore command.
*/
static int rescore_handle_command(vec0_vtab *p, const char *command) {
if (strncmp(command, "oversample=", 11) == 0) {
int val = atoi(command + 11);
if (val < 1) {
vtab_set_error(&p->base, "oversample must be >= 1");
return SQLITE_ERROR;
}
for (int i = 0; i < p->numVectorColumns; i++) {
if (p->vector_columns[i].index_type == VEC0_INDEX_TYPE_RESCORE) {
p->vector_columns[i].rescore.oversample_search = val;
}
}
return SQLITE_OK;
}
return SQLITE_EMPTY;
}
#ifdef SQLITE_VEC_TEST
void _test_rescore_quantize_float_to_bit(const float *src, uint8_t *dst, size_t dim) {
rescore_quantize_float_to_bit(src, dst, dim);