Add FTS5-style command column and runtime oversample for rescore

Replace the old INSERT INTO t(rowid) VALUES('command') hack with a
proper hidden command column named after the table (FTS5 pattern):

  INSERT INTO t(t) VALUES ('oversample=16')

The command column is the first hidden column (before distance and k)
to reserve ability for future table-valued function argument use.

Schema: CREATE TABLE x(rowid, <cols>, "<table>" hidden, distance hidden, k hidden)

For backwards compat, pre-v0.1.10 tables (detected via _info shadow
table version) skip the command column to avoid name conflicts with
user columns that may share the table's name. Verified with legacy
fixture DB generated by sqlite-vec v0.1.6.

Changes:
- Add hidden command column to sqlite3_declare_vtab for new tables
- Version-gate via _info shadow table for existing tables
- Validate at CREATE time that no column name matches table name
- Add rescore_handle_command() with oversample=N support
- rescore_knn() prefers runtime oversample_search over CREATE default
- Remove old rowid-based command dispatch
- Migrate all DiskANN/IVF/fuzz tests and benchmarks to new syntax
- Add legacy DB fixture (v0.1.6) and 9 backwards-compat tests

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alex Garcia 2026-03-31 22:39:18 -07:00
parent b7fc459be4
commit 6e2c4c6bab
21 changed files with 512 additions and 105 deletions

View file

@ -589,7 +589,7 @@ def test_diskann_command_search_list_size(db):
assert len(results_before) == 5
# Override search_list_size_search at runtime
db.execute("INSERT INTO t(rowid) VALUES ('search_list_size_search=256')")
db.execute("INSERT INTO t(t) VALUES ('search_list_size_search=256')")
# Query should still work
results_after = db.execute(
@ -598,14 +598,14 @@ def test_diskann_command_search_list_size(db):
assert len(results_after) == 5
# Override search_list_size_insert at runtime
db.execute("INSERT INTO t(rowid) VALUES ('search_list_size_insert=32')")
db.execute("INSERT INTO t(t) VALUES ('search_list_size_insert=32')")
# Inserts should still work
vec = struct.pack("64f", *[random.random() for _ in range(64)])
db.execute("INSERT INTO t(emb) VALUES (?)", [vec])
# Override unified search_list_size
db.execute("INSERT INTO t(rowid) VALUES ('search_list_size=64')")
db.execute("INSERT INTO t(t) VALUES ('search_list_size=64')")
results_final = db.execute(
"SELECT rowid, distance FROM t WHERE emb MATCH ? AND k = 5", [query]
@ -620,9 +620,9 @@ def test_diskann_command_search_list_size_error(db):
emb float[64] INDEXED BY diskann(neighbor_quantizer=binary)
)
""")
result = exec(db, "INSERT INTO t(rowid) VALUES ('search_list_size=0')")
result = exec(db, "INSERT INTO t(t) VALUES ('search_list_size=0')")
assert "error" in result
result = exec(db, "INSERT INTO t(rowid) VALUES ('search_list_size=-1')")
result = exec(db, "INSERT INTO t(t) VALUES ('search_list_size=-1')")
assert "error" in result