SurfSense/plans/git-native-kb/04-derived-index.md

Phase 4 — Derived index + reindex

Can build alongside Phase 3 (both only need Phase 1's commit/log). Umbrella: 00-umbrella-plan.md.

Objective

Make Postgres a derived, rebuildable chunk/embedding index of the git repo: incremental on each commit (keyed by blob SHA), fully reproducible via one reindex(workspace). Delete the three hand-rolled versioning systems — git history replaces them.

Locked model

• Post-commit incremental index. On each commit, diff the tree against the parent, and for changed blobs re-chunk + re-embed. Key embeddings by git blob SHA: unchanged blob → same SHA → reuse existing vectors (no re-embed). This is the correct, native form of what indexing_pipeline/chunk_reconciler.py::reconcile already approximates (it matches by chunk text; blob SHA generalizes it to file identity).
• Postgres is disposable. A single idempotent reindex(workspace) wipes chunks/embeddings and rebuilds from git HEAD (the Fossil rebuild discipline). Search (shared/retrieval/hybrid_search.py) is unchanged — it reads the same chunks table.
• History = git. Delete utils/document_versioning.py (DocumentVersion), services/revert_service.py + DocumentRevision/FolderRevision (Alembic migration to drop tables). git log/diff/revert/blame replace them.

Work items

1. app/kb_git/indexer.py — index_commit(workspace_id, sha): tree-diff parent→sha, map changed paths→documents/chunks, re-chunk changed files, embed only new blob SHAs, upsert chunks; delete chunks for removed paths.
2. Blob-SHA reuse is new (there is no embedding cache today — embed_texts calls the model directly; current reuse is chunk_reconciler matching by chunk text). Add a (embedding_model_version, blob_sha) reuse layer, or generalize the reconciler from chunk-text identity to blob identity. content_hash is workspace-salted, so it is NOT the git blob SHA — do not alias them. See 00c-shared-contract.md C5.
3. reindex(workspace_id) — wipe + full rebuild from HEAD; wire a Celery task entrypoint (Celery already runs; keep long rebuilds off the API process).
4. Trigger index_commit off the Phase-3 commit event.
5. Delete the three versioning systems + Alembic migration dropping document_versions, document_revisions, folder_revisions. This also removes the snapshot code embedded in commit_staged_filesystem_state (Phase 3) — coordinate the two.

Tests

• Edit one file → only its chunks re-embed; other files' vectors are byte-identical (blob-SHA reuse verified).
• reindex(workspace) produces a chunk set identical to the incremental path (determinism).
• Search results for a fixed query match the pre-pivot baseline for the same content.
• Deleting a file removes its chunks; renaming preserves vectors (same blob SHA).
• Post-migration: version/revert endpoints and tables are gone; no import references remain.

Out of scope

• Live connectors (Slack/Gmail) — never indexed.
• Zero row projection → Phase 6. Reranker/chunking strategy changes (separate search work).

Resolved (see 00c-shared-contract.md)

• content_hash vs blob SHA: they differ (content_hash is workspace-salted); key embedding reuse by blob SHA, keep content_hash through migration, drop later if redundant (C5).
• Cache location: none exists today — the reuse layer is new (C5).
• Where reindex runs: Celery task (C5).

Open questions

1. reindex progress/observability surface (log vs. status row) — cosmetic, not blocking.