From 30aaa55fd0a194c3c1c7a2b9d573ce2131606d11 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Fri, 24 Jul 2026 19:16:21 +0200 Subject: [PATCH] docs: add v1 scope and consequences to KB ADR --- docs/adr/0001-git-native-knowledge-base.md | 63 ++++++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/docs/adr/0001-git-native-knowledge-base.md b/docs/adr/0001-git-native-knowledge-base.md index 3506d54ec..05fe0c6cb 100644 --- a/docs/adr/0001-git-native-knowledge-base.md +++ b/docs/adr/0001-git-native-knowledge-base.md @@ -105,3 +105,66 @@ Every decision traces to a proven, battle-tested reference. The only SurfSense-s ### The counter-example we explicitly reject **Wiki.js** git module = DB is truth, git is a two-way mirror. This creates two sources of truth and a reconciliation cursor; it has a known class of silent-sync bugs (see requarks/wiki discussion #7860). We use **one-way derivation** (git → Postgres), never two-way sync. + +--- + +## Scope: v1 (keep it simple) + +Ship the smallest thing that removes the maintenance pain: + +1. **Git repo per workspace** (dulwich) holding indexed content as markdown. +2. **One commit per agent turn / editor save.** No branches, no merge, no review workflow. +3. **Delete** the three versioning systems; history/undo = git log + `git revert`. +4. **pgvector stays**, rebuilt from git, keyed by blob SHA (point existing `chunk_reconciler` at blob SHA so unchanged files skip re-embedding). +5. **Per-workspace lock/queue** around commits (git is single-writer; this is a data-integrity boundary, not a feature — non-negotiable). +6. **One `reindex(workspace)` function** that wipes and rebuilds Postgres chunks from the git repo (the Fossil `rebuild` discipline — makes the system safe to ship early). + +### Deferred (v2+, explicitly out of v1) + +- Connect-your-own-remote (GitHub/GitLab) — free later because the repo is real git. +- CRDT / Yjs real-time collaboration. +- Review / merge workflows (kherad's reviewer layer). +- Graphiti / bi-temporal fact graph. +- Karpathy `raw/` + `wiki/` content model, contradiction-flagging, lint. + +--- + +## Consequences + +### Positive + +- Large net **deletion** of bespoke code (the maintenance win). +- Storage and search **decouple** → the team can improve semantic search independently (Rohan's stated goal). +- Postgres becomes disposable/rebuildable → simpler recovery, fewer consistency bugs. +- Unlocks future "user owns their KB as a real git repo" differentiator. + +### Negative / risks + +- **Concurrency:** git is single-writer per repo → requires the per-workspace lock (mitigated in v1). +- **Repo hygiene:** many small commits → periodic `git gc`/repack (operational, manageable). +- **Migration:** existing Postgres KBs must be exported into git repos once, preserving `unique_identifier_hash` mapping. +- **Real-time UI (Zero):** currently driven by Postgres logical replication; still needs a git → Postgres projection to keep the web client live. This is *new* code that partially offsets deletions. + +--- + +## Open questions (for team discussion) + +1. **Zero / real-time UI:** confirm the git → Postgres projection path and whether Zero stays as-is. +2. **Binaries:** keep blob store vs. adopt Git-LFS. +3. **Migration cutover:** big-bang vs. per-workspace feature flag (recommend feature-flag rollout). +4. **Merge UX later:** CRDT (Yjs) vs. review-gate (kherad) when multi-writer becomes a requirement. + +--- + +## Appendix: key file index (current implementation) + +| Topic | Path | +|---|---| +| Virtual path resolver | `surfsense_backend/app/agents/chat/runtime/path_resolver.py` | +| Virtual FS read backend | `.../filesystem/backends/kb_postgres.py` | +| Virtual FS write commit | `.../main_agent/middleware/kb_persistence/middleware.py` | +| Hybrid search | `.../shared/retrieval/hybrid_search.py` | +| Chunk reconciliation | `surfsense_backend/app/indexing_pipeline/chunk_reconciler.py` | +| User version history | `surfsense_backend/app/utils/document_versioning.py` | +| Agent revert | `surfsense_backend/app/services/revert_service.py` | +| ORM models | `surfsense_backend/app/db.py` |