apunkt/omnigraph
1
0
Fork 0
mirror of https://github.com/ModernRelay/omnigraph.git synced 2026-06-12 01:45:14 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 5

lance: confirm MemWAL is opt-in, intra-table, no overlap with MR-847

Browse source 
Fetched https://lance.org/format/table/mem_wal/ in full via npx mdrip.
The "Overview / Details / Implementation" sidebar items turned out to
be anchor sections on the same URL, not separate pages.

Key findings (relevant to MR-847's recovery reconciler design):

* MemWAL is opt-in. Requires (1) unenforced primary key in schema,
  (2) explicit shard config, (3) writers using the LSM-tree write
  path. omnigraph does NOT enable it; we use direct write_fragments +
  commit(Operation::Append).

* MemWAL is intra-table — addresses streaming-write throughput for
  one Lance base table via MemTables → flushed MemTables → async
  merge. It does not coordinate across multiple tables.

* MemWAL's recovery is intra-table: WAL replay reconstructs MemTable
  state for one table. It does NOT help with omnigraph's cross-table
  manifest-pinned-vs-Lance-HEAD drift class.

Conclusion: MR-847's recovery reconciler design is unaffected. The
two operate at different abstraction layers.

Borrowable: MemWAL's epoch-based fencing pattern is structurally
similar to a future multi-coordinator sidecar protocol; noted on
MR-847 for if MR-668 (multi-process) ever lands.
This commit is contained in:
Ragnor Comerford 2026-05-02 19:44:37 +02:00
parent 5afde54d69
commit c9a81266e4
No known key found for this signature in database
1 changed files with 1 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/lance.md
View file

@ -160,7 +160,7 @@ When Lance ships a major release that changes any of the above (file format bump
A full read-through of every index page above was performed in the MR-793 cycle. Findings (no code changes required for PR #70):
- The MemWAL system index has three deeper sub-pages that this index does not yet list — they're load-bearing for understanding crash-recovery semantics and are needed before MR-847 (recovery reconciler) implementation. Add when located: `MemWAL Index Overview`, `MemWAL Index Details`, `MemWAL Implementation` (linked from the parent MemWAL page but at sub-URLs not currently in `lance.md`).
- The MemWAL "three sub-pages" (Overview / Details / Implementation) turned out to be **anchor sections on the single existing page** at `https://lance.org/format/table/mem_wal/` — not separate URLs. Fetched in full via `npx mdrip`. Findings: MemWAL is opt-in (requires an unenforced primary key + explicit shard config; omnigraph doesn't use it), operates intra-table (LSM-tree for streaming writes into one Lance table), and does NOT overlap with MR-847's cross-table manifest-vs-Lance-HEAD recovery problem. MR-847's design is unaffected.
- The distributed-indexing guide names Python APIs (`commit_existing_index_segments`, `merge_existing_index_segments`); the Rust analogues exist via `CreateIndexBuilder::execute_uncommitted` for scalar indices but **`build_index_metadata_from_segments` is `pub(crate)`** and blocks vector-index two-phase commits from outside the lance crate. Filed [lance-format/lance#6666](https://github.com/lance-format/lance/issues/6666) as a companion to [#6658](https://github.com/lance-format/lance/issues/6658).
- "Stable Row ID for Index" is documented as **experimental** in lance-4.0.x. Our datasets enable stable row IDs at the dataset level (`WriteParams::enable_stable_row_ids = true`); confirming whether our created indices opt into stable-row-id mode is a follow-up worth doing before MR-848 (index reconciler) lands.
- Fragment Reuse Index (FRI) is documented as one of three compaction strategies. omnigraph currently uses option 2 (immediate index rewrite at compaction time, via `omnigraph optimize`'s post-compaction rebuild). Adopting FRI is the explicit option for compaction-friendly index updates; relevant to MR-848.