omnigraph/docs
Ragnor Comerford 9b0920b5da
address PR #70 bot review (Cubic + Cursor): 7 inline + failpoint test + invariants notes
Cubic findings:
* `tests/forbidden_apis.rs`: expand `FORBIDDEN_PATTERNS` with `Dataset::write`
  / `Dataset::append` / `Dataset::delete` / `Dataset::merge_insert` /
  `Dataset::add_columns` / `update_columns` / `drop_columns` /
  `truncate_table` / `restore` and the bare `.merge_insert(` /
  `.add_columns(` / `.update_columns(` / `.drop_columns(` /
  `.truncate_table(` method patterns. Deliberately avoid `.append(` /
  `.delete(` / `.write(` (over-match `Vec::append`, `.delete_branch(`,
  arrow-array `.append(`, etc.). Allow-list `commit_graph.rs` and
  `graph_coordinator.rs` — they're manifest-layer infra that legitimately
  uses `Dataset::write` for system tables.
* `schema_apply.rs:253`: pass `entry.table_branch.as_deref()` (not
  `None`) to `open_dataset_head_for_write` for consistency with the
  sibling `indexed_tables` block. Schema apply rejects non-main
  branches at the lock-acquire step today, so behavior is unchanged;
  this is a defensive consistency fix that survives a future relaxation
  of the lock check.
* `storage_layer.rs:131` doc: was `Vec<&StagedWrite>` with lifetime
  claim; actually returns `Vec<StagedWrite>` (cloned). Fixed.
* `AGENTS.md:201` capability matrix row + `storage_layer.rs:1` module
  doc: softened the "stage_* + commit_staged are the only paths" /
  "trait funnels every write" overclaim. Inline-commit residuals
  (`delete_where`, `create_vector_index`) remain on the trait pending
  upstream Lance work (#6658, #6666); legacy `append_batch` etc.
  remain pending Phase 1b / Phase 9. Module doc now describes the
  current transitional state honestly.

Cursor Bugbot findings:
* `storage_layer.rs:360`: trait `delete_where` consumed `SnapshotHandle`
  but returned only `DeleteState`, dropping the post-delete dataset.
  Future callers migrating from the inherent `&mut Dataset` API would
  lose the post-delete dataset state needed for indexing /
  `table_state` queries. Fixed: returns `(SnapshotHandle, DeleteState)`
  matching `append_batch` / `overwrite_batch` shape.
* `storage_layer.rs:824`: removed dead `_scanner_type_marker` fn and
  the unused `Scanner` import (the marker existed only to suppress an
  unused-import warning — fixing the import is the cleaner answer).

Engine-level Phase A failpoint test (closes the partial-criterion
flagged in Cubic's acceptance-criteria checklist):
* `db/omnigraph/table_ops.rs::stage_and_commit_btree`: instrumented
  with `crate::failpoints::maybe_fail("ensure_indices.post_stage_pre_commit_btree")`
  between `stage_create_btree_index` and `commit_staged`.
* `tests/failpoints.rs::ensure_indices_phase_a_btree_failure_leaves_existing_tables_writable`:
  triggers the failpoint via a schema-apply that adds a new node type;
  proves that existing tables are unaffected (Person mutation succeeds
  after the failed apply) — i.e. Phase A failure leaves no Lance-HEAD
  drift on tables outside the failed `added_tables` iteration.

`docs/invariants.md` transitional notes:
* §VI.23 (atomicity per query): annotated as upheld at the
  writer-trait surface for inserts / updates / scalar-index builds /
  merge_insert / overwrite after MR-793 PR #70. Per-table
  commit_staged → manifest publish window remains; closing requires
  MR-847's recovery-on-open reconciler. `delete_where` and
  `create_vector_index` remain inline pending lance#6658 / #6666.
* §VII.35 (reconciler pattern): annotated as partial — staged
  primitives are the building blocks; the reconciler task itself is
  MR-848.
* §VIII.45 (reference impl per trait): `TableStorage` has its primary
  impl on `TableStore` with opaque-handle signatures; no test impl
  yet.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 18:47:07 +02:00
..
releases MR-794 step 2: address PR #68 review — merge semantics, cardinality, residual 2026-05-01 13:47:55 +02:00
architecture.md MR-794 step 2: docs — runs/invariants/architecture/execution + cleanup 2026-05-01 10:43:19 +02:00
audit.md MR-794 step 2: docs — runs/invariants/architecture/execution + cleanup 2026-05-01 10:43:19 +02:00
branches-commits.md MR-794 step 2: docs — runs/invariants/architecture/execution + cleanup 2026-05-01 10:43:19 +02:00
changes.md Refactor AGENTS.md from encyclopedia to map; move spec into docs/ 2026-04-28 23:31:08 +02:00
ci.md Address reviewer feedback (Cursor + cubic) on PR #60 2026-04-29 00:09:06 +02:00
cli-reference.md Address reviewer feedback (Cursor + cubic) on PR #60 2026-04-29 00:09:06 +02:00
cli.md MR-794 step 2: docs — runs/invariants/architecture/execution + cleanup 2026-05-01 10:43:19 +02:00
constants.md MR-794 step 2: docs — runs/invariants/architecture/execution + cleanup 2026-05-01 10:43:19 +02:00
deployment.md Document AWS build variant and bearer-token sources 2026-04-18 04:04:45 +03:00
embeddings.md Refactor AGENTS.md from encyclopedia to map; move spec into docs/ 2026-04-28 23:31:08 +02:00
errors.md MR-794 step 2: docs — runs/invariants/architecture/execution + cleanup 2026-05-01 10:43:19 +02:00
execution.md MR-794 step 2: docs — runs/invariants/architecture/execution + cleanup 2026-05-01 10:43:19 +02:00
indexes.md Refactor AGENTS.md from encyclopedia to map; move spec into docs/ 2026-04-28 23:31:08 +02:00
install.md Remove stale Homebrew source-build note 2026-04-11 14:12:49 +03:00
invariants.md address PR #70 bot review (Cubic + Cursor): 7 inline + failpoint test + invariants notes 2026-05-02 18:47:07 +02:00
lance.md MR-793 follow-up: lance docs alignment audit + mandate full-page fetch via mdrip 2026-05-02 17:41:32 +02:00
maintenance.md Add internal-schema versioning + auto-migration for __manifest 2026-04-29 11:44:14 +00:00
merge.md Refactor AGENTS.md from encyclopedia to map; move spec into docs/ 2026-04-28 23:31:08 +02:00
policy.md Refactor AGENTS.md from encyclopedia to map; move spec into docs/ 2026-04-28 23:31:08 +02:00
query-language.md MR-794 step 2: docs — runs/invariants/architecture/execution + cleanup 2026-05-01 10:43:19 +02:00
runs.md MR-793 phases 1-6: TableStorage trait + staged-write surface for engine writers 2026-05-02 11:03:15 +02:00
schema-language.md Address reviewer feedback (Cursor + cubic) on PR #60 2026-04-29 00:09:06 +02:00
server.md MR-771: demote Run to direct-publish via expected_table_versions CAS 2026-04-30 08:52:50 +02:00
storage.md MR-794 step 2: docs — runs/invariants/architecture/execution + cleanup 2026-05-01 10:43:19 +02:00
testing.md MR-794 step 2: docs — runs/invariants/architecture/execution + cleanup 2026-05-01 10:43:19 +02:00