perf(engine): scope CSR topology index to traversed edges, reuse it cross-branch (#312)

* perf(engine): scope the CSR topology index to traversed edges, reuse it cross-branch

The in-memory CSR graph index was built over every edge type in the catalog and
cache-keyed by the resolved snapshot id, so a single-edge join
(`$x identifiesPerson $p`) full-scanned every edge table in the graph (the
40-60s / 428s-first-traversal hang), and a lazy-fork branch cold-rebuilt main's
index. Two cuts close that:

- Scope (A2): build only the edge types the query traverses
  (`referenced_edge_types` over Expand/AntiJoin, exhaustive match), not the whole
  catalog. Threaded through GraphIndexHandle -> RuntimeCache; cache-keyed on the
  scoped set.
- Cross-branch reuse (A1): key RuntimeCache by each edge table's physical identity
  (table_key, version, table_branch, e_tag) instead of the snapshot id, so a
  lazy-fork branch whose edge tables physically are main's reuses main's built
  index. Local-FS (e_tag None) falls back to refresh-invalidation.

Adds graph_build_count/graph_edges_built probes for the cost tests.

* test(engine): cost tests for scoped + cross-branch-reused topology index

fresh_branch_traversal_reuses_main_graph_index (A1: a lazy-fork branch reuses
main's cached CSR index, 0 rebuilds) and single_edge_query_builds_only_referenced_edge
(A2: a one-edge query builds only that edge, not the whole catalog), via the
graph_build_count/graph_edges_built probes. Forced CSR mode, #[serial]. Updates the
recreated-branch incarnation test comment for the physical-identity key.

* docs(engine): topology-index scoping + physical-identity cache key

Document the scoped CSR build and the physical-identity (e_tag) graph-index cache
key with its local-FS refresh-invalidation fallback across invariants, testing,
execution, and architecture docs.

* fix(test): move CSR-forced topology cost tests to the all-serial binary

The two topology-build cost tests force OMNIGRAPH_TRAVERSAL_MODE via process-
global env mutation, which query.rs reads. In warm_read_cost.rs (a mixed
serial/non-serial binary) a concurrent non-serial traversal test could race the
env write (UB under Rust 2024's unsafe set_var contract) and be forced onto CSR.
Move them to traversal_indexed.rs — the dedicated all-serial binary with no
non-serial env reader (its documented-safe home) — and add a ModeGuard RAII
helper so a panic mid-test cannot leak the override. Addresses a PR review (P2).

* fix(engine): include edge endpoints in the graph-index cache key

The A1 physical-identity key omitted the edge's (from_type, to_type). GraphIndex
keys its TypeIndexes by those endpoint names and execute_expand_csr looks them up
by the current catalog's names, so a schema repoint of an edge type that leaves
the edge table's physical identity unchanged would reuse a stale index built with
the old endpoint namespace and fail with "no type index for <new type>". The old
snapshot_id (carrying the manifest version) masked this; dropping it exposed it.
Adding the endpoints to the key rebuilds on a repoint while preserving lazy-fork
cross-branch reuse (same endpoints -> same key). Addresses a PR review (P1).

* test(engine): scoped with_traversal_mode seam + e_tag graph-index coverage

Replace the process-global OMNIGRAPH_TRAVERSAL_MODE env-mutation test hack (which
forced #[serial] + dedicated all-serial binaries and was triplicated as ModeGuard
+ set_mode/clear_mode) with one general abstraction: a task-local
`with_traversal_mode` seam mirroring `with_query_io_probes`. It is scope-bound
(leak-free even on panic) and process-safe (never touches shared state), so a
forced-mode test cannot affect a concurrent test in the same binary.
`traversal_indexed_override` consults the seam first, then the env var (which
stays the documented ops escape hatch).

- Migrate traversal_indexed.rs, proptest_equivalence.rs, and the two topology cost
  tests (moved back to warm_read_cost.rs) to the seam; drop all ModeGuard /
  set_mode / clear_mode / #[serial] / per-file column0 helpers.
- Consolidate the duplicated first-column extractors into one shared
  `helpers::first_column_sorted`.
- Add `s3_storage.rs::s3_fresh_branch_traversal_reuses_main_graph_index_with_etags`:
  the CSR cache-key cross-branch reuse path on a REAL per-table e_tag (None on
  local FS, so local tests can't reach it). Confirmed empirically that RustFS — the
  CI S3 backend — surfaces ETags into version_metadata.e_tag(). CI path filter now
  triggers the rustfs job on runtime_cache/graph_index changes.
This commit is contained in:
Ragnor Comerford 2026-06-28 20:03:06 +02:00 committed by GitHub
parent 20e5fada8a
commit e7e057e26d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
17 changed files with 639 additions and 175 deletions

View file

@ -336,7 +336,29 @@ them explicit.
deferred — it needs the Q8 cleanup-resurrection watermark first). The commit
graph IS now reconcilable from the manifest (RFC-013 Phase 7 — it is a pure
projection of the `graph_commit`/`graph_head` rows); the traversal id-map is
still rebuilt.
still rebuilt. The CSR/CSC topology index is now **scoped and cross-branch
reused** (the two cuts that closed the cross-edge-join hang): the build covers
only the edge types a query traverses (`referenced_edge_types` over
`Expand`/`AntiJoin`, not every catalog edge — a single-edge join no longer
scans the whole graph's edge data), and the `RuntimeCache` cache key is each
edge table's physical identity `(table_key, version, table_branch, e_tag)`
plus the edge's `(from_type, to_type)` endpoint mapping — rather than the
resolved snapshot id — so a lazy-fork branch reuses main's built index instead
of cold-scanning it, while a schema repoint of an edge type (which changes the
built `TypeIndex` namespace) still rebuilds even if the edge table's physical
identity is unchanged. Residual: on stores without per-table e_tags (local FS)
a branch deleted and recreated at the same version with the same endpoints has
the same key, so the incarnation distinction falls back to the same-branch
manifest refresh clearing read caches (`invalidate_all`); production object
stores carry real e_tags, so the key alone distinguishes incarnations there
(the e_tag-present cross-branch-reuse path is exercised in CI by
`s3_storage.rs::s3_fresh_branch_traversal_reuses_main_graph_index_with_etags`
against RustFS, which surfaces real ETags — local-FS tests cannot reach it).
Known narrow gap (local FS only): a cold *cross-branch* resolve of a
recreated branch (a long-lived reader bound to another branch) does not trigger
that same-branch refresh, so an e_tag-less recreated branch can still reuse a
stale entry until a same-branch read refreshes — acceptable because local FS is
a dev/test substrate and production carries e_tags.
- **Commit-graph parent under concurrency — CLOSED (RFC-013 Phase 7):** the graph
commit is now recorded in the manifest publish CAS, and the publisher resolves
the new commit's parent INSIDE its retry loop, per attempt, from the just-loaded