recovery: address PR #72 review findings

Bot reviewers (cubic, cursor, chatgpt-codex) caught 4 merge-blocking
bugs + 3 strongly-recommended fixes + 3 doc errors in the initial PR.
Each fix has a paired test demonstrating the bug before the fix.

Merge-blocking fixes:

- BranchMerge moved to loose-match classifier arm. publish_rewritten_
  merge_table runs multiple commit_staged calls per table (merge_insert
  + delete_where + index rebuilds). Strict classification rolled back
  valid completed Phase B work as UnexpectedMultistep. Three new unit
  tests pin the loose-match behavior for BranchMerge.

- branch_merge sidecar uses self.active_branch() (the resolved target
  branch) instead of inferring from the first sorted table key. The
  previous heuristic could record None (== main) when the merge target
  was a non-main branch, causing recovery to publish to the wrong
  manifest namespace.

- Best-effort sidecar delete in all 5 writer sites (mutation, loader,
  schema_apply, branch_merge, ensure_indices). Previously, a sidecar
  cleanup failure after a successful manifest publish would error out
  the user's call for a write that already landed. Now: log a warning
  and ignore — the next open's recovery sweep tidies the stale sidecar
  via NoMovement classification.

- ensure_indices sidecar scoped to tables that need work via new
  helpers needs_index_work_node / needs_index_work_edge. Previously
  the sidecar pinned every catalog table; if only one needed indexing,
  the others classified as NoMovement and the all-or-nothing decision
  rolled back legitimate index work.

Strongly-recommended fixes:

- recover_manifest_drift now takes &mut GraphCoordinator and refreshes
  between sidecars. Sidecar B's classification needs to see sidecar
  A's manifest changes, otherwise B can be classified against stale
  pins and incorrectly roll back work that just landed.

- list_sidecars sorts URIs before reading. Sidecar filenames are
  ULIDs (chronologically sortable), so this gives deterministic,
  time-ordered processing. Filesystem-order was nondeterministic.

- ReadOnly opens skip recover_schema_state_files too (was: only the
  MR-847 sweep was gated). Read-only consumers may run with read-only
  credentials; silent open-time mutations violate the contract.

Doc cleanups:

- Removed stale "Phase 4 placeholder" comment from
  recover_manifest_drift.
- docs/runs.md decision-tree wording now correctly surfaces the
  InvariantViolation abort path.
- docs/branches-commits.md clarifies actor_id is in
  _graph_commit_actors.lance (joined by graph_commit_id), not on
  _graph_commits.lance itself.

Test surface (post-fixes):
- 25 unit tests in db::manifest::recovery (+4 from this commit).
- 10 integration tests in tests/recovery.rs (+3 from this commit).
- ~672 tests across ~25 binaries pass with --features failpoints.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ragnor Comerford 2026-05-03 12:21:40 +02:00
parent 932334ba01
commit 164bafbbe7
No known key found for this signature in database
10 changed files with 477 additions and 69 deletions

View file

@ -543,9 +543,18 @@ async fn load_jsonl_reader<R: BufRead>(
db.commit_updates_on_branch_with_expected(branch, &updates, &expected_versions)
.await?;
// MR-847: sidecar protects the per-table commit_staged →
// manifest publish window. Phase C succeeded — clean up.
// manifest publish window. Phase C succeeded — clean up
// best-effort (PR #72 review).
if let Some(handle) = sidecar_handle {
crate::db::manifest::delete_sidecar(&handle, db.storage_adapter()).await?;
if let Err(err) =
crate::db::manifest::delete_sidecar(&handle, db.storage_adapter()).await
{
tracing::warn!(
error = %err,
operation_id = handle.operation_id.as_str(),
"MR-847 sidecar cleanup failed; the next open's recovery sweep will resolve it"
);
}
}
} else {
// LoadMode::Overwrite keeps the legacy inline-commit path —