diff --git a/AGENTS.md b/AGENTS.md index 73f177b3..3796c3a1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -32,7 +32,7 @@ OmniGraph is a typed property-graph engine built as a coordination layer over ma - **Languages**: a `.pg` schema language and a `.gq` query language, both Pest-based, with a typed IR. - **Multi-modal querying**: vector ANN (`nearest`), full-text (`search`/`fuzzy`/`match_text`/`bm25`), Reciprocal Rank Fusion (`rrf`), and graph traversal (`Expand`, anti-join `not { … }`) in one runtime. - **Branches and commits across the whole graph**: Git-style — every successful publish appends to a commit DAG; merges are three-way at the row level. -- **Atomic per-query writes**: `mutate_as` and `load` accumulate insert/update batches into an in-memory `MutationStaging.pending` per touched table; one `stage_*` + `commit_staged` per table runs at end-of-query, then `ManifestBatchPublisher::publish` commits the manifest atomically with per-table `expected_table_versions` CAS. A mid-query failure leaves Lance HEAD untouched on staged tables — no drift, no run state machine, no staging branches. Deletes stage through the same path (MR-A: `stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`), so they no longer advance Lance HEAD inline. D₂ at parse time is a deliberate boundary — one mutation query is constructive (insert/update) XOR destructive (delete) — so read-your-writes within a query stays unambiguous and each table commits at most one version; compose mixed operations via separate mutations, or a branch for single-commit atomicity. +- **Atomic per-query writes**: `mutate_as` and `load` accumulate insert/update batches into an in-memory `MutationStaging.pending` per touched table. Their RFC-022 adapter resolves or rejects relevant recovery intents before base capture, captures `(native branch id, exact graph_head, schema identity)`, then rechecks recovery and authority under schema → branch → table gates. It arms a schema-v3 sidecar containing exact Lance transaction identities + pre-minted lineage, commits each table with zero transparent conflict retries, confirms the achieved effects, then publishes under the same token. Retryable pre-effect authority conflicts fully reprepare; strict conflicts return `ReadSetChanged`; an unresolved intent or any post-effect publish error returns `RecoveryRequired`. Deletes stage through the same path. D₂ at parse time remains the constructive (insert/update) XOR destructive (delete) boundary. - **HTTP server**: Axum + utoipa OpenAPI, bearer auth (SHA-256 hashed, optional AWS Secrets Manager). Cedar policy enforcement is engine-wide — every `_as` writer calls `Omnigraph::enforce(action, scope, actor)`, so HTTP, CLI, and embedded SDK consumers all hit the same gate. **Cluster-only boot** (RFC-011): the server always boots from a cluster directory (`--cluster `, RFC-005) and serves N graphs (N ≥ 1) under multi-graph routes (`/graphs/{graph_id}/...` + read-only `GET /graphs` enumeration); there are no single-graph flat routes and no positional-URI boot. Per-graph + server-level Cedar policies. Runtime add/remove (`POST /graphs`, `DELETE /graphs/{id}`) is not exposed — operators run `cluster apply` and restart. - **CLI** with two-surface config (RFC-007/008): the team-owned cluster directory (`cluster.yaml`) plus the per-operator `~/.omnigraph/config.yaml` (servers, clusters, credentials, actor, profiles, aliases, defaults). Graphs are addressed via `--store`/`--server`/`--cluster`/`--profile`/operator defaults (RFC-011). Multi-format output (json/jsonl/csv/kv/table). @@ -53,7 +53,7 @@ CLI (omnigraph) HTTP Server (omnigraph-server, Axum) omnigraph (engine) ── ManifestCoordinator, CommitGraph, RunRegistry, GraphIndex (CSR/CSC), exec │ ▼ - Lance 7.x ── columnar Arrow, fragments, per-dataset versions/branches, indexes + Lance 9.x ── columnar Arrow, fragments, per-dataset versions/branches, indexes │ ▼ Object store (file / s3 / RustFS / MinIO / S3-compat) @@ -253,11 +253,11 @@ omnigraph policy explain --cluster ./company-brain --graph knowledge --actor act | Columnar storage on object store | ✅ Arrow/Lance | URI normalization, S3 env-var plumbing | | Per-dataset versioning + time travel | ✅ | `snapshot_at_version`, `entity_at`, snapshot-pinned reads across many tables | | Per-dataset branches | ✅ | **Graph-level** branches (atomic across all sub-tables), lazy fork, system branch filtering | -| Atomic single-dataset commits | ✅ | **Multi-table publish via three layers**, NOT a single Lance primitive: (1) per-table Lance `commit_staged` for the data write, (2) `__manifest` row-level CAS via `ManifestBatchPublisher` for cross-table ordering, (3) the open-time recovery sweep for the residual gap between (1) and (2). All three layers ship; the five migrated writers (`MutationStaging::finalize`, `schema_apply`, `branch_merge`, `ensure_indices`, `optimize_all_tables`) write a `__recovery/{ulid}.json` sidecar before Phase B and delete it after Phase C. The next `Omnigraph::open` (gated on `OpenMode::ReadWrite`) runs the sweep in `db/manifest/recovery.rs`: classify, decide all-or-nothing per sidecar, roll forward via single `ManifestBatchPublisher::publish` or roll back via `Dataset::restore` followed by a manifest publish of the restored version (so both directions converge to `manifest == HEAD` — no residual drift), and record an audit row in `_graph_commit_recoveries.lance` (queryable via `omnigraph commit list --filter actor=omnigraph:recovery`). The write entry points (`load_as`, `mutate_as`, `apply_schema_as`, `branch_merge_as`) and `refresh` additionally run an in-process roll-forward-only heal (serialized against live writers via the per-table write queues), so a long-lived server converges on its next write without restart; only rollback-eligible sidecars still defer to the next read-write open (a future background reconciler's goal). Engine writes route through a sealed `TableStorage` trait (`db.storage()`) exposing only `stage_*` + `commit_staged` + reads; the sole inline-commit residual (`create_vector_index`) is split onto a separate sealed `InlineCommitResidual` trait reached via `db.storage_inline_residual()` (MR-854), so the default surface cannot couple a write with a HEAD advance — §1 holds by construction. `delete` migrated to the staged path in MR-A (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, [#6658](https://github.com/lance-format/lance/issues/6658)); `create_vector_index` stays inline until upstream Lance ships a public two-phase API ([#6666](https://github.com/lance-format/lance/issues/6666)); `LoadMode::Overwrite` uses Lance `Overwrite` staged transactions. | +| Atomic single-dataset commits | ✅ | **Multi-table publish via three layers**, NOT a single Lance primitive: (1) per-table Lance `commit_staged` for the data write, (2) `__manifest` row-level CAS via `ManifestBatchPublisher` for cross-table ordering, (3) the open-time recovery sweep for the residual gap between (1) and (2). All three layers ship; the five migrated writers (`MutationStaging::commit_all`, `schema_apply`, `branch_merge`, `ensure_indices`, `optimize_all_tables`) write a `__recovery/{ulid}.json` sidecar before Phase B and delete it after Phase C. The next `Omnigraph::open` (gated on `OpenMode::ReadWrite`) runs the sweep in `db/manifest/recovery.rs`: classify, decide all-or-nothing per sidecar, roll forward via single `ManifestBatchPublisher::publish` or roll back via `Dataset::restore` followed by a manifest publish of the restored version (so both directions converge to `manifest == HEAD` — no residual drift), and record an internal audit row in `_graph_commit_recoveries.lance`. A v3 roll-forward preserves the interrupted writer's fixed commit lineage and actor; rollback and legacy recovery commits use `omnigraph:recovery`. There is currently no public CLI query for the recovery-audit table, and ordinary commit history is not a complete recovery enumeration. The write entry points (`load_as`, `mutate_as`, `apply_schema_as`, `branch_merge_as`) and `refresh` additionally run an in-process roll-forward-only heal, serialized against same-process live writers through the shared root-scoped gate manager, so a long-lived server converges on its next write without restart; only rollback-eligible sidecars still defer to the next read-write open (a future background reconciler's goal). Engine writes route through a sealed `TableStorage` trait (`db.storage()`) exposing only `stage_*` + `commit_staged` + reads; the sole inline-commit residual (`create_vector_index`) is split onto a separate sealed `InlineCommitResidual` trait reached via `db.storage_inline_residual()` (MR-854), so the default surface cannot couple a write with a HEAD advance — §1 holds by construction. `delete` migrated to the staged path in MR-A (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, [#6658](https://github.com/lance-format/lance/issues/6658)); `create_vector_index` stays inline until upstream Lance ships a public two-phase API ([#6666](https://github.com/lance-format/lance/issues/6666)); `LoadMode::Overwrite` uses Lance `Overwrite` staged transactions. | | Compaction (`compact_files`) + reindex (`optimize_indices`) | ✅ | `omnigraph optimize` orchestrates over all node/edge tables, bounded concurrency; per table runs `compact_files` **then Lance `optimize_indices`** (folds appended/rewritten fragments back into existing indexes — incremental merge, not retrain) and **publishes the resulting version to `__manifest`** (so the manifest tracks the Lance HEAD — required for reads to observe the work and for schema apply / strict writes to pass their HEAD-vs-manifest precondition), under the per-`(table, main)` write queue with `SidecarKind::Optimize` recovery coverage spanning both ops; **commits even with no compaction work if index coverage is stale**; **refuses on an unrecovered graph**; **skips uncovered HEAD > manifest drift** with `DriftNeedsRepair`; **compacts blob-bearing tables** (the pre-9 `LANCE_SUPPORTS_BLOB_COMPACTION` skip was removed once Lance 8.0.0+ shipped blob-v2 compaction — see [docs/dev/invariants.md](docs/dev/invariants.md) Known Gaps) | | Repair uncovered drift | — | `omnigraph repair` explicitly classifies uncovered table `HEAD > manifest` drift: verified maintenance drift (`ReserveFragments`/`Rewrite`) can be published with `--confirm`; suspicious or unverifiable drift requires `--force --confirm`. Sidecar-covered crash residuals still recover automatically on open. | | Cleanup (`cleanup_old_versions`) | ✅ | `omnigraph cleanup` with `--keep` / `--older-than` policy | -| BTREE / inverted (FTS) / vector indexes | ✅ | `@index`/`@key` declares intent; the physical index is derived state that never fails a logical op. Built per column through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector); idempotent; lazy across branches. **Schema apply builds nothing** (records intent only); `load`/`mutate` build inline but **defer an untrainable Vector column** (no trainable vectors yet) as *pending* rather than aborting. `ensure_indices`/`optimize` is the reconciler that materializes declared-but-missing indexes and restores coverage of appended/rewritten fragments (`optimize_indices`), reporting still-pending columns (see Compaction row). | +| BTREE / inverted (FTS) / vector indexes | ✅ | `@index`/`@key` declares intent; the physical index is derived state that never fails a logical op. Built per column through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector); idempotent; lazy across branches. **Schema apply and mutation/load build no indexes inline**: the latter publish only their exact data effects, leaving physical intent pending. `ensure_indices`/`optimize` materializes declared-but-missing indexes, restores fragment coverage, and continues to report untrainable Vector columns as pending. | | `merge_insert` upsert | ✅ | `LoadMode::Merge`, mutation `update`/`insert`/`delete` lowering | | Vector search | ✅ | `nearest()` query op; embedding pipeline (Gemini / OpenAI clients); `@embed` in schema | | Full-text search | ✅ | `search/fuzzy/match_text/bm25` query ops | @@ -267,7 +267,7 @@ omnigraph policy explain --cluster ./company-brain --graph knowledge --actor act | Query language | — | `.gq` + Pest grammar + IR + lowering + linter | | Schema migration planning | — | `plan_schema_migration` + `apply_schema` step types + `__schema_apply_lock__` | | Commit graph (DAG) across whole graph | — | Lineage (linear + merge parents, ULID ids, actor) stored as `graph_commit`/`graph_head` rows in `__manifest`, written in the same publish CAS as the table-version rows (RFC-013 Phase 7 — atomic with the graph commit). The in-memory commit graph is a pure projection of those rows; the legacy `_graph_commits.lance` / `_graph_commit_actors.lance` tables are **retired** (a fresh graph creates neither) | -| Per-query atomic writes | — | In-memory `MutationStaging.pending` accumulator + `stage_*` / `commit_staged` per touched table at end-of-query + publisher CAS via `commit_with_expected` (single manifest commit per `mutate_as` / `load`); D₂ parse-time rule keeps inserts/updates and deletes from mixing | +| Per-query atomic writes | — | In-memory `MutationStaging.pending` accumulator + one exact staged Lance transaction per touched table + schema-v3 recovery confirmation + exact branch-head publisher CAS (single manifest commit per `mutate_as` / `load`); D₂ parse-time rule keeps inserts/updates and deletes from mixing | | Three-way row-level merge | — | `OrderedTableCursor` + `StagedTableWriter`, structured `MergeConflictKind` | | Change feeds | — | `diff_between` / `diff_commits` with manifest fast path + ID streaming | | Cedar policy | — | Per-graph actions plus server-scoped actions (see [docs/user/operations/policy.md](docs/user/operations/policy.md) for the current list), branch / target_branch / protected scopes, validate/test/explain CLI. **Engine-wide enforcement** (MR-722): every `_as` writer (`apply_schema_as`, `mutate_as`, `load_as` — the deprecated `ingest_as` shims route through it — `branch_create_as` / `branch_create_from_as`, `branch_delete_as`, `branch_merge_as`) calls `Omnigraph::enforce(action, scope, actor)` — HTTP, CLI, embedded SDK all hit the same gate. | diff --git a/crates/omnigraph-api-types/src/lib.rs b/crates/omnigraph-api-types/src/lib.rs index cff10f2a..feb4a777 100644 --- a/crates/omnigraph-api-types/src/lib.rs +++ b/crates/omnigraph-api-types/src/lib.rs @@ -577,6 +577,21 @@ pub struct ManifestConflictOutput { pub actual: u64, } +/// Structured authority mismatch for an RFC-022 prepared write. Values are +/// strings because members include optional graph commit ids and future +/// authority tokens, not only numeric table versions. +#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)] +pub struct ReadSetConflictOutput { + pub member: String, + pub expected: Option, + pub actual: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)] +pub struct RecoveryRequiredOutput { + pub operation_id: String, +} + #[derive(Debug, Clone, Serialize, Deserialize, ToSchema)] pub struct ErrorOutput { pub error: String, @@ -590,6 +605,13 @@ pub struct ErrorOutput { /// manifest is now at `actual`. Refresh and retry. #[serde(skip_serializing_if = "Option::is_none")] pub manifest_conflict: Option, + /// Set when a prepared write's logical authority changed before effects. + #[serde(skip_serializing_if = "Option::is_none")] + pub read_set_conflict: Option, + /// Set when an overlapping durable recovery intent must be resolved before + /// retry. Its table effects may or may not have started. + #[serde(skip_serializing_if = "Option::is_none")] + pub recovery_required: Option, } pub fn snapshot_payload( diff --git a/crates/omnigraph-server/src/handlers.rs b/crates/omnigraph-server/src/handlers.rs index 1d0a5360..c226a029 100644 --- a/crates/omnigraph-server/src/handlers.rs +++ b/crates/omnigraph-server/src/handlers.rs @@ -788,8 +788,9 @@ pub(crate) async fn run_query( (status = 400, description = "Bad request", body = ErrorOutput), (status = 401, description = "Unauthorized", body = ErrorOutput), (status = 403, description = "Forbidden", body = ErrorOutput), - (status = 409, description = "Merge conflict", body = ErrorOutput), + (status = 409, description = "Write-authority conflict", body = ErrorOutput), (status = 429, description = "Per-actor admission cap exceeded; honor `Retry-After` header", body = ErrorOutput), + (status = 503, description = "An overlapping durable recovery intent must be resolved before retry", body = ErrorOutput), ), security(("bearer_token" = [])), )] @@ -837,8 +838,9 @@ pub(crate) async fn server_change( (status = 400, description = "Bad request", body = ErrorOutput), (status = 401, description = "Unauthorized", body = ErrorOutput), (status = 403, description = "Forbidden", body = ErrorOutput), - (status = 409, description = "Merge conflict", body = ErrorOutput), + (status = 409, description = "Write-authority conflict", body = ErrorOutput), (status = 429, description = "Per-actor admission cap exceeded; honor `Retry-After` header", body = ErrorOutput), + (status = 503, description = "An overlapping durable recovery intent must be resolved before retry", body = ErrorOutput), ), security(("bearer_token" = [])), )] @@ -847,7 +849,8 @@ pub(crate) async fn server_change( /// Writes to the named `branch` (defaults to `main`). Mutations are atomic /// per call and produce a new commit. Returns counts of nodes and edges /// affected. **Destructive**: on success the branch is updated; rejected -/// mutations may still acquire locks briefly. Returns 409 on merge conflict. +/// mutations may still acquire locks briefly. Returns 409 when the prepared +/// write authority changes before effects. /// /// Pairs with `POST /query` (read-only). The legacy `POST /change` route /// has identical semantics and is kept as a deprecated alias. @@ -904,9 +907,10 @@ pub(crate) fn parse_optional_invoke_body( (status = 401, description = "Unauthorized", body = ErrorOutput), (status = 403, description = "Forbidden (the inner `change` gate for a stored mutation)", body = ErrorOutput), (status = 404, description = "Unknown stored query, or `invoke_query` denied — indistinguishable to a caller without the grant", body = ErrorOutput), - (status = 409, description = "Merge conflict", body = ErrorOutput), + (status = 409, description = "Stored mutation write-authority conflict", body = ErrorOutput), (status = 429, description = "Per-actor admission cap exceeded; honor `Retry-After` header", body = ErrorOutput), (status = 500, description = "Policy evaluation error (a denial is reported as 404, not 500)", body = ErrorOutput), + (status = 503, description = "A stored mutation is blocked by a durable recovery intent", body = ErrorOutput), ), security(("bearer_token" = [])), )] @@ -1203,25 +1207,11 @@ pub(crate) async fn server_schema_apply( .await .map_err(ApiError::from_omni)? }; - // Prompt index convergence (iss-848): schema apply records `@index` intent - // but defers the physical build. On a long-lived server, materialize it - // promptly rather than waiting for the next `optimize` cron — spawned - // detached so it never blocks or fails the apply response. Best-effort: a - // failure is logged and the index still converges on the next optimize. - // The CLI is one-shot, so it has no equivalent; its convergence path is the - // operator's optimize cadence. - if result.applied { - let engine = Arc::clone(&handle.engine); - tokio::spawn(async move { - if let Err(err) = engine.ensure_indices().await { - tracing::warn!( - target: "omnigraph::server", - error = %err, - "post-apply ensure_indices failed; indexes will converge on the next optimize", - ); - } - }); - } + // Physical indexes are derived state. Schema apply records intent only; + // explicit `ensure_indices` / `optimize` maintenance owns convergence on + // every surface, including a long-lived server. Keeping the handler free + // of detached physical writes also makes a successful response describe + // the complete effect envelope of this request. Ok(Json(schema_apply_output(handle.uri.as_str(), result))) } @@ -1313,7 +1303,9 @@ async fn run_ingest( (status = 400, description = "Bad request", body = ErrorOutput), (status = 401, description = "Unauthorized", body = ErrorOutput), (status = 403, description = "Forbidden", body = ErrorOutput), + (status = 409, description = "Prepared load authority changed before effects", body = ErrorOutput), (status = 429, description = "Per-actor admission cap exceeded; honor `Retry-After` header", body = ErrorOutput), + (status = 503, description = "An overlapping durable recovery intent must be resolved before retry", body = ErrorOutput), ), security(("bearer_token" = [])), )] @@ -1357,7 +1349,9 @@ pub(crate) async fn server_load( (status = 400, description = "Bad request", body = ErrorOutput), (status = 401, description = "Unauthorized", body = ErrorOutput), (status = 403, description = "Forbidden", body = ErrorOutput), + (status = 409, description = "Prepared load authority changed before effects", body = ErrorOutput), (status = 429, description = "Per-actor admission cap exceeded; honor `Retry-After` header", body = ErrorOutput), + (status = 503, description = "An overlapping durable recovery intent must be resolved before retry", body = ErrorOutput), ), security(("bearer_token" = [])), )] @@ -1437,6 +1431,7 @@ pub(crate) async fn server_branch_list( (status = 403, description = "Forbidden", body = ErrorOutput), (status = 409, description = "Branch already exists", body = ErrorOutput), (status = 429, description = "Per-actor admission cap exceeded; honor `Retry-After` header", body = ErrorOutput), + (status = 503, description = "An overlapping durable recovery intent must be resolved before retry", body = ErrorOutput), ), security(("bearer_token" = [])), )] @@ -1518,6 +1513,7 @@ pub(crate) struct BranchPath { (status = 403, description = "Forbidden", body = ErrorOutput), (status = 404, description = "Branch not found", body = ErrorOutput), (status = 429, description = "Per-actor admission cap exceeded; honor `Retry-After` header", body = ErrorOutput), + (status = 503, description = "An overlapping durable recovery intent must be resolved before retry", body = ErrorOutput), ), security(("bearer_token" = [])), )] @@ -1579,6 +1575,7 @@ pub(crate) async fn server_branch_delete( (status = 403, description = "Forbidden", body = ErrorOutput), (status = 409, description = "Merge conflict", body = ErrorOutput), (status = 429, description = "Per-actor admission cap exceeded; honor `Retry-After` header", body = ErrorOutput), + (status = 503, description = "An overlapping durable recovery intent must be resolved before retry", body = ErrorOutput), ), security(("bearer_token" = [])), )] diff --git a/crates/omnigraph-server/src/lib.rs b/crates/omnigraph-server/src/lib.rs index d60a0dc4..7a91abcd 100644 --- a/crates/omnigraph-server/src/lib.rs +++ b/crates/omnigraph-server/src/lib.rs @@ -286,10 +286,12 @@ impl Write for ExportStreamWriter { #[derive(Debug)] pub struct ApiError { status: StatusCode, - code: ErrorCode, + code: Option, message: String, merge_conflicts: Vec, manifest_conflict: Option, + read_set_conflict: Option, + recovery_required: Option, } impl AppState { @@ -616,40 +618,48 @@ impl ApiError { pub fn unauthorized(message: impl Into) -> Self { Self { status: StatusCode::UNAUTHORIZED, - code: ErrorCode::Unauthorized, + code: Some(ErrorCode::Unauthorized), message: message.into(), merge_conflicts: Vec::new(), manifest_conflict: None, + read_set_conflict: None, + recovery_required: None, } } pub fn forbidden(message: impl Into) -> Self { Self { status: StatusCode::FORBIDDEN, - code: ErrorCode::Forbidden, + code: Some(ErrorCode::Forbidden), message: message.into(), merge_conflicts: Vec::new(), manifest_conflict: None, + read_set_conflict: None, + recovery_required: None, } } pub fn bad_request(message: impl Into) -> Self { Self { status: StatusCode::BAD_REQUEST, - code: ErrorCode::BadRequest, + code: Some(ErrorCode::BadRequest), message: message.into(), merge_conflicts: Vec::new(), manifest_conflict: None, + read_set_conflict: None, + recovery_required: None, } } pub fn not_found(message: impl Into) -> Self { Self { status: StatusCode::NOT_FOUND, - code: ErrorCode::NotFound, + code: Some(ErrorCode::NotFound), message: message.into(), merge_conflicts: Vec::new(), manifest_conflict: None, + read_set_conflict: None, + recovery_required: None, } } @@ -660,30 +670,36 @@ impl ApiError { pub fn method_not_allowed(message: impl Into) -> Self { Self { status: StatusCode::METHOD_NOT_ALLOWED, - code: ErrorCode::MethodNotAllowed, + code: Some(ErrorCode::MethodNotAllowed), message: message.into(), merge_conflicts: Vec::new(), manifest_conflict: None, + read_set_conflict: None, + recovery_required: None, } } pub fn conflict(message: impl Into) -> Self { Self { status: StatusCode::CONFLICT, - code: ErrorCode::Conflict, + code: Some(ErrorCode::Conflict), message: message.into(), merge_conflicts: Vec::new(), manifest_conflict: None, + read_set_conflict: None, + recovery_required: None, } } pub fn internal(message: impl Into) -> Self { Self { status: StatusCode::INTERNAL_SERVER_ERROR, - code: ErrorCode::Internal, + code: Some(ErrorCode::Internal), message: message.into(), merge_conflicts: Vec::new(), manifest_conflict: None, + read_set_conflict: None, + recovery_required: None, } } @@ -694,10 +710,12 @@ impl ApiError { pub fn too_many_requests(message: impl Into) -> Self { Self { status: StatusCode::TOO_MANY_REQUESTS, - code: ErrorCode::TooManyRequests, + code: Some(ErrorCode::TooManyRequests), message: message.into(), merge_conflicts: Vec::new(), manifest_conflict: None, + read_set_conflict: None, + recovery_required: None, } } @@ -715,20 +733,51 @@ impl ApiError { fn merge_conflict(conflicts: Vec) -> Self { Self { status: StatusCode::CONFLICT, - code: ErrorCode::Conflict, + code: Some(ErrorCode::Conflict), message: summarize_merge_conflicts(&conflicts), merge_conflicts: conflicts, manifest_conflict: None, + read_set_conflict: None, + recovery_required: None, } } fn manifest_version_conflict(message: String, details: api::ManifestConflictOutput) -> Self { Self { status: StatusCode::CONFLICT, - code: ErrorCode::Conflict, + code: Some(ErrorCode::Conflict), message, merge_conflicts: Vec::new(), manifest_conflict: Some(details), + read_set_conflict: None, + recovery_required: None, + } + } + + fn read_set_conflict(message: String, details: api::ReadSetConflictOutput) -> Self { + Self { + status: StatusCode::CONFLICT, + code: Some(ErrorCode::Conflict), + message, + merge_conflicts: Vec::new(), + manifest_conflict: None, + read_set_conflict: Some(details), + recovery_required: None, + } + } + + fn recovery_required(message: String, operation_id: String) -> Self { + Self { + status: StatusCode::SERVICE_UNAVAILABLE, + // `ErrorCode` is a closed rolling wire contract. The additive + // `recovery_required` field carries the new meaning while older + // clients continue to deserialize the otherwise familiar body. + code: None, + message, + merge_conflicts: Vec::new(), + manifest_conflict: None, + read_set_conflict: None, + recovery_required: Some(api::RecoveryRequiredOutput { operation_id }), } } @@ -752,6 +801,18 @@ impl ApiError { actual, }, ), + Some(ManifestConflictDetails::ReadSetChanged { + member, + expected, + actual, + }) => Self::read_set_conflict( + err.message, + api::ReadSetConflictOutput { + member, + expected, + actual, + }, + ), _ => Self::conflict(err.message), }, ManifestErrorKind::Internal => Self::internal(err.message), @@ -762,6 +823,13 @@ impl ApiError { .map(api::MergeConflictOutput::from) .collect(), ), + OmniError::RecoveryRequired { + operation_id, + reason, + } => Self::recovery_required( + format!("recovery required for operation {operation_id}: {reason}"), + operation_id, + ), OmniError::Lance(message) => Self::internal(format!("storage: {message}")), OmniError::Io(err) => Self::internal(format!("io: {err}")), // Engine-layer policy enforcement (MR-722). All denials and @@ -815,7 +883,7 @@ const RETRY_AFTER_SECONDS: &str = "60"; impl IntoResponse for ApiError { fn into_response(self) -> Response { let mut headers = axum::http::HeaderMap::new(); - if matches!(self.code, ErrorCode::TooManyRequests) { + if matches!(self.code, Some(ErrorCode::TooManyRequests)) { headers.insert( axum::http::header::RETRY_AFTER, axum::http::HeaderValue::from_static(RETRY_AFTER_SECONDS), @@ -826,15 +894,42 @@ impl IntoResponse for ApiError { headers, Json(ErrorOutput { error: self.message, - code: Some(self.code), + code: self.code, merge_conflicts: self.merge_conflicts, manifest_conflict: self.manifest_conflict, + read_set_conflict: self.read_set_conflict, + recovery_required: self.recovery_required, }), ) .into_response() } } +#[cfg(test)] +mod api_error_tests { + use super::*; + + #[tokio::test] + async fn recovery_required_503_omits_closed_error_code() { + let response = ApiError::from_omni(OmniError::RecoveryRequired { + operation_id: "01JTESTRECOVERY".to_string(), + reason: "pending recovery intent".to_string(), + }) + .into_response(); + + assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE); + let body = axum::body::to_bytes(response.into_body(), usize::MAX) + .await + .unwrap(); + let error: ErrorOutput = serde_json::from_slice(&body).unwrap(); + assert_eq!(error.code, None); + assert_eq!( + error.recovery_required.unwrap().operation_id, + "01JTESTRECOVERY" + ); + } +} + pub fn init_tracing() { let filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")); let _ = tracing_subscriber::fmt().with_env_filter(filter).try_init(); diff --git a/crates/omnigraph-server/tests/data_routes.rs b/crates/omnigraph-server/tests/data_routes.rs index 3e533288..3db0ed78 100644 --- a/crates/omnigraph-server/tests/data_routes.rs +++ b/crates/omnigraph-server/tests/data_routes.rs @@ -1099,18 +1099,15 @@ query vector_search_string($q: String) { } #[tokio::test(flavor = "multi_thread")] -async fn change_conflict_returns_manifest_conflict_409() { - // A write that races with another writer surfaces as HTTP 409 with - // a structured `manifest_conflict` body — `table_key`, `expected`, - // and `actual` — so clients can detect-and-retry without parsing - // the message. +async fn change_long_lived_handle_refreshes_before_preparing_write() { + // A handle that merely predates another committed write is not stale + // authority: open_write_txn probes the manifest incarnation and prepares + // from the fresh head. ReadSetChanged is reserved for movement *during* an + // already-prepared attempt (covered by the concurrent test below). let temp = init_loaded_graph().await; let graph = graph_path(temp.path()); - // Build the server first so its handle pins the pre-mutation manifest - // version. Then advance the manifest from outside the server. The - // server's next /change call will capture stale `expected_versions` - // (from its still-pinned snapshot) and the publisher's CAS rejects. + // Build the server first, then advance the graph through another handle. let state = AppState::open(graph.to_string_lossy().to_string()) .await .unwrap(); @@ -1154,34 +1151,17 @@ async fn change_conflict_returns_manifest_conflict_409() { ) .await; - assert_eq!(status, StatusCode::CONFLICT); - let error: ErrorOutput = serde_json::from_value(body).unwrap(); - assert_eq!(error.code, Some(omnigraph_server::api::ErrorCode::Conflict)); - let conflict = error - .manifest_conflict - .expect("publisher CAS rejection must populate manifest_conflict body"); - assert_eq!(conflict.table_key, "node:Person"); - assert!( - conflict.actual > conflict.expected, - "actual ({}) should be ahead of expected ({})", - conflict.actual, - conflict.expected, - ); + assert_eq!(status, StatusCode::OK); + assert_eq!(body["affected_nodes"], 1); } #[tokio::test(flavor = "multi_thread", worker_threads = 4)] async fn change_concurrent_inserts_same_key_serialize_without_409() { - // PR 2 Phase 2 (MR-686): pin the design fix for the same-key - // concurrency hazard. Pre-fix, in-process concurrent inserts on - // the same `(table, branch)` rejected with 409 manifest_conflict - // because `ensure_expected_version` fired before the per-table - // queue was acquired and saw Lance HEAD already advanced by a - // peer writer. Post-fix, Insert/Merge skip the strict pre-stage - // check (see `MutationOpKind::strict_pre_stage_version_check`); - // the queue serializes commit_staged; Lance's natural rebase - // handles the in-flight stage; the publisher's CAS on a fresh - // per-branch snapshot under the queue catches genuine cross- - // process drift. + // RFC-022 preservation guard: concurrent retryable inserts still all + // succeed, but not by rebasing an already-validated Lance transaction. + // The coarse branch gate serializes effects; a waiter whose authority + // token changed discards its complete attempt and reprepares from the + // winner's committed branch state. // // This test spawns N concurrent /change inserts on a single // node type and asserts: every request returns 200 (no 409), @@ -1270,36 +1250,10 @@ async fn change_concurrent_inserts_same_key_serialize_without_409() { } #[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn change_concurrent_updates_same_key_serialize_via_publisher_cas() { - // Pin Update RYW semantics under in-process concurrency on the same - // `(table, branch)`. With per-table queue serialization and op-kind-aware - // drift detection at commit time, exactly one of N concurrent UPDATEs - // on the same row commits; the rest are rejected as 409 manifest_conflict. - // - // Pre-fix bug class: in `MutationStaging::commit_all`, after queue - // acquisition, the staged Lance transaction is handed straight to - // `commit_staged`. For a writer whose staged dataset is at V0 but - // Lance HEAD has advanced to V1 (because the queue's prior winner - // already published), Lance's transaction conflict resolver fires - // `RetryableCommitConflict` on Update vs Update on the same row. - // That error gets wrapped as `OmniError::Lance()` and the - // API surfaces it as **500 internal**, not 409. Users see "internal - // server error" instead of a retryable conflict, breaking the - // documented 409 contract for in-process drift. - // - // Post-fix invariant: `commit_all` does an op-kind-aware drift check - // before each `commit_staged`. For tables whose tracked op_kind has - // `strict_pre_stage_version_check() == true` (Update / Delete / - // SchemaRewrite), if the staged dataset's version doesn't match the - // fresh manifest pin, return `OmniError::manifest_expected_version_mismatch` - // → 409 ExpectedVersionMismatch. The N-1 losers see a clean 409 - // before Lance's commit_staged ever runs. - // - // Why correct-by-design: closing the class "Lance internal conflict - // surfaces as 500 instead of 409" rather than mapping the specific - // Lance error variant. The drift check fires at the right architectural - // layer (engine boundary, under the queue) and respects the existing - // `MutationOpKind` policy. +async fn change_concurrent_updates_same_key_return_typed_pre_effect_conflicts() { + // Strict read-modify-write attempts are never automatically reprepared. + // Exactly one concurrent UPDATE commits; once it changes branch authority, + // every waiter reports a typed 409 before any of its Lance effects begin. let temp = init_loaded_graph().await; let graph = graph_path(temp.path()); let state = AppState::open(graph.to_string_lossy().to_string()) @@ -1373,27 +1327,37 @@ async fn change_concurrent_updates_same_key_serialize_via_publisher_cas() { assert_eq!( ok_count, 1, - "expected exactly one update to commit and N-1 to receive 409 manifest_conflict \ - (op-kind-aware drift check rejects stale-V0 staged datasets at commit_all entry). \ - Got {} OK + {} 409 + {} other. \ - Pre-fix symptom: 1 OK + (N-1) x 500 because Lance's RetryableCommitConflict for \ - Update vs Update on the same row bubbles up as `OmniError::Lance()` and \ - the API maps it to 500 internal, not 409. Statuses: {:?}", + "expected exactly one update to commit and N-1 to receive typed 409 conflicts \ + before effects. Got {} OK + {} 409 + {} other. Statuses: {:?}", ok_count, conflict_count, statuses.len() - ok_count - conflict_count, statuses, ); + + for (status, bytes) in &results { + if *status != StatusCode::CONFLICT { + continue; + } + let error: ErrorOutput = serde_json::from_slice(bytes).unwrap(); + assert_eq!(error.code, Some(omnigraph_server::api::ErrorCode::Conflict)); + let conflict = error + .read_set_conflict + .expect("strict OCC loser must include structured read-set authority"); + assert_eq!(conflict.member, "graph_head:main"); + assert_ne!(conflict.actual, conflict.expected); + assert!(error.manifest_conflict.is_none()); + assert!(error.recovery_required.is_none()); + } } #[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn change_disjoint_table_concurrency_succeeds_at_http_level() { - // HTTP-level pin for MR-686's disjoint-table promise: concurrent /change - // requests touching different node types must coexist without admission - // rejection or publisher-CAS conflict. The bench harness measures - // throughput; this test is the regression sentinel that catches a - // future change which accidentally re-introduces graph-wide - // serialization on the disjoint path. +async fn change_disjoint_table_concurrency_succeeds_under_branch_occ_gate() { + // RFC-022 intentionally serializes effect publication per branch because + // graph-head authority protects validation dependencies across tables. + // Disjoint retryable inserts must nevertheless all succeed through bounded + // full-attempt repreparation, without admission rejection or a user-visible + // publisher conflict. // // Setup: test.jsonl seeds 4 Persons + 2 Companies. Spawn N=4 concurrent // /change inserts on `node:Person` and N=4 concurrent inserts on diff --git a/crates/omnigraph-server/tests/openapi.rs b/crates/omnigraph-server/tests/openapi.rs index 2341bbc2..060869b3 100644 --- a/crates/omnigraph-server/tests/openapi.rs +++ b/crates/omnigraph-server/tests/openapi.rs @@ -391,7 +391,9 @@ const EXPECTED_SCHEMAS: &[&str] = &[ "MergeConflictOutput", "ReadOutput", "ReadRequest", + "ReadSetConflictOutput", "ReadTargetOutput", + "RecoveryRequiredOutput", "ManifestConflictOutput", "SchemaApplyOutput", "SchemaApplyRequest", @@ -614,6 +616,8 @@ fn error_output_schema_has_expected_fields() { assert!(props.contains_key("code")); assert!(props.contains_key("merge_conflicts")); assert!(props.contains_key("manifest_conflict")); + assert!(props.contains_key("read_set_conflict")); + assert!(props.contains_key("recovery_required")); } #[test] @@ -626,6 +630,24 @@ fn manifest_conflict_output_schema_has_expected_fields() { assert!(props.contains_key("actual")); } +#[test] +fn read_set_conflict_output_schema_has_expected_fields() { + let doc = openapi_json(); + let schema = &doc["components"]["schemas"]["ReadSetConflictOutput"]; + let props = schema["properties"].as_object().unwrap(); + assert!(props.contains_key("member")); + assert!(props.contains_key("expected")); + assert!(props.contains_key("actual")); +} + +#[test] +fn recovery_required_output_schema_has_expected_fields() { + let doc = openapi_json(); + let schema = &doc["components"]["schemas"]["RecoveryRequiredOutput"]; + let props = schema["properties"].as_object().unwrap(); + assert!(props.contains_key("operation_id")); +} + #[test] fn commit_output_schema_has_expected_fields() { let doc = openapi_json(); @@ -694,12 +716,21 @@ fn error_code_schema_has_expected_variants() { let schema = &doc["components"]["schemas"]["ErrorCode"]; let variants = schema["enum"].as_array().unwrap(); let values: HashSet<&str> = variants.iter().map(|v| v.as_str().unwrap()).collect(); - assert!(values.contains("unauthorized")); - assert!(values.contains("forbidden")); - assert!(values.contains("bad_request")); - assert!(values.contains("not_found")); - assert!(values.contains("conflict")); - assert!(values.contains("internal")); + assert_eq!( + values, + HashSet::from([ + "unauthorized", + "forbidden", + "bad_request", + "not_found", + "method_not_allowed", + "conflict", + "too_many_requests", + "internal", + ]), + "ErrorCode is a rolling wire contract: new meanings belong in optional \ + structured fields, not new closed-enum values", + ); } #[test] @@ -919,6 +950,32 @@ fn error_responses_reference_error_output_schema() { } } +#[test] +fn recovery_barrier_write_endpoints_document_recovery_required() { + let doc = openapi_json(); + for (path, method) in [ + ("/graphs/{graph_id}/change", "post"), + ("/graphs/{graph_id}/mutate", "post"), + ("/graphs/{graph_id}/queries/{name}", "post"), + ("/graphs/{graph_id}/load", "post"), + ("/graphs/{graph_id}/ingest", "post"), + ("/graphs/{graph_id}/branches", "post"), + ("/graphs/{graph_id}/branches/{branch}", "delete"), + ("/graphs/{graph_id}/branches/merge", "post"), + ] { + let response = &doc["paths"][path][method]["responses"]["503"]; + assert!( + response.is_object(), + "{method} {path} must document the recovery-required 503 outcome" + ); + assert_eq!( + response["content"]["application/json"]["schema"]["$ref"], + "#/components/schemas/ErrorOutput", + "{method} {path} 503 must use ErrorOutput" + ); + } +} + // --------------------------------------------------------------------------- // Request body reference tests // --------------------------------------------------------------------------- diff --git a/crates/omnigraph-server/tests/schema_routes.rs b/crates/omnigraph-server/tests/schema_routes.rs index c73591ca..9acc9018 100644 --- a/crates/omnigraph-server/tests/schema_routes.rs +++ b/crates/omnigraph-server/tests/schema_routes.rs @@ -371,8 +371,7 @@ async fn schema_apply_route_can_rename_property() { #[tokio::test] async fn schema_apply_route_can_add_index() { - let (temp, app) = app_for_graph_with_auth_tokens_and_policy( - &fs::read_to_string(fixture("test.pg")).unwrap(), + let (temp, app) = app_for_loaded_graph_with_auth_tokens_and_policy( &[("act-ragnor", "admin-token")], SCHEMA_APPLY_POLICY_YAML, ) @@ -392,7 +391,9 @@ async fn schema_apply_route_can_add_index() { .header("authorization", "Bearer admin-token") .body(Body::from( serde_json::to_vec(&SchemaApplyRequest { - schema_source: indexed_name_schema(), + schema_source: fs::read_to_string(fixture("test.pg")) + .unwrap() + .replace("age: I32?", "age: I32? @index"), ..Default::default() }) .unwrap(), @@ -404,9 +405,24 @@ async fn schema_apply_route_can_add_index() { assert_eq!(payload["applied"], true); // iss-848: the /schema/apply route accepts the index-add and applies it as a // metadata change — it records the `@index` intent in the catalog/IR but does - // NOT build the physical index inline (the build is deferred to - // ensure_indices/optimize; on this empty table nothing would build anyway). - // So the physical index count is unchanged by the apply. + // NOT build the physical index inline or spawn a server-only build (the + // build is deferred to explicit ensure_indices/optimize maintenance). + // Pin the detached-work half structurally instead of waiting an arbitrary + // amount of wall-clock time for a task that must not exist. The physical + // assertion below independently covers synchronous/awaited index builds. + let handlers = include_str!("../src/handlers.rs"); + let (_, after_apply_signature) = handlers + .split_once("pub(crate) async fn server_schema_apply") + .expect("server_schema_apply handler must remain present"); + let (apply_handler, _) = after_apply_signature + .split_once("/// Shared body for `POST /load`") + .expect("schema apply handler must precede the load handler"); + assert!( + !apply_handler.contains("tokio::spawn") && !apply_handler.contains(".ensure_indices("), + "schema apply must not schedule implicit physical index convergence" + ); + + // The physical index count is unchanged by the apply. let reopened = Omnigraph::open(graph.to_str().unwrap()).await.unwrap(); let snapshot = reopened .snapshot_of(ReadTarget::branch("main")) diff --git a/crates/omnigraph-server/tests/support/mod.rs b/crates/omnigraph-server/tests/support/mod.rs index 694db467..6a7c0f82 100644 --- a/crates/omnigraph-server/tests/support/mod.rs +++ b/crates/omnigraph-server/tests/support/mod.rs @@ -502,12 +502,6 @@ pub fn renamed_age_schema() -> String { .replace("age: I32?", "years: I32? @rename_from(\"age\")") } -pub fn indexed_name_schema() -> String { - fs::read_to_string(fixture("test.pg")) - .unwrap() - .replace("name: String @key", "name: String @key @index") -} - pub fn unsupported_schema_change() -> String { fs::read_to_string(fixture("test.pg")) .unwrap() diff --git a/crates/omnigraph/src/db/commit_graph.rs b/crates/omnigraph/src/db/commit_graph.rs index ad6831ad..8a6814f1 100644 --- a/crates/omnigraph/src/db/commit_graph.rs +++ b/crates/omnigraph/src/db/commit_graph.rs @@ -49,6 +49,11 @@ impl CommitGraph { /// keeps the cache consistent for same-handle reads, with no storage I/O. /// Head selection matches the manifest-sourced load (`should_replace_head`). pub fn insert_committed(&mut self, commit: GraphCommit) { + debug_assert_eq!( + commit.manifest_branch.as_deref(), + self.active_branch.as_deref(), + "published lineage must target the commit graph's active branch" + ); if should_replace_head(self.head_commit.as_ref(), &commit) { self.head_commit = Some(commit.clone()); } @@ -71,7 +76,8 @@ impl CommitGraph { let root = root_uri.trim_end_matches('/'); // `load_commit_cache_for_branch` opens the branch's `__manifest` (the // authoritative table), so a truly absent branch fails loudly here. - let (commit_by_id, head_commit) = load_commit_cache_for_branch(root, Some(branch)).await?; + let (commit_by_id, head_commit) = + load_commit_cache_for_branch(root, Some(branch)).await?; Ok(Self { root_uri: root.to_string(), active_branch: Some(branch.to_string()), @@ -180,7 +186,7 @@ async fn load_commit_cache_from_manifest( root_uri: &str, branch: Option<&str>, ) -> Result<(HashMap, Option)> { - let (rows, _heads) = + let (rows, _) = crate::db::manifest::ManifestCoordinator::read_graph_lineage_at(root_uri, branch).await?; let mut commit_by_id = HashMap::with_capacity(rows.len()); let mut head_commit = None; diff --git a/crates/omnigraph/src/db/graph_coordinator.rs b/crates/omnigraph/src/db/graph_coordinator.rs index 0b016f4d..ca9b9d3f 100644 --- a/crates/omnigraph/src/db/graph_coordinator.rs +++ b/crates/omnigraph/src/db/graph_coordinator.rs @@ -11,7 +11,8 @@ use crate::storage::{StorageAdapter, normalize_root_uri}; use super::commit_graph::{CommitGraph, GraphCommit}; use super::is_internal_system_branch; use super::manifest::{ - ManifestChange, ManifestCoordinator, ManifestIncarnation, Snapshot, SubTableUpdate, + LineageIntent, ManifestChange, ManifestCoordinator, ManifestIncarnation, PublishPrecondition, + Snapshot, SubTableUpdate, }; #[derive(Debug, Clone, PartialEq, Eq, Hash)] @@ -167,6 +168,20 @@ impl GraphCoordinator { self.manifest.incarnation() } + /// Lance-native identity of the active `__manifest` branch. Stable across + /// commits; changes when a named branch is deleted and recreated. + pub(crate) async fn branch_identifier(&self) -> Result { + self.manifest.branch_identifier().await + } + + /// Exact `graph_head:` pointer, preserving `None` for a + /// freshly-created named branch even though its inherited commit history has + /// an inferred head. Sourced from the manifest coordinator's SAME pinned + /// state as [`Self::snapshot`], not the separately refreshed lineage cache. + pub(crate) fn exact_graph_head(&self) -> Option { + self.manifest.exact_graph_head() + } + pub fn snapshot(&self) -> Snapshot { self.manifest.snapshot() } @@ -393,10 +408,35 @@ impl GraphCoordinator { actor_id: Option<&str>, ) -> Result { let intent = self.new_lineage_intent(actor_id, None)?; + self.commit_changes_with_intent_and_expected( + changes, + expected_table_versions, + intent, + &PublishPrecondition::Any, + ) + .await + } + + /// Publish a pre-minted lineage intent under an explicit authority + /// precondition. The intent's identity and timestamp remain stable across + /// publisher retries and can also be persisted by the caller's recovery + /// protocol before this method is invoked. + pub(crate) async fn commit_changes_with_intent_and_expected( + &mut self, + changes: &[ManifestChange], + expected_table_versions: &HashMap, + intent: LineageIntent, + precondition: &PublishPrecondition, + ) -> Result { failpoints::maybe_fail(crate::failpoints::names::GRAPH_PUBLISH_BEFORE_COMMIT_APPEND)?; let outcome = self .manifest - .commit_changes_with_lineage(changes, expected_table_versions, Some(&intent)) + .commit_changes_with_lineage_and_precondition( + changes, + expected_table_versions, + Some(&intent), + precondition, + ) .await?; failpoints::maybe_fail(crate::failpoints::names::GRAPH_PUBLISH_AFTER_MANIFEST_COMMIT)?; let snapshot_id = self.apply_lineage_to_cache(intent, &outcome); @@ -434,12 +474,12 @@ impl GraphCoordinator { /// fresh ULID (stable across the publisher's CAS retries) and a timestamp. /// The parent is NOT chosen here — the publisher resolves it per attempt /// against the manifest it commits against. - fn new_lineage_intent( + pub(crate) fn new_lineage_intent( &self, actor_id: Option<&str>, merged_parent_commit_id: Option, - ) -> Result { - Ok(crate::db::manifest::LineageIntent { + ) -> Result { + Ok(LineageIntent { graph_commit_id: ulid::Ulid::new().to_string(), branch: self.current_branch().map(str::to_string), actor_id: actor_id.map(str::to_string), diff --git a/crates/omnigraph/src/db/manifest.rs b/crates/omnigraph/src/db/manifest.rs index 9b9853d2..2458cb25 100644 --- a/crates/omnigraph/src/db/manifest.rs +++ b/crates/omnigraph/src/db/manifest.rs @@ -28,25 +28,26 @@ mod recovery; mod state; use graph::{init_manifest_graph, open_manifest_graph, snapshot_state_at}; -use layout::{open_manifest_dataset, table_uri_for_path, type_name_hash}; pub(crate) use layout::manifest_uri; +use layout::{open_manifest_dataset, table_uri_for_path, type_name_hash}; pub(crate) use metadata::TableVersionMetadata; #[cfg(test)] use metadata::{OMNIGRAPH_ROW_COUNT_KEY, table_version_metadata_for_state}; #[cfg(test)] use namespace::{branch_manifest_namespace, staged_table_namespace}; -pub(crate) use publisher::LineageIntent; +pub(crate) use publisher::{GraphHeadExpectation, LineageIntent, PublishPrecondition}; use publisher::{GraphNamespacePublisher, ManifestBatchPublisher, PublishOutcome}; pub(crate) use recovery::{ - RecoveryMode, RecoverySidecar, RecoverySidecarHandle, SidecarKind, SidecarTablePin, - SidecarTableRegistration, SidecarTombstone, confirm_sidecar_phase_b, delete_sidecar, - has_schema_apply_sidecar, heal_pending_sidecars_roll_forward, list_sidecars, new_sidecar, - recover_manifest_drift, schema_apply_serial_queue_key, write_sidecar, + HealPendingOutcome, RecoveryAuthorityToken, RecoveryLineageIntent, RecoveryMode, + RecoverySidecar, RecoverySidecarHandle, SidecarKind, SidecarTablePin, SidecarTableRegistration, + SidecarTombstone, confirm_occ_sidecar_phase_b, confirm_sidecar_phase_b, delete_sidecar, + has_schema_apply_sidecar, heal_pending_sidecars_roll_forward, list_sidecars, new_occ_sidecar, + new_sidecar, recover_manifest_drift, schema_apply_serial_queue_key, write_sidecar, }; pub use state::SubTableEntry; -pub(crate) use state::{GraphLineageRow, read_graph_lineage}; #[cfg(test)] use state::string_column; +pub(crate) use state::{GraphLineageRow, read_graph_lineage}; use state::{ManifestState, read_manifest_state}; /// The internal-schema (storage-format) version this binary writes and reads. @@ -498,7 +499,30 @@ impl ManifestCoordinator { expected_table_versions: &HashMap, lineage: Option<&LineageIntent>, ) -> Result { - if changes.is_empty() && expected_table_versions.is_empty() && lineage.is_none() { + self.commit_changes_with_lineage_and_precondition( + changes, + expected_table_versions, + lineage, + &PublishPrecondition::Any, + ) + .await + } + + /// The token-aware form of [`commit_changes_with_lineage`]. Exact authority + /// is checked by the publisher from every CAS attempt's existing one-scan + /// state; legacy callers continue through `Any` above. + pub(crate) async fn commit_changes_with_lineage_and_precondition( + &mut self, + changes: &[ManifestChange], + expected_table_versions: &HashMap, + lineage: Option<&LineageIntent>, + precondition: &PublishPrecondition, + ) -> Result { + if changes.is_empty() + && expected_table_versions.is_empty() + && lineage.is_none() + && matches!(precondition, PublishPrecondition::Any) + { return Ok(CommitOutcome { version: self.version(), parent_commit_id: None, @@ -511,7 +535,7 @@ impl ManifestCoordinator { known_state, } = self .publisher - .publish(changes, expected_table_versions, lineage) + .publish_with_precondition(changes, expected_table_versions, lineage, precondition) .await?; // RFC-013 PR2 #1b: the publisher folded the new visible state in-memory // (byte-identical to a re-scan via the shared `assemble_manifest_state`), @@ -550,6 +574,28 @@ impl ManifestCoordinator { .map_err(|e| OmniError::Lance(e.to_string())) } + /// Lance-native stable identity for the active manifest branch. Unlike a + /// manifest version/eTag, this remains stable across ordinary commits and + /// changes when a named branch is deleted and recreated (ABA protection). + pub(crate) async fn branch_identifier(&self) -> Result { + self.dataset + .branch_identifier() + .await + .map_err(|e| OmniError::Lance(e.to_string())) + } + + /// Exact materialized `graph_head:` from the same pinned + /// manifest version as [`Self::snapshot`]. This is write authority, not a + /// lineage-cache query: a read may refresh only the manifest, so consulting + /// `CommitGraph` here would combine a fresh table snapshot with a stale head. + pub(crate) fn exact_graph_head(&self) -> Option { + let branch_key = self + .active_branch + .as_deref() + .unwrap_or(MAIN_BRANCH_HEAD_KEY); + self.known_state.graph_heads.get(branch_key).cloned() + } + pub(crate) fn incarnation(&self) -> ManifestIncarnation { ManifestIncarnation { version: self.version(), diff --git a/crates/omnigraph/src/db/manifest/publisher.rs b/crates/omnigraph/src/db/manifest/publisher.rs index 39d7932a..e448344b 100644 --- a/crates/omnigraph/src/db/manifest/publisher.rs +++ b/crates/omnigraph/src/db/manifest/publisher.rs @@ -36,12 +36,12 @@ use super::metadata::{TableVersionMetadata, parse_namespace_version_request}; use super::migrations::{read_stamp, refuse_if_stamp_unsupported}; use super::state::{ GraphLineageRow, GraphLineageRowPart, ManifestState, assemble_manifest_state, - graph_lineage_row_parts, head_lineage_row, manifest_rows_batch, manifest_schema, - read_manifest_state, read_publish_scan, + graph_head_object_id, graph_lineage_row_parts, head_lineage_row, manifest_rows_batch, + manifest_schema, read_manifest_state, read_publish_scan, }; use super::{ - ManifestChange, OBJECT_TYPE_TABLE, OBJECT_TYPE_TABLE_TOMBSTONE, OBJECT_TYPE_TABLE_VERSION, - SubTableEntry, TableRegistration, TableTombstone, + MAIN_BRANCH_HEAD_KEY, ManifestChange, OBJECT_TYPE_TABLE, OBJECT_TYPE_TABLE_TOMBSTONE, + OBJECT_TYPE_TABLE_VERSION, SubTableEntry, TableRegistration, TableTombstone, }; /// Bound on the publisher-level retry loop that wraps Lance's row-level CAS @@ -54,11 +54,10 @@ const PUBLISHER_RETRY_BUDGET: u32 = 5; /// The graph-lineage commit to record atomically with a manifest publish /// (RFC-013 Phase 7). One logical commit per publish: the `graph_commit_id` is /// minted once by the caller and stays stable across the publisher's CAS -/// retries; only the parent re-resolves per attempt (against the freshly loaded -/// `__manifest`), so a retry after a concurrent commit parents off the new head -/// — the TOCTOU the dual-write era's `commit_graph.refresh()` guarded is closed -/// by construction. -#[derive(Debug, Clone)] +/// retries. Legacy [`PublishPrecondition::Any`] publishes re-resolve the parent +/// per attempt. An exact-head publish instead rejects a retry once that authority +/// changed, so a prepared write can never be silently re-parented. +#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)] pub(crate) struct LineageIntent { /// ULID minted once before the publish loop; the graph commit's identity. pub graph_commit_id: String, @@ -73,6 +72,55 @@ pub(crate) struct LineageIntent { pub created_at: i64, } +/// The exact mutable graph-head authority a prepared write observed. A missing +/// row is first-class: a freshly-created named branch inherits lineage commits +/// but has no `graph_head:` until its first graph commit. +#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +pub(crate) struct GraphHeadExpectation { + /// `None` means main; `Some("main")` is normalized to `None` by [`new`]. + pub(crate) branch: Option, + /// Lance-native stable branch identity. This detects delete/recreate ABA; + /// manifest versions/eTags are deliberately not branch identity. + pub(crate) branch_identifier: lance::dataset::refs::BranchIdentifier, + /// Exact commit id stored in the branch's head row, or `None` when absent. + pub(crate) head_commit_id: Option, +} + +impl GraphHeadExpectation { + pub(crate) fn new( + branch: Option<&str>, + branch_identifier: lance::dataset::refs::BranchIdentifier, + head_commit_id: Option, + ) -> Self { + Self { + branch: branch + .filter(|branch| *branch != "main") + .map(ToOwned::to_owned), + branch_identifier, + head_commit_id, + } + } + + fn object_id(&self) -> String { + graph_head_object_id(self.branch.as_deref()) + } +} + +/// Authority checked by the manifest publisher on every CAS attempt. +/// +/// `Any` preserves the legacy dispatcher semantics: row-level contention may +/// retry and re-parent a lineage intent. `ExactGraphHead` is the RFC-022 +/// foundation for prepared writes: after contention, any head movement becomes +/// `ReadSetChanged` rather than a transparent re-parent. Its native branch-id +/// check detects delete/recreate ABA on every attempt; it is not a distributed +/// ref-control fence (Lance branch create/delete still lacks conditional CAS), +/// so branch control remains within the documented single-writer-process bound. +#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +pub(crate) enum PublishPrecondition { + Any, + ExactGraphHead(GraphHeadExpectation), +} + /// The result of a manifest publish that may have folded in a graph commit. #[derive(Debug)] pub(super) struct PublishOutcome { @@ -92,11 +140,29 @@ pub(super) struct PublishOutcome { #[async_trait] pub(super) trait ManifestBatchPublisher: Send + Sync { + /// Legacy publish behavior. All existing writers deliberately retain + /// `Any` until enrolled in their RFC-022 adapter. async fn publish( &self, changes: &[ManifestChange], expected_table_versions: &HashMap, lineage: Option<&LineageIntent>, + ) -> Result { + self.publish_with_precondition( + changes, + expected_table_versions, + lineage, + &PublishPrecondition::Any, + ) + .await + } + + async fn publish_with_precondition( + &self, + changes: &[ManifestChange], + expected_table_versions: &HashMap, + lineage: Option<&LineageIntent>, + precondition: &PublishPrecondition, ) -> Result; } @@ -119,15 +185,16 @@ struct PendingVersionRow { /// Everything one CAS attempt needs out of a single `__manifest` scan /// (RFC-013 P2): the open dataset, table state for the pre-check + pending-row -/// build, and the `graph_commit` lineage rows for parent resolution. Folding the -/// lineage into this struct is what lets `resolve_lineage_rows` skip its own -/// `read_graph_lineage` scan. +/// build, `graph_commit` lineage rows for parent resolution, and exact +/// `graph_head` rows for OCC. Folding lineage authority into this struct is what +/// lets both checks skip their own `read_graph_lineage` scan. struct LoadedPublishState { dataset: Dataset, registered_tables: HashMap, existing_versions: HashMap<(String, u64), SubTableEntry>, existing_tombstones: HashMap<(String, u64), ()>, lineage_rows: Vec, + graph_heads: HashMap, } impl GraphNamespacePublisher { @@ -159,12 +226,11 @@ impl GraphNamespacePublisher { // `db/manifest/migrations.rs`. refuse_if_stamp_unsupported(read_stamp(&dataset))?; // ONE `__manifest` scan for everything the publish needs: table - // locations, version entries, tombstones, AND the `graph_commit` lineage - // rows for parent resolution (RFC-013 P2). The lineage extraction rides - // this pass instead of a second `read_graph_lineage` scan in - // `resolve_lineage_rows`; the per-attempt re-read is preserved because - // `load_publish_state` runs once per CAS attempt, so a retry sees the - // advanced head and re-parents correctly. + // locations, version entries, tombstones, `graph_commit` lineage rows + // for parent resolution, AND exact `graph_head` rows for OCC (RFC-013 + // P2 / RFC-022). Extraction rides this pass instead of a second + // `read_graph_lineage` scan; the per-attempt re-read is preserved because + // `load_publish_state` runs once per CAS attempt. let scan = read_publish_scan(&dataset).await?; let existing_versions = scan .version_entries @@ -183,6 +249,7 @@ impl GraphNamespacePublisher { existing_versions, existing_tombstones, lineage_rows: scan.lineage_rows, + graph_heads: scan.graph_heads, }) } @@ -506,12 +573,15 @@ impl GraphNamespacePublisher { )) })?; let table_path = - table_locations.get(&row.table_key).cloned().ok_or_else(|| { - OmniError::manifest_internal(format!( - "post-publish fold: missing table row for {}", - row.table_key - )) - })?; + table_locations + .get(&row.table_key) + .cloned() + .ok_or_else(|| { + OmniError::manifest_internal(format!( + "post-publish fold: missing table row for {}", + row.table_key + )) + })?; let metadata_json = row.metadata.as_deref().ok_or_else(|| { OmniError::manifest_internal(format!( "post-publish fold: table_version row missing metadata for {}", @@ -572,6 +642,80 @@ impl GraphNamespacePublisher { Ok(()) } + /// Check authority inside the publisher retry loop before pending rows are + /// built. The graph head comes from the SAME scan used to build this CAS + /// attempt; Lance's native branch identifier is re-read from the ref. Thus a + /// row-level-CAS loser with an exact expectation cannot silently re-parent + /// on its next attempt. + async fn check_publish_precondition( + &self, + dataset: &Dataset, + graph_heads: &HashMap, + precondition: &PublishPrecondition, + ) -> Result<()> { + let PublishPrecondition::ExactGraphHead(expected) = precondition else { + return Ok(()); + }; + + let expected_branch = expected + .branch + .as_deref() + .filter(|branch| *branch != "main"); + if expected_branch != self.branch.as_deref() { + return Err(OmniError::manifest_internal(format!( + "publish graph-head precondition targets branch '{}' but publisher is bound to '{}'", + expected_branch.unwrap_or("main"), + self.branch.as_deref().unwrap_or("main"), + ))); + } + + let branch_identity_member = + format!("branch_identifier:{}", expected_branch.unwrap_or("main")); + let expected_branch_identifier = serde_json::to_string(&expected.branch_identifier) + .map_err(|e| { + OmniError::manifest_internal(format!( + "failed to encode expected Lance branch identifier: {e}" + )) + })?; + let actual_branch_identifier = match dataset.branch_identifier().await { + Ok(identifier) => identifier, + Err(LanceError::RefNotFound { .. }) => { + return Err(OmniError::manifest_read_set_changed( + branch_identity_member, + Some(expected_branch_identifier), + None, + )); + } + Err(err) => return Err(OmniError::Lance(err.to_string())), + }; + if actual_branch_identifier != expected.branch_identifier { + let actual = serde_json::to_string(&actual_branch_identifier).map_err(|e| { + OmniError::manifest_internal(format!( + "failed to encode current Lance branch identifier: {e}" + )) + })?; + return Err(OmniError::manifest_read_set_changed( + branch_identity_member, + Some(expected_branch_identifier), + Some(actual), + )); + } + + let object_id = expected.object_id(); + let branch_key = object_id + .strip_prefix("graph_head:") + .expect("graph_head_object_id always supplies the prefix"); + let actual = graph_heads.get(branch_key).cloned(); + if actual != expected.head_commit_id { + return Err(OmniError::manifest_read_set_changed( + object_id, + expected.head_commit_id.clone(), + actual, + )); + } + Ok(()) + } + async fn merge_rows(&self, dataset: Dataset, rows: Vec) -> Result { let batch = Self::pending_rows_to_batch(rows)?; let reader = RecordBatchIterator::new(vec![Ok(batch)], manifest_schema()); @@ -671,13 +815,18 @@ pub(crate) fn map_lance_publish_error(err: LanceError) -> OmniError { #[async_trait] impl ManifestBatchPublisher for GraphNamespacePublisher { - async fn publish( + async fn publish_with_precondition( &self, changes: &[ManifestChange], expected_table_versions: &HashMap, lineage: Option<&LineageIntent>, + precondition: &PublishPrecondition, ) -> Result { - if changes.is_empty() && expected_table_versions.is_empty() && lineage.is_none() { + if changes.is_empty() + && expected_table_versions.is_empty() + && lineage.is_none() + && matches!(precondition, PublishPrecondition::Any) + { // Defensive no-op (never reached from `commit_changes_with_lineage`, // which short-circuits the all-empty case): state is unchanged, so a // re-scan here is acceptable. @@ -709,8 +858,16 @@ impl ManifestBatchPublisher for GraphNamespacePublisher { existing_versions, existing_tombstones, lineage_rows, + graph_heads, } = loaded; + // Exact logical authority is checked on EVERY attempt from this + // attempt's single manifest scan. In particular, a CAS retry after + // another writer creates or advances `graph_head:` fails + // here instead of transparently re-parenting the prepared intent. + self.check_publish_precondition(&dataset, &graph_heads, precondition) + .await?; + let latest_per_table = Self::latest_visible_per_table(&existing_versions, &existing_tombstones); // Pre-check on every attempt against freshly loaded state so a @@ -754,6 +911,7 @@ impl ManifestBatchPublisher for GraphNamespacePublisher { existing_tombstones .keys() .map(|(key, version)| (key.clone(), *version)), + graph_heads, ); return Ok(PublishOutcome { dataset, @@ -765,8 +923,23 @@ impl ManifestBatchPublisher for GraphNamespacePublisher { // Build the post-publish fold inputs from the pre-publish state ∪ the // rows we are about to commit, BEFORE `rows` is moved into merge_rows // (RFC-013 PR2 #1b). Recomputed per attempt from freshly-loaded state. - let (fold_entries, fold_tombstones) = - Self::fold_inputs(&existing_versions, &existing_tombstones, &rows, &known_tables)?; + let (fold_entries, fold_tombstones) = Self::fold_inputs( + &existing_versions, + &existing_tombstones, + &rows, + &known_tables, + )?; + let mut fold_graph_heads = graph_heads; + if let Some(intent) = lineage { + fold_graph_heads.insert( + intent + .branch + .as_deref() + .unwrap_or(MAIN_BRANCH_HEAD_KEY) + .to_string(), + intent.graph_commit_id.clone(), + ); + } match self.merge_rows(dataset, rows).await { Ok(new_dataset) => { @@ -774,6 +947,7 @@ impl ManifestBatchPublisher for GraphNamespacePublisher { new_dataset.version().version, fold_entries, fold_tombstones, + fold_graph_heads, ); return Ok(PublishOutcome { dataset: new_dataset, diff --git a/crates/omnigraph/src/db/manifest/recovery.rs b/crates/omnigraph/src/db/manifest/recovery.rs index fbc02be1..e272149e 100644 --- a/crates/omnigraph/src/db/manifest/recovery.rs +++ b/crates/omnigraph/src/db/manifest/recovery.rs @@ -28,13 +28,12 @@ //! CreateIndex/Merge — see `check_restore_txn` at lance-6.0.1 //! `src/io/commit/conflict_resolver.rs:986`. The hazard is documented //! by `tests/staged_writes.rs::lance_restore_loses_to_concurrent_append_via_orphaning`. -//! The open-time sweep sidesteps the hazard by running before any -//! other writers can race; the in-process heal -//! ([`heal_pending_sidecars_roll_forward`]) never restores (roll- -//! forward only) and guards via per-(table_key, branch) queue -//! acquisition. +//! The open-time sweep and live healer both join the root-scoped ordered +//! gates. The in-process healer ([`heal_pending_sidecars_roll_forward`]) +//! additionally never restores (roll-forward only); foreign processes remain +//! outside this serialization boundary. -use std::collections::HashMap; +use std::collections::{HashMap, HashSet}; use serde::{Deserialize, Serialize}; use tracing::warn; @@ -44,30 +43,38 @@ use crate::db::recovery_audit::{RecoveryAudit, RecoveryAuditRecord, RecoveryKind use crate::db::schema_state::SchemaStateRecovery; use crate::error::{OmniError, Result}; use crate::storage::StorageAdapter; +use crate::table_store::StagedTransactionIdentity; use super::Snapshot; -use super::publisher::{GraphNamespacePublisher, LineageIntent, ManifestBatchPublisher}; -use super::{ManifestChange, SubTableUpdate, TableRegistration, TableTombstone}; +use super::publisher::{ + GraphHeadExpectation, GraphNamespacePublisher, LineageIntent, ManifestBatchPublisher, + PublishPrecondition, +}; +use super::{ + ManifestChange, ManifestCoordinator, SubTableUpdate, TableRegistration, TableTombstone, +}; -/// System actor identifier recorded on every recovery commit. Operators -/// distinguish recovery commits from user commits in `omnigraph commit list` -/// by filtering on this actor; the original sidecar's actor (if any) flows -/// into the audit row's `recovery_for_actor` field. +/// System actor identifier for recovery-owned lineage: legacy recovery, +/// v3 rollback, and orphan discard. A v3 roll-forward deliberately publishes +/// the original writer's fixed lineage and actor instead. The recovery audit +/// row is the authoritative attribution surface in both cases; the original +/// sidecar actor flows into its `recovery_for_actor` field. pub(crate) const RECOVERY_ACTOR: &str = "omnigraph:recovery"; -/// Publish a recovery action's manifest `updates` AND its recovery commit in one -/// CAS (RFC-013 Phase 7). The recovery commit's lineage (`graph_commit` + -/// `graph_head`) rides the same merge-insert as the table-version re-pin — there -/// is no separate `_graph_commits.lance` write and no manifest→commit-graph gap. +/// Publish a recovery action's manifest `updates` AND its lineage in one CAS +/// (RFC-013 Phase 7). Legacy recovery and v3 rollback publish a recovery commit; +/// v3 roll-forward publishes the original writer's fixed lineage intent. The +/// lineage (`graph_commit` + `graph_head`) rides the same merge-insert as the +/// table-version re-pin — there is no separate `_graph_commits.lance` write and +/// no manifest→commit-graph gap. /// `updates` is empty for the no-table-change recovery paths (all-NoMovement /// roll-back, stale-sidecar cleanup, orphaned-branch discard); the lineage rows /// still publish, so the recovery commit is always durable. /// -/// The commit's first parent is resolved by the publisher (the live head of the -/// recovery's branch); its merged-in parent is the sidecar's recorded source -/// head for a rolled-forward branch merge, matching the pre-Phase-7 merge-commit -/// shape. Returns the new manifest version and the minted recovery commit id -/// (which the audit row references). +/// Legacy commits resolve their parent from the live branch head. A v3 +/// roll-forward instead carries an exact graph-head precondition, so it cannot +/// be silently re-parented after authority changes. Returns the new manifest +/// version and the stable commit id the audit row references. async fn publish_recovery_commit( root_uri: &str, sidecar: &RecoverySidecar, @@ -75,21 +82,53 @@ async fn publish_recovery_commit( updates: &[ManifestChange], expected: &HashMap, ) -> Result<(u64, String)> { - let merged_parent_commit_id = match (sidecar.writer_kind, kind) { - (SidecarKind::BranchMerge, RecoveryKind::RolledForward) => { - sidecar.merge_source_commit_id.clone() + let intent = match (sidecar.protocol_v3.as_ref(), kind) { + (Some(protocol), RecoveryKind::RolledForward) => LineageIntent::from(&protocol.lineage), + (Some(protocol), RecoveryKind::RolledBack) => LineageIntent { + graph_commit_id: protocol.rollback_graph_commit_id.clone(), + branch: sidecar.branch.clone(), + actor_id: Some(RECOVERY_ACTOR.to_string()), + merged_parent_commit_id: None, + created_at: sidecar + .started_at + .parse::() + .unwrap_or(crate::db::now_micros()?), + }, + (Some(_), RecoveryKind::OrphanedBranchDiscarded) => { + return Err(OmniError::manifest_internal( + "orphaned-branch discard cannot publish through the v3 OCC recovery path", + )); + } + (None, _) => { + let merged_parent_commit_id = match (sidecar.writer_kind, kind) { + (SidecarKind::BranchMerge, RecoveryKind::RolledForward) => { + sidecar.merge_source_commit_id.clone() + } + _ => None, + }; + LineageIntent { + graph_commit_id: ulid::Ulid::new().to_string(), + branch: sidecar.branch.clone(), + actor_id: Some(RECOVERY_ACTOR.to_string()), + merged_parent_commit_id, + created_at: crate::db::now_micros()?, + } } - _ => None, - }; - let intent = LineageIntent { - graph_commit_id: ulid::Ulid::new().to_string(), - branch: sidecar.branch.clone(), - actor_id: Some(RECOVERY_ACTOR.to_string()), - merged_parent_commit_id, - created_at: crate::db::now_micros()?, }; let publisher = GraphNamespacePublisher::new(root_uri, sidecar.branch.as_deref()); - let outcome = publisher.publish(updates, expected, Some(&intent)).await?; + let precondition = match (sidecar.protocol_v3.as_ref(), kind) { + (Some(protocol), RecoveryKind::RolledForward) => { + PublishPrecondition::ExactGraphHead(GraphHeadExpectation::new( + sidecar.branch.as_deref(), + protocol.authority.branch_identifier.clone(), + protocol.authority.graph_head.clone(), + )) + } + _ => PublishPrecondition::Any, + }; + let outcome = publisher + .publish_with_precondition(updates, expected, Some(&intent), &precondition) + .await?; Ok((outcome.dataset.version().version, intent.graph_commit_id)) } @@ -109,7 +148,18 @@ pub(crate) const RECOVERY_DIR_NAME: &str = "__recovery"; /// back); at v1 it predates confirmation and keeps the loose roll-forward. The /// reader must distinguish the two, so this is a real version bump, not an /// additive field. -pub(crate) const SIDECAR_SCHEMA_VERSION: u32 = 2; +/// +/// v2 → v3: RFC-022 exact mutation/load effects. An enrolled sidecar carries +/// the captured authority, stable lineage/rollback ids, planned Lance +/// transaction identities, a canonical manifest delta, and an explicit +/// Armed → EffectsConfirmed transition. Legacy writers deliberately continue +/// producing v2 until their adapter opts into [`new_occ_sidecar`]. +pub(crate) const SIDECAR_SCHEMA_VERSION: u32 = 3; + +/// Schema version emitted by the legacy constructor. Fixed at v2 so merely +/// teaching this binary to understand v3 does not silently enroll every writer +/// in new recovery semantics. +pub(crate) const LEGACY_SIDECAR_SCHEMA_VERSION: u32 = 2; /// The version at which Phase-B confirmation shipped. A `BranchMerge` sidecar is /// confirmation-aware (strict classification) iff `schema_version >=` this. @@ -117,6 +167,16 @@ pub(crate) const SIDECAR_SCHEMA_VERSION: u32 = 2; /// v3+ still treats v2 sidecars as confirmation-aware. pub(crate) const CONFIRMATION_SCHEMA_VERSION: u32 = 2; +/// Mutation/load exact-effect semantics shipped at v3. Fixed rather than tied +/// to the current maximum so future sidecar versions retain this floor. +pub(crate) const EXACT_EFFECT_IDENTITY_SCHEMA_VERSION: u32 = 3; + +/// Bound the cold-path transaction-history probe used to find a v3 sidecar's +/// UUID after unexpected drift. Normal recovery reads one version; a larger +/// gap indicates foreign activity and is surfaced as unverifiable rather than +/// turning open into an unbounded history walk. +const MAX_EFFECT_IDENTITY_SCAN_VERSIONS: u64 = 1024; + /// Selects which recovery actions are allowed in a sweep. /// /// Open-time recovery (`Omnigraph::open` with `OpenMode::ReadWrite`) @@ -184,6 +244,10 @@ pub(crate) enum ClassificationMode { /// CONFIRMATION_SCHEMA_VERSION`): roll forward ONLY to the confirmed /// version; an unconfirmed moved HEAD is a partial publish and rolls back. Confirmed, + /// RFC-022 mutation/load effect: the sidecar must be durably confirmed and + /// the observed Lance HEAD transaction `(read_version, uuid)` must exactly + /// equal the planned and confirmed transaction identity. + ExactEffect, } impl SidecarKind { @@ -192,7 +256,13 @@ impl SidecarKind { /// compile error here until its recovery semantics are declared. pub(crate) fn classification_mode(self, schema_version: u32) -> ClassificationMode { match self { - SidecarKind::Mutation | SidecarKind::Load => ClassificationMode::Strict, + SidecarKind::Mutation | SidecarKind::Load => { + if schema_version >= EXACT_EFFECT_IDENTITY_SCHEMA_VERSION { + ClassificationMode::ExactEffect + } else { + ClassificationMode::Strict + } + } // BranchMerge gained two-phase confirmation at // `CONFIRMATION_SCHEMA_VERSION`. A sidecar written before that // carries no `confirmed_version` and must keep the prior loose @@ -284,6 +354,114 @@ pub(crate) struct SidecarTombstone { pub tombstone_version: u64, } +/// Authority captured before an RFC-022 mutation/load attempt was prepared. +/// Recovery compares this complete token before it may roll confirmed effects +/// forward. The native Lance branch identifier closes delete/recreate ABA; +/// `graph_head: None` deliberately preserves absence on a fresh branch. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub(crate) struct RecoveryAuthorityToken { + pub branch_identifier: lance::dataset::refs::BranchIdentifier, + pub graph_head: Option, + pub schema_ir_hash: String, + pub schema_identity_version: u32, +} + +/// Serializable copy of the original graph lineage intent. It is minted once +/// before recovery is armed and never re-parented or replaced during recovery. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub(crate) struct RecoveryLineageIntent { + pub graph_commit_id: String, + pub branch: Option, + pub actor_id: Option, + pub merged_parent_commit_id: Option, + pub created_at: i64, +} + +impl From<&RecoveryLineageIntent> for LineageIntent { + fn from(intent: &RecoveryLineageIntent) -> Self { + Self { + graph_commit_id: intent.graph_commit_id.clone(), + branch: intent.branch.clone(), + actor_id: intent.actor_id.clone(), + merged_parent_commit_id: intent.merged_parent_commit_id.clone(), + created_at: intent.created_at, + } + } +} + +/// Durable phase of an RFC-022 effect set. `Armed` is written before the first +/// HEAD advance. `EffectsConfirmed` is written only after every planned table +/// effect landed with its exact transaction identity and full manifest update. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "PascalCase")] +pub(crate) enum RecoveryEffectPhase { + Armed, + EffectsConfirmed, +} + +/// One exact physical effect enrolled in an RFC-022 sidecar. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub(crate) struct RecoveryTableEffect { + pub table_key: String, + pub planned_transaction: StagedTransactionIdentity, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub confirmed_transaction: Option, +} + +/// The confirmed value for one table-version output slot in the intended +/// manifest delta. Metadata is copied from `SubTableUpdate` so recovery can +/// publish exactly what the original writer prepared rather than re-deriving a +/// wider delta from an arbitrary live HEAD. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub(crate) struct RecoveryConfirmedTableUpdate { + pub table_version: u64, + pub table_branch: Option, + pub row_count: u64, + pub version_metadata: super::TableVersionMetadata, +} + +/// Canonical manifest-delta output slot for one touched table. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub(crate) struct RecoveryTableUpdateSlot { + pub table_key: String, + pub expected_version: u64, + pub table_branch: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub confirmed: Option, +} + +/// Complete manifest delta carried by an RFC-022 mutation/load sidecar. The +/// table keys and expectations are immutable at arm time; physical output +/// slots are bound exactly once by Phase-B confirmation. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub(crate) struct RecoveryManifestDelta { + pub table_updates: Vec, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub registrations: Vec, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub tombstones: Vec, +} + +/// v3 payload. Kept optional on [`RecoverySidecar`] so v1/v2 continue to parse +/// with their original semantics; [`parse_sidecar`] requires it for v3. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub(crate) struct RecoveryProtocolV3 { + pub authority: RecoveryAuthorityToken, + pub lineage: RecoveryLineageIntent, + /// Stable id for the rollback lineage outcome. A recovery retry reuses it + /// instead of manufacturing an unbounded series of indistinguishable ids. + pub rollback_graph_commit_id: String, + /// Exact operator-facing outcomes for the fixed rollback commit. Recovery + /// persists this plan before the first restore or rollback publish and + /// reuses it verbatim after a crash. `None` means no rollback has been + /// prepared; `Some([])` is a prepared lineage-only rollback. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub rollback_audit_outcomes: Option>, + pub effect_phase: RecoveryEffectPhase, + pub effects: Vec, + pub intended_delta: RecoveryManifestDelta, +} + /// In-memory representation of the on-disk JSON sidecar. #[derive(Debug, Clone, Serialize, Deserialize)] pub(crate) struct RecoverySidecar { @@ -318,6 +496,9 @@ pub(crate) struct RecoverySidecar { /// non-SchemaApply writers. #[serde(default, skip_serializing_if = "Vec::is_empty")] pub tombstones: Vec, + /// RFC-022 exact-effect protocol. Absent for every v1/v2 sidecar. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub protocol_v3: Option, } /// Opaque handle returned by [`write_sidecar`] so the caller can delete @@ -384,6 +565,11 @@ pub(crate) enum TableClassification { /// previous restore attempt or an external mutation. Roll back to /// the manifest pin. UnexpectedMultistep, + /// The observed HEAD version may have the expected number, but the Lance + /// transaction identity does not equal the exact planned+confirmed effect. + /// A foreign writer or transparent rebase produced it; never roll it + /// forward as this sidecar's result. + UnexpectedEffectIdentity, /// A confirmation-using writer (`BranchMerge`) advanced this table's HEAD /// (`lance_head > manifest_pinned`) but the sidecar carries no /// `confirmed_version` — its multi-commit publish crashed mid-flight, so @@ -407,7 +593,8 @@ pub(crate) enum TableClassification { /// based on the worst classification: /// /// - Any `InvariantViolation` → `Abort` (operator action required). -/// - Any `UnexpectedAtP1` / `UnexpectedMultistep` / `NoMovement` → +/// - Any `UnexpectedAtP1` / `UnexpectedMultistep` / +/// `UnexpectedEffectIdentity` / `NoMovement` → /// `RollBack` all drifted tables to the manifest pin. /// - All `RolledPastExpected` → `RollForward` every table in one /// manifest publish. @@ -450,8 +637,9 @@ pub(crate) async fn write_sidecar( // Failpoint: models a storage put failure (S3 PutObject / fs write) // in Phase A — every writer must abort before any HEAD advance. crate::failpoints::maybe_fail(crate::failpoints::names::RECOVERY_SIDECAR_WRITE)?; - debug_assert_eq!(sidecar.schema_version, SIDECAR_SCHEMA_VERSION); + debug_assert!(sidecar.schema_version <= SIDECAR_SCHEMA_VERSION); let uri = sidecar_uri(root_uri, &sidecar.operation_id); + validate_sidecar_shape(&uri, sidecar)?; let json = serde_json::to_string_pretty(sidecar).map_err(|err| { OmniError::manifest_internal(format!("failed to serialize recovery sidecar: {}", err)) })?; @@ -564,6 +752,53 @@ pub(crate) async fn list_sidecars( Ok(out) } +/// Re-read one listed sidecar after its process-local recovery gates are held. +/// +/// `list_sidecars` necessarily runs before gate acquisition so the caller knows +/// which branch/table keys to take. The sidecar body is mutable, however: +/// BranchMerge stamps `confirmed_version`, and v3 writers move Armed -> +/// EffectsConfirmed (or persist a rollback audit plan) while holding those same +/// gates. Existence alone therefore is not a sufficient post-wait recheck. A +/// recovery pass must classify the body that is durable *after* it wins the +/// gates, not the stale body it used only for coordination discovery. +async fn reread_sidecar_under_gates( + root_uri: &str, + storage: &dyn StorageAdapter, + listed: &RecoverySidecar, +) -> Result> { + let uri = sidecar_uri(root_uri, &listed.operation_id); + if !storage.exists(&uri).await? { + return Ok(None); + } + let body = storage.read_text(&uri).await?; + let current = parse_sidecar(&uri, &body)?; + + // Confirmation/rollback-plan rewrites may change only recovery payload, not + // the keys whose gates protect the artifact. Refuse a coordination-shape + // rewrite rather than processing it under the wrong locks. + let listed_keys = listed + .tables + .iter() + .map(|pin| (&pin.table_key, &pin.table_path, &pin.table_branch)) + .collect::>(); + let current_keys = current + .tables + .iter() + .map(|pin| (&pin.table_key, &pin.table_path, &pin.table_branch)) + .collect::>(); + if current.operation_id != listed.operation_id + || current.writer_kind != listed.writer_kind + || current.branch != listed.branch + || current_keys != listed_keys + { + return Err(OmniError::manifest_internal(format!( + "recovery sidecar '{}' changed its coordination keys while recovery waited for gates", + listed.operation_id + ))); + } + Ok(Some(current)) +} + /// Parse a sidecar body, enforcing the schema-version refusal policy. /// Exposed separately so unit tests can exercise the parse path without /// going through storage. @@ -593,12 +828,198 @@ pub(crate) fn parse_sidecar(sidecar_uri: &str, body: &str) -> Result Result<()> { + let malformed = |reason: String| { + OmniError::manifest_internal(format!( + "recovery sidecar at '{}' has an invalid schema-v{} shape: {}", + sidecar_uri, sidecar.schema_version, reason + )) + }; + + if sidecar.schema_version < EXACT_EFFECT_IDENTITY_SCHEMA_VERSION { + if sidecar.protocol_v3.is_some() { + return Err(malformed( + "protocol_v3 is present on a pre-v3 sidecar".to_string(), + )); + } + return Ok(()); + } + + if !matches!( + sidecar.writer_kind, + SidecarKind::Mutation | SidecarKind::Load + ) { + return Err(malformed(format!( + "protocol_v3 exact effects are only defined for Mutation/Load, found {:?}", + sidecar.writer_kind + ))); + } + let protocol = sidecar + .protocol_v3 + .as_ref() + .ok_or_else(|| malformed("missing required protocol_v3 payload".to_string()))?; + if !sidecar.additional_registrations.is_empty() + || !sidecar.tombstones.is_empty() + || !protocol.intended_delta.registrations.is_empty() + || !protocol.intended_delta.tombstones.is_empty() + { + return Err(malformed( + "v3 Mutation/Load sidecars cannot register or tombstone tables".to_string(), + )); + } + + let sidecar_branch = sidecar.branch.as_deref().filter(|branch| *branch != "main"); + let lineage_branch = protocol + .lineage + .branch + .as_deref() + .filter(|branch| *branch != "main"); + if sidecar_branch != lineage_branch { + return Err(malformed( + "sidecar branch does not match original lineage branch".to_string(), + )); + } + if sidecar.actor_id != protocol.lineage.actor_id { + return Err(malformed( + "sidecar actor does not match original lineage actor".to_string(), + )); + } + if protocol.lineage.merged_parent_commit_id.is_some() { + return Err(malformed( + "v3 Mutation/Load lineage cannot carry a merged parent".to_string(), + )); + } + if protocol.rollback_graph_commit_id == protocol.lineage.graph_commit_id { + return Err(malformed( + "original and rollback graph commit ids must differ".to_string(), + )); + } + + let pin_keys: HashSet<&str> = sidecar + .tables + .iter() + .map(|pin| pin.table_key.as_str()) + .collect(); + if pin_keys.len() != sidecar.tables.len() { + return Err(malformed("duplicate table pin".to_string())); + } + if let Some(outcomes) = protocol.rollback_audit_outcomes.as_ref() { + let outcome_keys: HashSet<&str> = outcomes + .iter() + .map(|outcome| outcome.table_key.as_str()) + .collect(); + if outcome_keys.len() != outcomes.len() || !outcome_keys.is_subset(&pin_keys) { + return Err(malformed( + "rollback audit outcomes must name a unique subset of table pins".to_string(), + )); + } + } + let effect_keys: HashSet<&str> = protocol + .effects + .iter() + .map(|effect| effect.table_key.as_str()) + .collect(); + let effect_uuids: HashSet<&str> = protocol + .effects + .iter() + .map(|effect| effect.planned_transaction.uuid.as_str()) + .collect(); + let delta_keys: HashSet<&str> = protocol + .intended_delta + .table_updates + .iter() + .map(|slot| slot.table_key.as_str()) + .collect(); + if effect_keys.len() != protocol.effects.len() + || effect_uuids.len() != protocol.effects.len() + || delta_keys.len() != protocol.intended_delta.table_updates.len() + || effect_keys != pin_keys + || delta_keys != pin_keys + { + return Err(malformed( + "table pins, planned transaction identities, and intended-delta key sets are not one-to-one" + .to_string(), + )); + } + + for pin in &sidecar.tables { + let effect = protocol + .effects + .iter() + .find(|effect| effect.table_key == pin.table_key) + .expect("key sets checked above"); + let slot = protocol + .intended_delta + .table_updates + .iter() + .find(|slot| slot.table_key == pin.table_key) + .expect("key sets checked above"); + if effect.planned_transaction.read_version != pin.expected_version { + return Err(malformed(format!( + "planned transaction for '{}' reads version {}, pin expects {}", + pin.table_key, effect.planned_transaction.read_version, pin.expected_version + ))); + } + if slot.expected_version != pin.expected_version || slot.table_branch != pin.table_branch { + return Err(malformed(format!( + "intended-delta pre-state for '{}' does not match its table pin", + pin.table_key + ))); + } + match protocol.effect_phase { + RecoveryEffectPhase::Armed => { + if effect.confirmed_transaction.is_some() + || slot.confirmed.is_some() + || pin.confirmed_version.is_some() + { + return Err(malformed(format!( + "Armed effect '{}' already carries confirmation", + pin.table_key + ))); + } + } + RecoveryEffectPhase::EffectsConfirmed => { + let confirmed_transaction = + effect.confirmed_transaction.as_ref().ok_or_else(|| { + malformed(format!( + "EffectsConfirmed effect '{}' lacks a transaction identity", + pin.table_key + )) + })?; + let confirmed_update = slot.confirmed.as_ref().ok_or_else(|| { + malformed(format!( + "EffectsConfirmed effect '{}' lacks a manifest update", + pin.table_key + )) + })?; + if confirmed_transaction != &effect.planned_transaction { + return Err(malformed(format!( + "confirmed transaction for '{}' differs from its planned identity", + pin.table_key + ))); + } + if pin.confirmed_version != Some(confirmed_update.table_version) + || confirmed_update.table_branch != pin.table_branch + { + return Err(malformed(format!( + "confirmed update for '{}' does not match its table pin", + pin.table_key + ))); + } + } + } + } + Ok(()) } /// Classify one table's observed state vs. the sidecar's intent. @@ -630,12 +1051,33 @@ pub(crate) fn parse_sidecar(sidecar_uri: &str, body: &str) -> Result TableClassification { + classify_table_observation( + pin, + lance_head, + None, + manifest_pinned, + kind, + schema_version, + None, + ) +} + +fn classify_table_observation( + pin: &SidecarTablePin, + lance_head: u64, + observed_transaction: Option<&StagedTransactionIdentity>, + manifest_pinned: u64, + kind: SidecarKind, + schema_version: u32, + protocol_v3: Option<&RecoveryProtocolV3>, ) -> TableClassification { use TableClassification::*; if lance_head < manifest_pinned { @@ -690,6 +1132,49 @@ pub(crate) fn classify_table( UnexpectedMultistep } } + ClassificationMode::ExactEffect => { + let Some(protocol) = protocol_v3 else { + return IncompletePhaseB; + }; + if protocol.effect_phase != RecoveryEffectPhase::EffectsConfirmed { + return IncompletePhaseB; + } + let Some(effect) = protocol + .effects + .iter() + .find(|effect| effect.table_key == pin.table_key) + else { + return IncompletePhaseB; + }; + let Some(slot) = protocol + .intended_delta + .table_updates + .iter() + .find(|slot| slot.table_key == pin.table_key) + else { + return IncompletePhaseB; + }; + let (Some(confirmed_transaction), Some(confirmed_update)) = ( + effect.confirmed_transaction.as_ref(), + slot.confirmed.as_ref(), + ) else { + return IncompletePhaseB; + }; + if pin.expected_version != manifest_pinned { + return UnexpectedAtP1; + } + if lance_head != confirmed_update.table_version + || pin.confirmed_version != Some(confirmed_update.table_version) + { + return UnexpectedMultistep; + } + if confirmed_transaction != &effect.planned_transaction + || observed_transaction != Some(confirmed_transaction) + { + return UnexpectedEffectIdentity; + } + RolledPastExpected + } } } @@ -705,10 +1190,16 @@ pub(crate) fn decide(classifications: &[TableClassification]) -> SidecarDecision { return Abort; } - if classifications - .iter() - .any(|c| matches!(c, NoMovement | UnexpectedAtP1 | UnexpectedMultistep | IncompletePhaseB)) - { + if classifications.iter().any(|c| { + matches!( + c, + NoMovement + | UnexpectedAtP1 + | UnexpectedMultistep + | UnexpectedEffectIdentity + | IncompletePhaseB + ) + }) { return RollBack; } // All RolledPastExpected (or the slice is empty — no-op trivially). @@ -774,18 +1265,17 @@ pub(crate) async fn restore_table_to_version( /// violating sidecars are deferred to the next ReadWrite open, exactly /// as `Omnigraph::refresh` documents. /// -/// Concurrency: unlike the open-time sweep, this runs while other -/// writers may be in flight. Every sidecar writer (mutation/load -/// finalize, schema_apply, branch_merge, ensure_indices, optimize) -/// acquires its per-`(table_key, table_branch)` write queues *before* -/// `write_sidecar` and holds them until after `delete_sidecar` — so -/// acquiring the same queues here blocks until that writer either -/// finished (sidecar deleted; the existence re-check skips it) or died -/// (sidecar is genuinely orphaned; safe to process). Without this, the +/// Concurrency: unlike the open-time sweep, this runs while other writers may +/// be in flight. RFC-022 mutation/load holds root-scoped schema → branch → +/// sorted table gates across its sidecar lifetime; legacy adapters hold their +/// applicable schema/table gates. Healing takes the ordered superset, so it +/// blocks until the sidecar writer either +/// finished (sidecar deleted; the under-gate reread skips it) or died +/// (the freshly parsed sidecar is genuinely orphaned and safe to process). Without this, the /// heal could observe a live writer's sidecar in its commit→publish /// window, roll it forward, and fail that writer's own publish CAS. -/// Lock order is queues → coordinator, matching every writer's -/// commit→publish path. +/// Lock order is schema → branch → sorted tables → coordinator, matching the +/// RFC-022 writer and Full-recovery paths. /// /// The schema-staging reconcile runs lazily, per SchemaApply sidecar, /// AFTER that sidecar's queue guards are held and its existence is @@ -793,49 +1283,71 @@ pub(crate) async fn restore_table_to_version( /// LIVE schema apply's staging files and steal its commit (pinned by /// `tests/failpoints.rs::heal_does_not_promote_live_schema_apply_staging`). /// -/// Returns `true` when at least one sidecar was processed (the caller -/// should invalidate per-snapshot caches). +/// One sidecar that RollForwardOnly deliberately left in place after acquiring +/// its root-scoped gates and re-checking that the artifact still exists. +/// +/// Write-entry barriers use the captured graph branch to reject only relevant +/// intents. SchemaApply is graph-global and is filtered specially by the caller. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct UnresolvedRecoveryIntent { + pub(crate) operation_id: String, + /// Normalized graph branch (`None` means main; `Some("main")` is folded). + pub(crate) branch: Option, + pub(crate) writer_kind: SidecarKind, + /// Diagnostic ownership surface for future table-aware barriers. RFC-022's + /// first adapter fences mutation/load at graph-branch granularity. + pub(crate) table_keys: Vec, +} + +/// Returns whether durable state changed and every guarded intent that could +/// not be resolved without destructive recovery. Callers invalidate caches on +/// `processed_any`; mutation/load barriers additionally filter `unresolved` by +/// their captured graph branches. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct HealPendingOutcome { + pub(crate) processed_any: bool, + pub(crate) unresolved: Vec, +} + pub(crate) async fn heal_pending_sidecars_roll_forward( root_uri: &str, storage: std::sync::Arc, coordinator: &tokio::sync::RwLock, write_queue: &crate::db::write_queue::WriteQueueManager, -) -> Result { +) -> Result { let sidecars = list_sidecars(root_uri, storage.as_ref()).await?; if sidecars.is_empty() { - return Ok(false); + return Ok(HealPendingOutcome { + processed_any: false, + unresolved: Vec::new(), + }); } + crate::failpoints::maybe_fail(crate::failpoints::names::RECOVERY_POST_LIST_PRE_GATES)?; let mut processed_any = false; + let mut unresolved = Vec::new(); for sidecar in sidecars { - // Serialize against a possibly-live writer (see fn docs). Guards - // are scoped per sidecar so two sidecars never hold queues - // simultaneously (no cross-sidecar lock-order surface). - let mut queue_keys: Vec = sidecar + // Serialize against a possibly-live writer (see fn docs). Guards are + // scoped per sidecar and follow the one total order shared by writers, + // Full recovery, and live healing. Taking schema + branch even for an + // empty/legacy sidecar also serializes its audit/delete lifecycle. + let _schema_guard = write_queue.acquire(&schema_apply_serial_queue_key()).await; + let _branch_guard = write_queue.acquire_branch(sidecar.branch.as_deref()).await; + let queue_keys: Vec = sidecar .tables .iter() .map(|pin| (pin.table_key.clone(), pin.table_branch.clone())) .collect(); let is_schema_apply = matches!(sidecar.writer_kind, SidecarKind::SchemaApply); - if is_schema_apply { - // A SchemaApply sidecar's per-table pins don't cover a - // registration-only migration (no existing tables touched, - // but staging files + a sidecar on disk). The schema-apply - // writer holds this serialization key from before its - // sidecar write until after its sidecar delete, so blocking - // on it — then re-checking sidecar existence — guarantees - // the writer is gone before the reconcile below mutates - // schema staging. - queue_keys.push(schema_apply_serial_queue_key()); - } - let _guards = write_queue.acquire_many(&queue_keys).await; - // Re-check after the wait: the writer we blocked on may have - // completed Phase C and deleted its sidecar. - if !storage - .exists(&sidecar_uri(root_uri, &sidecar.operation_id)) - .await? - { + let _table_guards = write_queue.acquire_many(&queue_keys).await; + // Re-read after the wait: the writer we blocked on may have completed + // Phase C and deleted the sidecar, or may have durably confirmed Phase B + // before releasing its gates. Processing the pre-wait body would turn a + // fully confirmed effect set back into an apparent partial one. + let Some(sidecar) = + reread_sidecar_under_gates(root_uri, storage.as_ref(), &sidecar).await? + else { continue; - } + }; // Schema-staging reconcile, per SchemaApply sidecar, UNDER the // sidecar's guards: a sidecar still on disk after the queue wait // belongs to a dead writer, so promoting its staging files can no @@ -901,12 +1413,29 @@ pub(crate) async fn heal_pending_sidecars_roll_forward( .await? { processed_any = true; + } else { + let mut table_keys = sidecar + .tables + .iter() + .map(|pin| pin.table_key.clone()) + .collect::>(); + table_keys.sort(); + table_keys.dedup(); + unresolved.push(UnresolvedRecoveryIntent { + operation_id: sidecar.operation_id.clone(), + branch: sidecar.branch.clone().filter(|branch| branch != "main"), + writer_kind: sidecar.writer_kind, + table_keys, + }); } } // Re-read coordinator state so the caller's handle observes the // post-heal manifest. coordinator.write().await.refresh().await?; - Ok(processed_any) + Ok(HealPendingOutcome { + processed_any, + unresolved, + }) } /// Discard a sidecar whose branch no longer exists in the manifest (the @@ -917,8 +1446,8 @@ pub(crate) async fn heal_pending_sidecars_roll_forward( /// unreachable and the sidecar is provably moot; leaving it would wedge /// every heal (write entry) and every ReadWrite open on a dead-branch /// open, with `repair` refusing while it pends. Records an -/// `OrphanedBranchDiscarded` audit row (commit appended on main — the -/// sidecar's own branch has no commit graph anymore). +/// `OrphanedBranchDiscarded` audit row (lineage published on main — the +/// sidecar's own branch no longer has a live graph head). async fn discard_orphaned_branch_sidecar( root_uri: &str, storage: &dyn StorageAdapter, @@ -937,15 +1466,13 @@ async fn discard_orphaned_branch_sidecar( // the retry's sole remaining job is finishing the delete. (Cold // path: the list scan runs only when an orphaned sidecar exists.) // - // Documented residual: the commit append and the audit append are - // two writes. A failure BETWEEN them leaves a recovery commit with - // no audit row; the retry (keyed on the audit row, the operator- - // facing record) appends a second commit before the audit lands — - // bounded commit-graph noise, audit row still exactly-once. Same - // not-atomic-pair-write tolerance as `record_audit` and the - // manifest→commit-graph Known Gap; keying on commit rows instead - // would need an operation_id column on `_graph_commits`, and - // audit-before-commit would dangle the `graph_commit_id` join. + // Documented residual: the lineage CAS in `__manifest` and the audit append + // are two writes. A failure between them leaves a recovery commit with no + // audit row; a legacy sidecar carries no fixed recovery-commit id, so the + // retry (keyed on the operator-facing audit row) may append a second lineage + // commit before that one audit row lands. Keying idempotency on lineage + // would require a durable operation id there, while audit-before-lineage + // would leave a dangling `graph_commit_id` join. let already_recorded = audit.list().await?.iter().any(|record| { record.operation_id == sidecar.operation_id && record.recovery_kind == RecoveryKind::OrphanedBranchDiscarded @@ -963,10 +1490,14 @@ async fn discard_orphaned_branch_sidecar( created_at: crate::db::now_micros()?, }; let publisher = GraphNamespacePublisher::new(root_uri, None); - publisher.publish(&[], &HashMap::new(), Some(&intent)).await?; + publisher + .publish(&[], &HashMap::new(), Some(&intent)) + .await?; // Failpoint: the residual window above — commit published, audit // not yet durable. - crate::failpoints::maybe_fail(crate::failpoints::names::RECOVERY_ORPHAN_DISCARD_AUDIT_APPEND)?; + crate::failpoints::maybe_fail( + crate::failpoints::names::RECOVERY_ORPHAN_DISCARD_AUDIT_APPEND, + )?; audit .append(RecoveryAuditRecord { graph_commit_id: intent.graph_commit_id, @@ -1004,28 +1535,35 @@ pub(crate) fn schema_apply_serial_queue_key() -> crate::db::write_queue::TableQu /// roll-forward (every table eligible), roll-back (mixed or unexpected /// state), or abort (invariant violation). /// -/// Idempotency: a crash mid-sweep leaves the sidecar (deletion is the -/// final step). Re-opening re-classifies; repeated rollbacks of the -/// same table append extra Lance restore commits which `omnigraph -/// cleanup` reclaims. +/// Idempotency: a crash mid-sweep leaves the sidecar (deletion is the final +/// step). v3 persists an exact rollback audit plan before restoring and uses a +/// fixed rollback commit id, so re-entry resumes that outcome without another +/// restore or synthetic commit. Legacy sidecars re-classify and may append an +/// extra restore commit, which `omnigraph cleanup` reclaims. /// -/// Concurrency: the open-time sweep runs synchronously in `Omnigraph::open` -/// before the engine handle is published to any caller, so no request -/// handler can race it and it does NOT acquire write queues. In-process -/// callers (refresh, write entry points) must use -/// [`heal_pending_sidecars_roll_forward`] instead, which serializes -/// against live writers via per-(table_key, branch) queue acquisition. +/// Concurrency: a newly-opening handle is not yet published, but another handle +/// for the same root may already be serving writes. Every handle obtains the +/// root-scoped [`WriteQueueManager`](crate::db::write_queue::WriteQueueManager). +/// The caller holds its schema gate across schema-file recovery and this whole +/// pass; Full recovery adds branch → sorted table gates per sidecar and +/// re-reads/re-parses the sidecar after waiting. A live writer either finishes +/// and deletes its sidecar (recovery skips it), durably confirms a newer body +/// (recovery classifies that body), or releases the gates with a genuinely pending +/// intent (recovery owns it). This is process-local serialization only; +/// the documented foreign-process boundary remains. pub(crate) async fn recover_manifest_drift( root_uri: &str, storage: std::sync::Arc, coordinator: &mut GraphCoordinator, mode: RecoveryMode, schema_state_recovery: SchemaStateRecovery, + write_queue: &crate::db::write_queue::WriteQueueManager, ) -> Result<()> { let sidecars = list_sidecars(root_uri, storage.as_ref()).await?; if sidecars.is_empty() { return Ok(()); } + crate::failpoints::maybe_fail(crate::failpoints::names::RECOVERY_POST_LIST_PRE_GATES)?; // For each sidecar, classify against a FRESH snapshot AT THE // SIDECAR'S BRANCH. Two reasons: @@ -1039,6 +1577,28 @@ pub(crate) async fn recover_manifest_drift( // no-op'ing or rolling back the wrong table version. Open a // separate per-branch coordinator and use ITS snapshot. for sidecar in sidecars { + // Total order shared with RFC-022 writers. The caller already holds the + // root-scoped schema gate; add branch then sorted table gates so a + // concurrent open/refresh cannot Restore or delete a ref underneath a + // live writer from another `Omnigraph` handle. + let _branch_guard = write_queue.acquire_branch(sidecar.branch.as_deref()).await; + let table_keys = sidecar + .tables + .iter() + .map(|pin| (pin.table_key.clone(), pin.table_branch.clone())) + .collect::>(); + let _table_guards = write_queue.acquire_many(&table_keys).await; + + // The writer may have completed Phase C/D or rewritten Armed -> + // confirmed while this recovery instance waited. Re-read the complete + // body under the gates; an existence-only check still leaves recovery + // classifying stale confirmation state. + let Some(sidecar) = + reread_sidecar_under_gates(root_uri, storage.as_ref(), &sidecar).await? + else { + continue; + }; + let branch_snapshot = match sidecar.branch.as_deref() { Some(b) => { // Orphan check against the manifest's branch list (the @@ -1093,25 +1653,236 @@ async fn process_sidecar( // stale-sidecar audit recovery). `false` = the sidecar was deferred // untouched -- callers must not treat that as a completed heal (no // schema reload / cache invalidation is warranted). + if sidecar.protocol_v3.is_some() { + if let Some(outcome) = detect_visible_v3_outcome(root_uri, sidecar).await? { + return finalize_visible_v3_outcome(root_uri, storage.as_ref(), sidecar, outcome).await; + } + } let mut states = Vec::with_capacity(sidecar.tables.len()); for pin in &sidecar.tables { - let lance_head = open_lance_head(&pin.table_path, pin.table_branch.as_deref()).await?; let manifest_pinned = snapshot .entry(&pin.table_key) .map(|e| e.table_version) .unwrap_or(0); + // A first-touch named-branch mutation/load stages against the inherited + // source snapshot, arms its v3 sidecar, and only then creates the Lance + // target ref. Until Phase C publishes, the branch snapshot therefore + // still points at the source ref. Armed recovery must tolerate the + // target ref not existing yet: that is the crash-before-fork state, not + // storage corruption. Once effects are confirmed, a missing ref is + // impossible and remains a loud error. + let unpublished_fork = sidecar.protocol_v3.is_some() + && pin + .table_branch + .as_deref() + .is_some_and(|branch| branch != "main") + && sidecar.branch.as_deref() == pin.table_branch.as_deref() + && snapshot + .entry(&pin.table_key) + .map(|entry| entry.table_branch != pin.table_branch) + .unwrap_or(true); + let allow_missing_target_ref = unpublished_fork + && sidecar + .protocol_v3 + .as_ref() + .is_some_and(|protocol| protocol.effect_phase == RecoveryEffectPhase::Armed); + let planned_effect = sidecar.protocol_v3.as_ref().and_then(|protocol| { + protocol + .effects + .iter() + .find(|effect| effect.table_key == pin.table_key) + .map(|effect| { + ( + pin.post_commit_pin, + &effect.planned_transaction, + manifest_pinned, + ) + }) + }); + let observation = open_lance_head_if_present( + &pin.table_path, + pin.table_branch.as_deref(), + planned_effect, + allow_missing_target_ref, + ) + .await?; + let observation = observation.unwrap_or(LanceHeadObservation { + version: manifest_pinned, + transaction: None, + effect_ownership: EffectOwnership::None, + }); + let lance_head = observation.version; states.push(ClassifiedTable { - classification: classify_table( + classification: classify_table_observation( pin, lance_head, + observation.transaction.as_ref(), manifest_pinned, sidecar.writer_kind, sidecar.schema_version, + sidecar.protocol_v3.as_ref(), ), manifest_pinned, lance_head, + effect_ownership: observation.effect_ownership, + unpublished_fork, }); } + + if let Some(protocol) = sidecar.protocol_v3.as_ref() { + let live_authority = + read_live_recovery_authority(root_uri, storage, sidecar.branch.as_deref()).await?; + let branch_recreated = + live_authority.branch_identifier != protocol.authority.branch_identifier; + let authority_changed = live_authority != protocol.authority; + let any_unverifiable = states + .iter() + .any(|state| state.effect_ownership == EffectOwnership::Unverifiable); + let any_own_effect = states.iter().any(|state| { + matches!( + state.effect_ownership, + EffectOwnership::OwnAtHead + | EffectOwnership::OwnBeforeHead + | EffectOwnership::OwnCompensatedAtHead + ) + }); + let own_effect_not_at_head = states + .iter() + .any(|state| state.effect_ownership == EffectOwnership::OwnBeforeHead); + let ambiguous_manifest_advance = + sidecar + .tables + .iter() + .zip(states.iter()) + .any(|(pin, state)| { + // Lance's one-attempt commit can preflight-rebase our + // staged append over a winner that already committed and + // published: expected=v1, manifest winner=v2, our UUID + // still HEAD=v3. Restoring to CURRENT manifest pin v2 + // preserves the winner and removes only our stale effect. + // A manifest that selects our UUID (OwnBeforeHead or + // manifest_pinned == lance_head) remains unsafe. + // The same exception remains safe after a previous Full + // pass has already appended Restore(current manifest pin) + // but crashed before publishing that restore commit. + // `OwnCompensatedAtHead` is assigned only when the HEAD + // Restore target exactly equals this current pin. + let safe_published_lower_winner = matches!( + state.effect_ownership, + EffectOwnership::OwnAtHead | EffectOwnership::OwnCompensatedAtHead + ) && state.manifest_pinned + > pin.expected_version + && state.manifest_pinned < state.lance_head; + state.effect_ownership != EffectOwnership::None + && state.manifest_pinned != pin.expected_version + && !safe_published_lower_winner + }); + let foreign_table_drift = states.iter().any(|state| { + state.effect_ownership == EffectOwnership::None + && state.lance_head > state.manifest_pinned + }); + + if any_unverifiable { + let message = format!( + "OCC recovery sidecar '{}' cannot verify one or more planned Lance transaction \ + files; refusing destructive recovery", + sidecar.operation_id + ); + return match mode { + RecoveryMode::RollForwardOnly => { + warn!(operation_id = sidecar.operation_id.as_str(), "{message}"); + Ok(false) + } + RecoveryMode::Full => Err(OmniError::manifest_internal(message)), + }; + } + + if !any_own_effect { + // The sidecar was armed but this writer never landed a physical + // effect. RollForwardOnly recovery may be looking at a LIVE writer, + // so an Armed sidecar is still ownership and must be left untouched + // even though root-scoped queues make it wait for in-process + // handles. A quiesced Full sweep may abandon it; first remove only + // exact, unpublished first-touch forks that this intent owns. Do + // not manufacture lineage for an empty intent and never restore a + // foreign advance. + if matches!(mode, RecoveryMode::RollForwardOnly) + && (protocol.effect_phase == RecoveryEffectPhase::Armed + || states.iter().any(|state| state.unpublished_fork)) + { + warn!( + operation_id = sidecar.operation_id.as_str(), + "recovery: deferring v3 sidecar with no physical effects and live ownership" + ); + return Ok(false); + } + if matches!(mode, RecoveryMode::Full) { + cleanup_unpublished_no_effect_forks(root_uri, storage.as_ref(), sidecar, &states) + .await?; + } + warn!( + operation_id = sidecar.operation_id.as_str(), + authority_changed, "recovery: abandoning v3 sidecar with no owned physical effects" + ); + delete_sidecar_by_operation_id(root_uri, storage.as_ref(), &sidecar.operation_id) + .await?; + return Ok(true); + } + + if branch_recreated { + // A restore through the reused branch name would target the NEW + // incarnation and can destroy unrelated data. Leave the sidecar for + // operator review in every mode; the old incarnation's effects are + // not safe to compensate through this ref. + let message = format!( + "OCC recovery sidecar '{}' targets a branch incarnation that was deleted and \ + recreated; refusing to restore through the reused branch name", + sidecar.operation_id + ); + return match mode { + RecoveryMode::RollForwardOnly => { + warn!(operation_id = sidecar.operation_id.as_str(), "{message}"); + Ok(false) + } + RecoveryMode::Full => Err(OmniError::manifest_internal(message)), + }; + } + + if own_effect_not_at_head || ambiguous_manifest_advance || foreign_table_drift { + let message = format!( + "OCC recovery sidecar '{}' is interleaved with a foreign table/manifest advance; \ + refusing a restore that could discard another writer", + sidecar.operation_id + ); + return match mode { + RecoveryMode::RollForwardOnly => { + warn!(operation_id = sidecar.operation_id.as_str(), "{message}"); + Ok(false) + } + RecoveryMode::Full => Err(OmniError::manifest_internal(message)), + }; + } + + if authority_changed { + // At least one exact owned effect is still HEAD, but the branch + // authority used to prepare it changed. It must never roll forward. + // In-process healing cannot restore safely; a quiesced Full sweep + // compensates only the owned-at-HEAD tables. + if matches!(mode, RecoveryMode::RollForwardOnly) { + warn!( + operation_id = sidecar.operation_id.as_str(), + "recovery: deferring token-mismatched v3 sidecar with physical effects" + ); + return Ok(false); + } + warn!( + operation_id = sidecar.operation_id.as_str(), + "recovery: rolling back v3 sidecar because its captured authority changed" + ); + roll_back_sidecar(root_uri, storage.as_ref(), sidecar, &states).await?; + return Ok(true); + } + } let classifications = states .iter() .map(|state| state.classification) @@ -1167,6 +1938,19 @@ async fn process_sidecar( .zip(states.iter()) .any(|(pin, state)| state.manifest_pinned > pin.expected_version); if all_no_movement && any_pin_advanced { + if sidecar.protocol_v3.is_some() { + // Fixed outcome ids were checked before table + // classification. Alignment without either id is not proof + // of this sidecar's outcome. + return match mode { + RecoveryMode::RollForwardOnly => Ok(false), + RecoveryMode::Full => Err(OmniError::manifest_internal(format!( + "OCC recovery sidecar '{}' is manifest-aligned but neither fixed \ + outcome id is visible; operator review required", + sidecar.operation_id + ))), + }; + } if matches!(mode, RecoveryMode::RollForwardOnly) { // Refresh-time audit-recovery is safe: no // Dataset::restore involved; just an audit-row write @@ -1186,7 +1970,10 @@ async fn process_sidecar( ); } return record_audit_recovery_rollforward( - root_uri, storage.as_ref(), sidecar, &states, + root_uri, + storage.as_ref(), + sidecar, + &states, ) .await .map(|()| true); @@ -1256,6 +2043,23 @@ async fn process_sidecar( let (new_manifest_version, published_versions, graph_commit_id) = match roll_forward_all(root_uri, sidecar, &states, snapshot).await { Ok(published) => published, + // Another recovery instance may have published this exact + // fixed intent after our classification but before our CAS. + // Exact-head OCC correctly rejects our stale attempt; prove + // the original id + delta are now visible and converge + // without manufacturing another commit. + Err(err) if sidecar.protocol_v3.is_some() && err.is_read_set_changed() => { + if let Some(outcome) = detect_visible_v3_outcome(root_uri, sidecar).await? { + return finalize_visible_v3_outcome( + root_uri, + storage.as_ref(), + sidecar, + outcome, + ) + .await; + } + return Err(err); + } // Convergence-idempotent (invariants 7 & 15): a roll-forward's // postcondition is "the manifest reflects the sidecar's committed // Lance state", NOT "this sweep personally won the CAS". A @@ -1387,6 +2191,30 @@ async fn fresh_snapshot_for_sidecar( Ok(coordinator.snapshot()) } +/// Read the complete coarse authority token from current storage state. Called +/// only for a pending v3 sidecar after fixed outcome ids were proven absent. +async fn read_live_recovery_authority( + root_uri: &str, + storage: &std::sync::Arc, + branch: Option<&str>, +) -> Result { + let coordinator = match branch { + Some(branch) if branch != "main" => { + GraphCoordinator::open_branch(root_uri, branch, std::sync::Arc::clone(storage)).await? + } + _ => GraphCoordinator::open(root_uri, std::sync::Arc::clone(storage)).await?, + }; + let schema_state = + crate::db::schema_state::validate_schema_contract(root_uri, std::sync::Arc::clone(storage)) + .await?; + Ok(RecoveryAuthorityToken { + branch_identifier: coordinator.branch_identifier().await?, + graph_head: coordinator.exact_graph_head(), + schema_ir_hash: schema_state.schema_ir_hash, + schema_identity_version: schema_state.schema_identity_version, + }) +} + /// Convergence-idempotent handling of a roll-forward publish CAS that lost to a /// concurrent writer (`ExpectedVersionMismatch`). A roll-forward's postcondition /// is "the manifest reflects the sidecar's committed Lance state", not "this @@ -1424,7 +2252,7 @@ async fn converge_or_defer_roll_forward( // `expected_version` first healed THIS sidecar (recorded its RolledForward // audit and deleted it). So the audit row already exists; recording another // here would put two RolledForward rows in `_graph_commit_recoveries` for - // one recovery event (visible in `commit list --filter actor=…recovery`). + // one recovery event. // Only finish the bookkeeping if the sidecar is still on disk (the winner // crashed between audit and delete); if it is already gone, the winner // completed it — return success WITHOUT a duplicate audit, keeping the @@ -1471,11 +2299,12 @@ async fn converge_or_defer_roll_forward( .unwrap_or(0), }); } - // RFC-013 Phase 7: the winning writer folded its recovery commit into the - // manifest CAS, so the converge audit references THAT commit. We lost the CAS - // and never minted it, but a recovery commit is distinguishable by its - // `RECOVERY_ACTOR` authorship (`publish_recovery_commit`), so the latest - // recovery-actored commit on this branch IS it. Do NOT use the branch head: + // RFC-013 Phase 7: the winning legacy recovery folded its commit into the + // manifest CAS, so the converge audit references THAT commit. (v3 fixed-id + // outcomes are detected and finalized before this fallback.) We lost the + // CAS and never minted it, but a legacy recovery commit is distinguishable + // by `RECOVERY_ACTOR` authorship, so the latest recovery-actored commit on + // this branch is it. Do NOT use the branch head: // a concurrent USER write can advance `graph_head` past the recovery commit // between the winner's publish and this read, which would attribute the audit // row to the wrong (later, user) commit. (We only reach here with the sidecar @@ -1520,6 +2349,195 @@ struct ClassifiedTable { /// `_graph_commit_recoveries.lance` see actual drift, not /// `from_version == to_version == manifest_pinned`). lance_head: u64, + effect_ownership: EffectOwnership, + /// The branch snapshot still inherits this table from another Lance ref, + /// while the sidecar's final pin names a first-touch target ref. This is the + /// durable distinction between an unpublished fork and an already-owned + /// table on the branch. + unpublished_fork: bool, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum EffectOwnership { + None, + OwnAtHead, + OwnBeforeHead, + /// A previous Full recovery restored this owned effect to the manifest pin + /// but crashed before publishing the restored HEAD. The compensation is + /// complete and recovery should publish it, not restore again or wedge. + OwnCompensatedAtHead, + Unverifiable, +} + +/// Remove first-touch named-branch refs created by an Armed v3 attempt that +/// never landed this table's planned transaction. +/// +/// The sidecar is durable before the ref is created, so it is the ownership +/// record while the manifest still inherits the table from another branch. +/// Destruction is deliberately narrow: the manifest must not select the ref, +/// no other pending sidecar may claim the same `(table_path, branch)`, and the +/// live ref must still be exactly the fork point. Full recovery is quiesced, so +/// this fresh re-check closes ordinary crash/retry races. Lance does not expose +/// a compare-and-delete-by-branch-identifier primitive; callers must retain the +/// Full-sweep quiescence guarantee rather than reusing this helper in a live +/// reconciler. +async fn cleanup_unpublished_no_effect_forks( + root_uri: &str, + storage: &dyn StorageAdapter, + sidecar: &RecoverySidecar, + states: &[ClassifiedTable], +) -> Result<()> { + if sidecar.protocol_v3.is_none() { + return Ok(()); + } + + let all_sidecars = list_sidecars(root_uri, storage).await?; + for (pin, state) in sidecar.tables.iter().zip(states.iter()) { + if !state.unpublished_fork || state.effect_ownership != EffectOwnership::None { + continue; + } + let Some(target_branch) = pin + .table_branch + .as_deref() + .filter(|branch| *branch != "main") + else { + continue; + }; + + let has_competing_claim = all_sidecars.iter().any(|candidate| { + candidate.operation_id != sidecar.operation_id + && candidate.tables.iter().any(|candidate_pin| { + candidate_pin.table_path == pin.table_path + && candidate_pin.table_branch.as_deref() == Some(target_branch) + }) + }); + if has_competing_claim { + // Never delete a ref while another pending intent claims it. Full + // recovery is quiesced by the shared gates, so this no-effect + // sidecar can safely discard itself without touching the ref. A + // later/last claimant either cleans the still-untouched fork or + // recovers its owned effect. RollForwardOnly never calls this + // helper and continues to defer the unresolved ownership. + continue; + } + + let mut dataset = crate::instrumentation::open_dataset( + &pin.table_path, + crate::instrumentation::VersionResolution::Latest, + None, + crate::instrumentation::table_wrapper(), + ) + .await?; + let branches = dataset + .list_branches() + .await + .map_err(|error| OmniError::Lance(error.to_string()))?; + let Some(contents) = branches.get(target_branch) else { + // Crash-before-fork, or a previous cleanup pass that crashed before + // its rollback publish. Both are already converged physically. + continue; + }; + if contents.parent_version != pin.expected_version { + return Err(OmniError::manifest_internal(format!( + "OCC recovery sidecar '{}' cannot discard unpublished fork '{}:{}': \ + parent version is {}, expected exact fork point {}", + sidecar.operation_id, + pin.table_path, + target_branch, + contents.parent_version, + pin.expected_version + ))); + } + let target = dataset + .checkout_branch(target_branch) + .await + .map_err(|error| OmniError::Lance(error.to_string()))?; + if target.version().version != pin.expected_version { + return Err(OmniError::manifest_internal(format!( + "OCC recovery sidecar '{}' cannot discard unpublished fork '{}:{}': \ + live HEAD is {}, expected untouched version {}", + sidecar.operation_id, + pin.table_path, + target_branch, + target.version().version, + pin.expected_version + ))); + } + dataset + .force_delete_branch(target_branch) + .await + .map_err(|error| OmniError::Lance(error.to_string()))?; + } + Ok(()) +} + +fn table_requires_rollback_effect(state: &ClassifiedTable) -> bool { + matches!( + state.classification, + TableClassification::RolledPastExpected + | TableClassification::UnexpectedAtP1 + | TableClassification::UnexpectedMultistep + | TableClassification::UnexpectedEffectIdentity + | TableClassification::IncompletePhaseB + ) +} + +fn v3_rollback_audit_outcomes( + sidecar: &RecoverySidecar, + states: &[ClassifiedTable], +) -> Vec { + sidecar + .tables + .iter() + .zip(states.iter()) + .filter(|(_, state)| { + matches!( + state.effect_ownership, + EffectOwnership::OwnAtHead | EffectOwnership::OwnCompensatedAtHead + ) && table_requires_rollback_effect(state) + }) + .map(|(pin, state)| TableOutcome { + table_key: pin.table_key.clone(), + from_version: state.lance_head, + to_version: state.manifest_pinned, + }) + .collect() +} + +/// Durably bind the exact audit payload for a v3 rollback before the first +/// restore or rollback publish. A retry must replay the original observation, +/// not reconstruct it from post-restore HEADs. In particular, partial +/// multi-table attempts omit untouched pins, and a preflight-rebased v3 effect +/// records its actual `own_head -> current_manifest_pin` compensation. +async fn prepare_v3_rollback_audit_plan( + root_uri: &str, + storage: &dyn StorageAdapter, + sidecar: &RecoverySidecar, + states: &[ClassifiedTable], +) -> Result { + let mut prepared = sidecar.clone(); + let protocol = prepared + .protocol_v3 + .as_mut() + .expect("caller checked protocol_v3"); + if protocol.rollback_audit_outcomes.is_some() { + return Ok(prepared); + } + + protocol.rollback_audit_outcomes = Some(v3_rollback_audit_outcomes(sidecar, states)); + let uri = sidecar_uri(root_uri, &prepared.operation_id); + validate_sidecar_shape(&uri, &prepared)?; + let json = serde_json::to_string_pretty(&prepared).map_err(|error| { + OmniError::manifest_internal(format!( + "failed to serialize v3 rollback audit plan for sidecar '{}': {}", + prepared.operation_id, error + )) + })?; + // `write_text` replaces one complete object atomically (temp+rename for + // local storage, PutObject for object stores), the same durability contract + // as arming and Phase-B confirmation. + storage.write_text(&uri, &json).await?; + Ok(prepared) } async fn roll_back_sidecar( @@ -1528,6 +2546,24 @@ async fn roll_back_sidecar( sidecar: &RecoverySidecar, states: &[ClassifiedTable], ) -> Result<()> { + // Once the fixed rollback commit is visible, early recovery finalization no + // longer has pre-restore table observations. Persist the exact audit plan + // first so that path can replay it without fabricating outcomes from pins. + let prepared_v3 = if sidecar.protocol_v3.is_some() { + Some(prepare_v3_rollback_audit_plan(root_uri, storage, sidecar, states).await?) + } else { + None + }; + let sidecar = prepared_v3.as_ref().unwrap_or(sidecar); + + // An Armed multi-table attempt can create every first-touch ref and then + // land effects on only a subset. No-effect refs are not selected by the + // rollback manifest publish, so they must be removed BEFORE that publish. + // If recovery crashed after publishing first, the fixed rollback outcome + // would make the next pass finalize/delete the sidecar without ever seeing + // the still-orphaned refs. + cleanup_unpublished_no_effect_forks(root_uri, storage, sidecar, states).await?; + // Restore every drifted table (RolledPastExpected / UnexpectedAtP1 / // UnexpectedMultistep) to its manifest-pinned content, then PUBLISH so // `manifest == Lance HEAD` for each — symmetric with roll-forward. The @@ -1547,19 +2583,26 @@ async fn roll_back_sidecar( let mut updates: Vec = Vec::with_capacity(sidecar.tables.len()); let mut expected: HashMap = HashMap::with_capacity(sidecar.tables.len()); for (pin, state) in sidecar.tables.iter().zip(states.iter()) { - if matches!( - state.classification, - TableClassification::RolledPastExpected - | TableClassification::UnexpectedAtP1 - | TableClassification::UnexpectedMultistep - | TableClassification::IncompletePhaseB - ) { - restore_table_to_version( - &pin.table_path, - pin.table_branch.as_deref(), - state.manifest_pinned, + if sidecar.protocol_v3.is_some() + && !matches!( + state.effect_ownership, + EffectOwnership::OwnAtHead | EffectOwnership::OwnCompensatedAtHead ) - .await?; + { + // v3 compensation is ownership-exact. Never restore a table whose + // current HEAD is foreign (or whose owned effect is buried beneath + // another writer); callers reject the latter before reaching here. + continue; + } + if table_requires_rollback_effect(state) { + if state.effect_ownership != EffectOwnership::OwnCompensatedAtHead { + restore_table_to_version( + &pin.table_path, + pin.table_branch.as_deref(), + state.manifest_pinned, + ) + .await?; + } // Publish the post-restore HEAD (the restore commit we just made), // CAS against the current (unmoved) manifest pin — the same helper // roll-forward uses. `None` target: there is no prior observation to @@ -1589,9 +2632,22 @@ async fn roll_back_sidecar( // roll-back restores no table — `updates` is empty — but the recovery commit // lineage still publishes (a lineage-only merge), so the rollback is recorded // in the commit history just like a roll-forward. - let (_manifest_version, graph_commit_id) = - publish_recovery_commit(root_uri, sidecar, RecoveryKind::RolledBack, &updates, &expected) - .await?; + let (_manifest_version, graph_commit_id) = publish_recovery_commit( + root_uri, + sidecar, + RecoveryKind::RolledBack, + &updates, + &expected, + ) + .await?; + crate::failpoints::maybe_fail( + crate::failpoints::names::RECOVERY_POST_ROLLBACK_PUBLISH_PRE_AUDIT, + )?; + let outcomes = sidecar + .protocol_v3 + .as_ref() + .and_then(|protocol| protocol.rollback_audit_outcomes.clone()) + .unwrap_or(outcomes); record_audit( root_uri, sidecar, @@ -1658,6 +2714,180 @@ async fn record_audit_recovery_rollforward( Ok(()) } +/// Finalize a v3 sidecar whose tables are already manifest-aligned. +/// +/// Numeric table alignment alone is not evidence of WHICH outcome happened: +/// the original manifest CAS may have succeeded before sidecar deletion, or a +/// prior full recovery may have rolled the effects back and then failed during +/// audit/delete. v3 gives both outcomes fixed graph-commit ids, so inspect the +/// authoritative lineage and finish exactly that outcome without publishing a +/// second synthetic commit. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum VisibleV3Outcome { + Original, + RolledBack, +} + +async fn detect_visible_v3_outcome( + root_uri: &str, + sidecar: &RecoverySidecar, +) -> Result> { + let protocol = sidecar + .protocol_v3 + .as_ref() + .expect("caller checked protocol_v3"); + let (commits, _) = + ManifestCoordinator::read_graph_lineage_at(root_uri, sidecar.branch.as_deref()).await?; + let original_commit = commits + .iter() + .find(|commit| commit.graph_commit_id == protocol.lineage.graph_commit_id); + let rollback_visible = commits + .iter() + .any(|commit| commit.graph_commit_id == protocol.rollback_graph_commit_id); + let original_visible = if let Some(commit) = original_commit { + let expected_branch = protocol + .lineage + .branch + .as_deref() + .filter(|branch| *branch != "main"); + if commit.manifest_branch.as_deref() != expected_branch + || protocol + .authority + .graph_head + .as_ref() + .is_some_and(|head| commit.parent_commit_id.as_ref() != Some(head)) + || commit.merged_parent_commit_id != protocol.lineage.merged_parent_commit_id + || commit.actor_id != protocol.lineage.actor_id + || commit.created_at != protocol.lineage.created_at + { + return Err(OmniError::manifest_internal(format!( + "OCC recovery sidecar '{}' found original commit id '{}' with mismatched lineage", + sidecar.operation_id, protocol.lineage.graph_commit_id + ))); + } + let committed_snapshot = ManifestCoordinator::snapshot_at( + root_uri, + sidecar.branch.as_deref(), + commit.manifest_version, + ) + .await?; + let delta_matches = protocol.intended_delta.table_updates.iter().all(|slot| { + let Some(confirmed) = slot.confirmed.as_ref() else { + return false; + }; + committed_snapshot + .entry(&slot.table_key) + .is_some_and(|entry| { + entry.table_version == confirmed.table_version + && entry.table_branch == confirmed.table_branch + && entry.row_count == confirmed.row_count + && entry.version_metadata == confirmed.version_metadata + }) + }); + if !delta_matches { + return Err(OmniError::manifest_internal(format!( + "OCC recovery sidecar '{}' found original commit id '{}' but its manifest delta differs", + sidecar.operation_id, protocol.lineage.graph_commit_id + ))); + } + true + } else { + false + }; + + match (original_visible, rollback_visible) { + (true, false) => Ok(Some(VisibleV3Outcome::Original)), + (false, true) => Ok(Some(VisibleV3Outcome::RolledBack)), + (false, false) => Ok(None), + (true, true) => Err(OmniError::manifest_internal(format!( + "OCC recovery sidecar '{}' has both original commit '{}' and rollback commit '{}' \ + visible; refusing ambiguous finalization", + sidecar.operation_id, + protocol.lineage.graph_commit_id, + protocol.rollback_graph_commit_id + ))), + } +} + +async fn finalize_visible_v3_outcome( + root_uri: &str, + storage: &dyn StorageAdapter, + sidecar: &RecoverySidecar, + outcome: VisibleV3Outcome, +) -> Result { + let protocol = sidecar + .protocol_v3 + .as_ref() + .expect("caller checked protocol_v3"); + + let (kind, graph_commit_id, outcomes) = match outcome { + VisibleV3Outcome::Original => { + let outcomes = sidecar + .tables + .iter() + .map(|pin| { + let confirmed = protocol + .intended_delta + .table_updates + .iter() + .find(|slot| slot.table_key == pin.table_key) + .and_then(|slot| slot.confirmed.as_ref()) + .expect("validated EffectsConfirmed v3 sidecar"); + TableOutcome { + table_key: pin.table_key.clone(), + from_version: pin.expected_version, + to_version: confirmed.table_version, + } + }) + .collect(); + ( + RecoveryKind::RolledForward, + protocol.lineage.graph_commit_id.clone(), + outcomes, + ) + } + VisibleV3Outcome::RolledBack => { + let outcomes = protocol.rollback_audit_outcomes.clone().ok_or_else(|| { + OmniError::manifest_internal(format!( + "OCC recovery sidecar '{}' has visible fixed rollback commit '{}' but no \ + durable rollback audit outcomes; refusing to infer them from table pins", + sidecar.operation_id, protocol.rollback_graph_commit_id + )) + })?; + ( + RecoveryKind::RolledBack, + protocol.rollback_graph_commit_id.clone(), + outcomes, + ) + } + }; + + // Idempotent across an audit-success / sidecar-delete failure. The fixed + // outcome id is already durable in __manifest; the only remaining work is + // to ensure one audit row exists and delete the artifact. + let mut audit = RecoveryAudit::open(root_uri).await?; + let already_recorded = + audit.list().await?.iter().any(|record| { + record.operation_id == sidecar.operation_id && record.recovery_kind == kind + }); + if !already_recorded { + crate::failpoints::maybe_fail(crate::failpoints::names::RECOVERY_RECORD_AUDIT)?; + audit + .append(RecoveryAuditRecord { + graph_commit_id, + recovery_kind: kind, + recovery_for_actor: sidecar.actor_id.clone(), + operation_id: sidecar.operation_id.clone(), + sidecar_writer_kind: format!("{:?}", sidecar.writer_kind), + per_table_outcomes: outcomes, + created_at: crate::db::now_micros()?, + }) + .await?; + } + delete_sidecar_by_operation_id(root_uri, storage, &sidecar.operation_id).await?; + Ok(true) +} + /// Atomically extend every table's manifest pin from `expected_version` to /// `post_commit_pin` via a single `ManifestBatchPublisher::publish` call. /// Returns the new manifest version produced by the publish. @@ -1690,6 +2920,45 @@ async fn roll_forward_all( HashMap::with_capacity(sidecar.tables.len() + sidecar.additional_registrations.len()); for (pin, state) in sidecar.tables.iter().zip(states.iter()) { + if let Some(protocol) = sidecar.protocol_v3.as_ref() { + // Exact-effect recovery publishes the durably confirmed output + // slot, byte-for-byte, rather than deriving metadata from whatever + // HEAD happens to be live later. Classification already proved the + // observed HEAD transaction identity and version equal this slot. + let slot = protocol + .intended_delta + .table_updates + .iter() + .find(|slot| slot.table_key == pin.table_key) + .ok_or_else(|| { + OmniError::manifest_internal(format!( + "OCC recovery delta missing table '{}'", + pin.table_key + )) + })?; + let confirmed = slot.confirmed.as_ref().ok_or_else(|| { + OmniError::manifest_internal(format!( + "OCC recovery delta table '{}' is not confirmed", + pin.table_key + )) + })?; + if confirmed.table_version != state.lance_head { + return Err(OmniError::manifest_internal(format!( + "OCC recovery table '{}' observed HEAD {} but confirmed delta names {}", + pin.table_key, state.lance_head, confirmed.table_version + ))); + } + expected.insert(pin.table_key.clone(), slot.expected_version); + updates.push(ManifestChange::Update(SubTableUpdate { + table_key: pin.table_key.clone(), + table_version: confirmed.table_version, + table_branch: confirmed.table_branch.clone(), + row_count: confirmed.row_count, + version_metadata: confirmed.version_metadata.clone(), + })); + published_versions.insert(pin.table_key.clone(), confirmed.table_version); + continue; + } // Publish the version classification OBSERVED (`state.lance_head`), not a // fresh HEAD re-read. For a `Confirmed` pin classify already validated // `lance_head == confirmed_version`, so this publishes the recorded WAL @@ -1796,9 +3065,14 @@ async fn roll_forward_all( ); } - let (new_manifest_version, graph_commit_id) = - publish_recovery_commit(root_uri, sidecar, RecoveryKind::RolledForward, &updates, &expected) - .await?; + let (new_manifest_version, graph_commit_id) = publish_recovery_commit( + root_uri, + sidecar, + RecoveryKind::RolledForward, + &updates, + &expected, + ) + .await?; Ok((new_manifest_version, published_versions, graph_commit_id)) } @@ -1880,9 +3154,9 @@ async fn push_table_update( /// CAS as the table re-pin), so this only writes the `_graph_commit_recoveries` /// row, referencing that commit by `graph_commit_id`. A crash between the /// recovery publish and this audit append leaves a recovery commit with no audit -/// row — the same not-atomic-pair-write shape as before; the sweep tolerates it -/// (on re-entry the classifier surfaces `NoMovement`, the action is a no-op, and -/// the audit append is retried, minting a fresh recovery commit). +/// row. For v3, re-entry detects the fixed original/rollback commit and replays +/// its durable exact audit payload without minting another commit. Legacy +/// sidecars retain their stale-sidecar cleanup behavior. async fn record_audit( root_uri: &str, sidecar: &RecoverySidecar, @@ -1934,7 +3208,38 @@ pub(crate) async fn has_schema_apply_sidecar( /// branch ref (or default if `branch` is None or "main") and return its /// HEAD version. Recovery uses this so feature-branch sidecars classify /// against the feature-branch's Lance HEAD, not main's. -async fn open_lance_head(table_path: &str, branch: Option<&str>) -> Result { +struct LanceHeadObservation { + version: u64, + transaction: Option, + effect_ownership: EffectOwnership, +} + +#[cfg(test)] +async fn open_lance_head( + table_path: &str, + branch: Option<&str>, + planned_effect: Option<(u64, &StagedTransactionIdentity, u64)>, +) -> Result { + open_lance_head_if_present(table_path, branch, planned_effect, false) + .await? + .ok_or_else(|| { + OmniError::manifest_internal(format!( + "required Lance branch '{}' is missing for table '{}'", + branch.unwrap_or("main"), + table_path + )) + }) +} + +/// Variant used only for an Armed v3 first-touch fork. `allow_missing_branch` +/// turns an absent named target ref into `None`; every storage/list failure and +/// every missing ref outside that exact recovery state remains a loud error. +async fn open_lance_head_if_present( + table_path: &str, + branch: Option<&str>, + planned_effect: Option<(u64, &StagedTransactionIdentity, u64)>, + allow_missing_branch: bool, +) -> Result> { let ds = crate::instrumentation::open_dataset( table_path, crate::instrumentation::VersionResolution::Latest, @@ -1943,13 +3248,97 @@ async fn open_lance_head(table_path: &str, branch: Option<&str>) -> Result ) .await?; let ds = match branch { - Some(b) if b != "main" => ds - .checkout_branch(b) - .await - .map_err(|e| OmniError::Lance(e.to_string()))?, + Some(b) if b != "main" => { + if allow_missing_branch { + let branches = ds + .list_branches() + .await + .map_err(|error| OmniError::Lance(error.to_string()))?; + if !branches.contains_key(b) { + return Ok(None); + } + } + ds.checkout_branch(b) + .await + .map_err(|e| OmniError::Lance(e.to_string()))? + } _ => ds, }; - Ok(ds.version().version) + let head_transaction = if planned_effect.is_some() { + ds.read_transaction() + .await + .map_err(|error| OmniError::Lance(error.to_string()))? + } else { + None + }; + let transaction = head_transaction + .as_ref() + .map(StagedTransactionIdentity::from); + let head_restore_target = head_transaction + .as_ref() + .and_then(|transaction| match &transaction.operation { + lance::dataset::transaction::Operation::Restore { version } => Some(*version), + _ => None, + }); + let mut effect_ownership = EffectOwnership::None; + if let Some((first_possible_version, planned, compensation_target)) = planned_effect { + if ds.version().version >= first_possible_version { + if ds + .version() + .version + .saturating_sub(first_possible_version) + .saturating_add(1) + > MAX_EFFECT_IDENTITY_SCAN_VERSIONS + { + return Ok(Some(LanceHeadObservation { + version: ds.version().version, + transaction, + effect_ownership: EffectOwnership::Unverifiable, + })); + } + let mut unverifiable = false; + for version in first_possible_version..=ds.version().version { + let observed = if version == ds.version().version { + transaction.clone() + } else { + ds.read_transaction_by_version(version) + .await + .map_err(|error| OmniError::Lance(error.to_string()))? + .as_ref() + .map(StagedTransactionIdentity::from) + }; + match observed { + // UUID is the durable ownership marker. A transparent + // Lance rebase is still this writer's physical effect + // whether or not Lance preserves `read_version`; exact + // classification separately refuses to roll it forward. + Some(observed) if observed.uuid == planned.uuid => { + effect_ownership = if version == ds.version().version { + EffectOwnership::OwnAtHead + } else { + EffectOwnership::OwnBeforeHead + }; + break; + } + Some(_) => {} + None => unverifiable = true, + } + } + if effect_ownership == EffectOwnership::None && unverifiable { + effect_ownership = EffectOwnership::Unverifiable; + } + if effect_ownership == EffectOwnership::OwnBeforeHead + && head_restore_target == Some(compensation_target) + { + effect_ownership = EffectOwnership::OwnCompensatedAtHead; + } + } + } + Ok(Some(LanceHeadObservation { + version: ds.version().version, + transaction, + effect_ownership, + })) } async fn delete_sidecar_by_operation_id( @@ -1975,6 +3364,81 @@ pub(crate) fn new_sidecar( Err(_) => "0".to_string(), }; RecoverySidecar { + schema_version: LEGACY_SIDECAR_SCHEMA_VERSION, + operation_id, + started_at, + branch, + actor_id, + writer_kind, + tables, + merge_source_commit_id: None, + additional_registrations: Vec::new(), + tombstones: Vec::new(), + protocol_v3: None, + } +} + +/// Arm an RFC-022 mutation/load recovery sidecar. +/// +/// The returned v3 sidecar is still purely an intent (`Armed`): every staged +/// transaction identity and manifest output slot is durable, but no achieved +/// version is recorded. The caller must persist it with [`write_sidecar`] +/// before the first table HEAD advance and later call +/// [`confirm_occ_sidecar_phase_b`] after every exact effect completes. +pub(crate) fn new_occ_sidecar( + writer_kind: SidecarKind, + branch: Option, + actor_id: Option, + tables: Vec, + authority: RecoveryAuthorityToken, + lineage: RecoveryLineageIntent, + planned_transactions: HashMap, +) -> Result { + if !matches!(writer_kind, SidecarKind::Mutation | SidecarKind::Load) { + return Err(OmniError::manifest_internal(format!( + "new_occ_sidecar only supports Mutation/Load, found {:?}", + writer_kind + ))); + } + let pin_keys: HashSet<&str> = tables.iter().map(|pin| pin.table_key.as_str()).collect(); + if tables.is_empty() + || pin_keys.len() != tables.len() + || planned_transactions.len() != tables.len() + || !tables + .iter() + .all(|pin| planned_transactions.contains_key(&pin.table_key)) + { + return Err(OmniError::manifest_internal( + "new_occ_sidecar requires at least one pin and exactly one planned transaction per unique table pin", + )); + } + + let operation_id = ulid::Ulid::new().to_string(); + let started_at = match std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH) { + Ok(duration) => duration.as_micros().to_string(), + Err(_) => "0".to_string(), + }; + let effects = tables + .iter() + .map(|pin| RecoveryTableEffect { + table_key: pin.table_key.clone(), + planned_transaction: planned_transactions + .get(&pin.table_key) + .expect("planned key set checked above") + .clone(), + confirmed_transaction: None, + }) + .collect(); + let table_updates = tables + .iter() + .map(|pin| RecoveryTableUpdateSlot { + table_key: pin.table_key.clone(), + expected_version: pin.expected_version, + table_branch: pin.table_branch.clone(), + confirmed: None, + }) + .collect(); + let sidecar = RecoverySidecar { schema_version: SIDECAR_SCHEMA_VERSION, operation_id, started_at, @@ -1985,17 +3449,167 @@ pub(crate) fn new_sidecar( merge_source_commit_id: None, additional_registrations: Vec::new(), tombstones: Vec::new(), + protocol_v3: Some(RecoveryProtocolV3 { + authority, + lineage, + rollback_graph_commit_id: ulid::Ulid::new().to_string(), + rollback_audit_outcomes: None, + effect_phase: RecoveryEffectPhase::Armed, + effects, + intended_delta: RecoveryManifestDelta { + table_updates, + registrations: Vec::new(), + tombstones: Vec::new(), + }, + }), + }; + validate_sidecar_shape("", &sidecar)?; + Ok(sidecar) +} + +/// Bind every physical output slot of an RFC-022 sidecar and durably transition +/// it from `Armed` to `EffectsConfirmed`. +/// +/// Validation happens against a clone first. A missing table, a rebased Lance +/// transaction, or a version/branch mismatch leaves the on-disk sidecar Armed, +/// which recovery interprets as partial Phase B and rolls back. Only the exact +/// planned transaction set can become eligible for roll-forward. +pub(crate) async fn confirm_occ_sidecar_phase_b( + root_uri: &str, + storage: &dyn StorageAdapter, + sidecar: &mut RecoverySidecar, + updates: &[SubTableUpdate], + committed_transactions: &HashMap, +) -> Result<()> { + crate::failpoints::maybe_fail(crate::failpoints::names::RECOVERY_SIDECAR_CONFIRM)?; + validate_sidecar_shape(&sidecar_uri(root_uri, &sidecar.operation_id), sidecar)?; + + let protocol = sidecar.protocol_v3.as_ref().ok_or_else(|| { + OmniError::manifest_internal("confirm_occ_sidecar_phase_b requires a v3 OCC sidecar") + })?; + if protocol.effect_phase != RecoveryEffectPhase::Armed { + return Err(OmniError::manifest_internal(format!( + "OCC sidecar '{}' is already EffectsConfirmed", + sidecar.operation_id + ))); } + let update_keys: HashSet<&str> = updates + .iter() + .map(|update| update.table_key.as_str()) + .collect(); + let planned_keys: HashSet<&str> = protocol + .effects + .iter() + .map(|effect| effect.table_key.as_str()) + .collect(); + if update_keys.len() != updates.len() + || committed_transactions.len() != planned_keys.len() + || update_keys != planned_keys + || !planned_keys + .iter() + .all(|key| committed_transactions.contains_key(*key)) + { + return Err(OmniError::manifest_internal(format!( + "OCC sidecar '{}' confirmation key set differs from its planned effects", + sidecar.operation_id + ))); + } + + // Validate every effect before mutating the clone. Transaction identity is + // necessary but not sufficient: the version/branch check below also + // rejects Lance's preflight rebase, which can preserve both transaction + // fields while landing at a later version. + for effect in &protocol.effects { + let committed = committed_transactions + .get(&effect.table_key) + .expect("confirmed key set checked above"); + if committed != &effect.planned_transaction { + return Err(OmniError::manifest_internal(format!( + "OCC sidecar '{}' table '{}' committed transaction {:?}, planned {:?}; \ + leaving recovery Armed", + sidecar.operation_id, effect.table_key, committed, effect.planned_transaction + ))); + } + let pin = sidecar + .tables + .iter() + .find(|pin| pin.table_key == effect.table_key) + .expect("sidecar key sets validated"); + let update = updates + .iter() + .find(|update| update.table_key == effect.table_key) + .expect("confirmed key set checked above"); + if update.table_version != pin.post_commit_pin || update.table_branch != pin.table_branch { + return Err(OmniError::manifest_internal(format!( + "OCC sidecar '{}' table '{}' achieved version/branch ({}, {:?}), \ + planned ({}, {:?}); leaving recovery Armed", + sidecar.operation_id, + effect.table_key, + update.table_version, + update.table_branch, + pin.post_commit_pin, + pin.table_branch + ))); + } + } + + let mut confirmed = sidecar.clone(); + let confirmed_protocol = confirmed + .protocol_v3 + .as_mut() + .expect("validated v3 protocol"); + for effect in &mut confirmed_protocol.effects { + effect.confirmed_transaction = Some( + committed_transactions + .get(&effect.table_key) + .expect("confirmed key set checked above") + .clone(), + ); + } + for slot in &mut confirmed_protocol.intended_delta.table_updates { + let update = updates + .iter() + .find(|update| update.table_key == slot.table_key) + .expect("confirmed key set checked above"); + slot.confirmed = Some(RecoveryConfirmedTableUpdate { + table_version: update.table_version, + table_branch: update.table_branch.clone(), + row_count: update.row_count, + version_metadata: update.version_metadata.clone(), + }); + } + for pin in &mut confirmed.tables { + let update = updates + .iter() + .find(|update| update.table_key == pin.table_key) + .expect("confirmed key set checked above"); + pin.confirmed_version = Some(update.table_version); + } + confirmed_protocol.effect_phase = RecoveryEffectPhase::EffectsConfirmed; + + let uri = sidecar_uri(root_uri, &confirmed.operation_id); + validate_sidecar_shape(&uri, &confirmed)?; + let json = serde_json::to_string_pretty(&confirmed).map_err(|error| { + OmniError::manifest_internal(format!( + "failed to serialize confirmed OCC recovery sidecar: {}", + error + )) + })?; + storage.write_text(&uri, &json).await?; + *sidecar = confirmed; + Ok(()) } #[cfg(test)] mod tests { use super::*; use crate::storage::ObjectStorageAdapter; - use lance::Dataset; use crate::table_store::TableStore; - use arrow_array::{Int32Array, RecordBatch, StringArray}; + use arrow_array::{Array, Int32Array, RecordBatch, StringArray}; use arrow_schema::{DataType, Field, Schema}; + #[cfg(feature = "failpoints")] + use futures::TryStreamExt; + use lance::Dataset; use std::sync::Arc; fn person_schema() -> Arc { @@ -2029,6 +3643,38 @@ mod tests { } } + fn transaction(read_version: u64, uuid: &str) -> StagedTransactionIdentity { + StagedTransactionIdentity { + read_version, + uuid: uuid.to_string(), + } + } + + fn occ_sidecar() -> RecoverySidecar { + let planned = transaction(5, "planned-transaction"); + new_occ_sidecar( + SidecarKind::Mutation, + None, + Some("act-alice".to_string()), + vec![make_pin("node:Person", "file:///tmp/people.lance", 5, 6)], + RecoveryAuthorityToken { + branch_identifier: lance::dataset::refs::BranchIdentifier::main(), + graph_head: Some("01H000000000000000000000AA".to_string()), + schema_ir_hash: "schema-hash".to_string(), + schema_identity_version: 1, + }, + RecoveryLineageIntent { + graph_commit_id: "01H000000000000000000000BB".to_string(), + branch: None, + actor_id: Some("act-alice".to_string()), + merged_parent_commit_id: None, + created_at: 123, + }, + HashMap::from([("node:Person".to_string(), planned)]), + ) + .unwrap() + } + #[test] fn sidecar_round_trips_through_json() { let original = new_sidecar( @@ -2039,7 +3685,7 @@ mod tests { ); let json = serde_json::to_string(&original).unwrap(); let parsed = parse_sidecar("file:///tmp/__recovery/x.json", &json).unwrap(); - assert_eq!(parsed.schema_version, SIDECAR_SCHEMA_VERSION); + assert_eq!(parsed.schema_version, LEGACY_SIDECAR_SCHEMA_VERSION); assert_eq!(parsed.operation_id, original.operation_id); assert_eq!(parsed.writer_kind, SidecarKind::Mutation); assert_eq!(parsed.branch.as_deref(), Some("main")); @@ -2048,6 +3694,191 @@ mod tests { assert_eq!(parsed.tables[0].table_key, "node:Person"); } + #[test] + fn occ_sidecar_round_trips_armed_with_stable_ids_and_exact_effect() { + let original = occ_sidecar(); + assert_eq!(original.schema_version, SIDECAR_SCHEMA_VERSION); + let protocol = original.protocol_v3.as_ref().unwrap(); + assert_eq!(protocol.effect_phase, RecoveryEffectPhase::Armed); + assert_eq!(protocol.effects.len(), 1); + assert_eq!( + protocol.effects[0].planned_transaction, + transaction(5, "planned-transaction") + ); + assert!(!protocol.rollback_graph_commit_id.is_empty()); + assert_eq!( + protocol.lineage.graph_commit_id, + "01H000000000000000000000BB" + ); + + let json = serde_json::to_string(&original).unwrap(); + let parsed = parse_sidecar("memory://graph/__recovery/op.json", &json).unwrap(); + assert_eq!( + parsed.protocol_v3.as_ref().unwrap().authority, + protocol.authority + ); + + let pin = &parsed.tables[0]; + assert_eq!( + classify_table_observation( + pin, + 6, + Some(&transaction(5, "planned-transaction")), + 5, + SidecarKind::Mutation, + SIDECAR_SCHEMA_VERSION, + parsed.protocol_v3.as_ref(), + ), + TableClassification::IncompletePhaseB, + "an Armed sidecar must roll back even when its planned transaction is at HEAD" + ); + } + + #[test] + fn parse_sidecar_refuses_v3_without_protocol_payload() { + let mut value = serde_json::to_value(occ_sidecar()).unwrap(); + value.as_object_mut().unwrap().remove("protocol_v3"); + let error = parse_sidecar( + "memory://graph/__recovery/op.json", + &serde_json::to_string(&value).unwrap(), + ) + .unwrap_err(); + assert!(error.to_string().contains("missing required protocol_v3")); + } + + #[test] + fn rollback_audit_plan_is_an_explicit_unique_subset_of_pins() { + let mut empty = occ_sidecar(); + empty.protocol_v3.as_mut().unwrap().rollback_audit_outcomes = Some(Vec::new()); + let json = serde_json::to_string(&empty).unwrap(); + assert!( + json.contains("rollback_audit_outcomes"), + "Some(empty) must remain distinguishable from an unprepared None plan" + ); + parse_sidecar("memory://graph/__recovery/empty.json", &json).unwrap(); + + let mut duplicate = occ_sidecar(); + duplicate + .protocol_v3 + .as_mut() + .unwrap() + .rollback_audit_outcomes = Some(vec![ + TableOutcome { + table_key: "node:Person".to_string(), + from_version: 6, + to_version: 5, + }, + TableOutcome { + table_key: "node:Person".to_string(), + from_version: 7, + to_version: 5, + }, + ]); + let duplicate_error = parse_sidecar( + "memory://graph/__recovery/duplicate.json", + &serde_json::to_string(&duplicate).unwrap(), + ) + .unwrap_err(); + assert!(duplicate_error.to_string().contains("unique subset")); + + let mut foreign = occ_sidecar(); + foreign + .protocol_v3 + .as_mut() + .unwrap() + .rollback_audit_outcomes = Some(vec![TableOutcome { + table_key: "node:Foreign".to_string(), + from_version: 2, + to_version: 1, + }]); + let foreign_error = parse_sidecar( + "memory://graph/__recovery/foreign.json", + &serde_json::to_string(&foreign).unwrap(), + ) + .unwrap_err(); + assert!(foreign_error.to_string().contains("unique subset")); + } + + #[tokio::test] + async fn occ_confirmation_binds_exact_delta_and_rejects_foreign_identity() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_str().unwrap(); + let storage = ObjectStorageAdapter::local(); + let mut sidecar = occ_sidecar(); + write_sidecar(root, &storage, &sidecar).await.unwrap(); + + let version_metadata: crate::db::manifest::TableVersionMetadata = + serde_json::from_value(serde_json::json!({ + "manifest_path": "nodes/person/_versions/6.manifest", + "manifest_size": 42, + "e_tag": "etag-6", + "naming_scheme": "V2" + })) + .unwrap(); + let updates = vec![SubTableUpdate { + table_key: "node:Person".to_string(), + table_version: 6, + table_branch: None, + row_count: 7, + version_metadata, + }]; + let committed = HashMap::from([( + "node:Person".to_string(), + transaction(5, "planned-transaction"), + )]); + let rebased = HashMap::from([( + "node:Person".to_string(), + transaction(6, "planned-transaction"), + )]); + let error = confirm_occ_sidecar_phase_b(root, &storage, &mut sidecar, &updates, &rebased) + .await + .unwrap_err(); + assert!(error.to_string().contains("leaving recovery Armed")); + assert_eq!( + sidecar.protocol_v3.as_ref().unwrap().effect_phase, + RecoveryEffectPhase::Armed + ); + confirm_occ_sidecar_phase_b(root, &storage, &mut sidecar, &updates, &committed) + .await + .unwrap(); + + let protocol = sidecar.protocol_v3.as_ref().unwrap(); + assert_eq!(protocol.effect_phase, RecoveryEffectPhase::EffectsConfirmed); + assert_eq!( + protocol.intended_delta.table_updates[0] + .confirmed + .as_ref() + .unwrap() + .row_count, + 7 + ); + let pin = &sidecar.tables[0]; + assert_eq!( + classify_table_observation( + pin, + 6, + Some(&transaction(5, "planned-transaction")), + 5, + SidecarKind::Mutation, + SIDECAR_SCHEMA_VERSION, + Some(protocol), + ), + TableClassification::RolledPastExpected + ); + assert_eq!( + classify_table_observation( + pin, + 6, + Some(&transaction(5, "foreign-transaction")), + 5, + SidecarKind::Mutation, + SIDECAR_SCHEMA_VERSION, + Some(protocol), + ), + TableClassification::UnexpectedEffectIdentity + ); + } + #[test] fn parse_sidecar_refuses_future_but_accepts_older_schema_version() { let body = |version: u32| { @@ -2081,7 +3912,13 @@ mod tests { fn classify_no_movement_when_head_equals_pinned() { let pin = make_pin("node:Person", "irrelevant", 5, 6); assert_eq!( - classify_table(&pin, 5, 5, SidecarKind::Mutation, SIDECAR_SCHEMA_VERSION), + classify_table( + &pin, + 5, + 5, + SidecarKind::Mutation, + LEGACY_SIDECAR_SCHEMA_VERSION, + ), TableClassification::NoMovement, ); } @@ -2090,7 +3927,13 @@ mod tests { fn classify_rolled_past_expected_when_sidecar_matches_strict() { let pin = make_pin("node:Person", "irrelevant", 5, 6); assert_eq!( - classify_table(&pin, 6, 5, SidecarKind::Mutation, SIDECAR_SCHEMA_VERSION), + classify_table( + &pin, + 6, + 5, + SidecarKind::Mutation, + LEGACY_SIDECAR_SCHEMA_VERSION, + ), TableClassification::RolledPastExpected, ); } @@ -2100,7 +3943,13 @@ mod tests { // Same +1 drift but post_commit_pin says it should be 7, not 6. let pin = make_pin("node:Person", "irrelevant", 5, 7); assert_eq!( - classify_table(&pin, 6, 5, SidecarKind::Mutation, SIDECAR_SCHEMA_VERSION), + classify_table( + &pin, + 6, + 5, + SidecarKind::Mutation, + LEGACY_SIDECAR_SCHEMA_VERSION, + ), TableClassification::UnexpectedAtP1, ); } @@ -2109,7 +3958,13 @@ mod tests { fn classify_unexpected_multistep_when_head_jumped_more_than_one_strict() { let pin = make_pin("node:Person", "irrelevant", 5, 6); assert_eq!( - classify_table(&pin, 8, 5, SidecarKind::Mutation, SIDECAR_SCHEMA_VERSION), + classify_table( + &pin, + 8, + 5, + SidecarKind::Mutation, + LEGACY_SIDECAR_SCHEMA_VERSION, + ), TableClassification::UnexpectedMultistep, ); } @@ -2118,7 +3973,13 @@ mod tests { fn classify_invariant_violation_when_head_below_pinned() { let pin = make_pin("node:Person", "irrelevant", 5, 6); assert_eq!( - classify_table(&pin, 3, 5, SidecarKind::Mutation, SIDECAR_SCHEMA_VERSION), + classify_table( + &pin, + 3, + 5, + SidecarKind::Mutation, + LEGACY_SIDECAR_SCHEMA_VERSION, + ), TableClassification::InvariantViolation { observed: 3 }, ); } @@ -2143,7 +4004,13 @@ mod tests { fn classify_loose_match_accepts_multi_commit_drift_for_ensure_indices() { let pin = make_pin("node:Person", "irrelevant", 5, 6); assert_eq!( - classify_table(&pin, 9, 5, SidecarKind::EnsureIndices, SIDECAR_SCHEMA_VERSION), + classify_table( + &pin, + 9, + 5, + SidecarKind::EnsureIndices, + SIDECAR_SCHEMA_VERSION + ), TableClassification::RolledPastExpected, ); } @@ -2183,7 +4050,13 @@ mod tests { // partial Phase B → roll back. let unconfirmed = make_pin("node:Person", "irrelevant", 5, 6); assert_eq!( - classify_table(&unconfirmed, 8, 5, SidecarKind::BranchMerge, SIDECAR_SCHEMA_VERSION), + classify_table( + &unconfirmed, + 8, + 5, + SidecarKind::BranchMerge, + SIDECAR_SCHEMA_VERSION + ), TableClassification::IncompletePhaseB, ); // Backward-compat: the SAME unconfirmed pin in a PRE-confirmation (v1) @@ -2205,12 +4078,24 @@ mod tests { ..make_pin("node:Person", "irrelevant", 5, 6) }; assert_eq!( - classify_table(&confirmed, 8, 5, SidecarKind::BranchMerge, SIDECAR_SCHEMA_VERSION), + classify_table( + &confirmed, + 8, + 5, + SidecarKind::BranchMerge, + SIDECAR_SCHEMA_VERSION + ), TableClassification::RolledPastExpected, ); // Confirmed, but HEAD drifted past it (foreign writer) → roll back. assert_eq!( - classify_table(&confirmed, 9, 5, SidecarKind::BranchMerge, SIDECAR_SCHEMA_VERSION), + classify_table( + &confirmed, + 9, + 5, + SidecarKind::BranchMerge, + SIDECAR_SCHEMA_VERSION + ), TableClassification::UnexpectedMultistep, ); } @@ -2352,6 +4237,542 @@ mod tests { ); } + #[tokio::test] + async fn exact_effect_observation_recognizes_interrupted_rollback_restore() { + let dir = tempfile::tempdir().unwrap(); + let uri = format!("{}/people.lance", dir.path().to_str().unwrap()); + let store = TableStore::new( + dir.path().to_str().unwrap(), + Arc::new(lance::session::Session::default()), + ); + let ds = TableStore::write_dataset(&uri, person_batch(&[("alice", Some(30))])) + .await + .unwrap(); + let staged = store + .stage_append(&ds, person_batch(&[("bob", Some(25))]), &[]) + .await + .unwrap(); + let planned = staged.transaction_identity(); + let (committed, observed) = store + .commit_staged_exact(Arc::new(ds), staged) + .await + .unwrap(); + assert_eq!(observed, planned); + assert_eq!(committed.version().version, 2); + + // Model Full recovery crashing after Dataset::restore but before the + // restored HEAD is published to __manifest. + restore_table_to_version(&uri, None, 1).await.unwrap(); + let observation = open_lance_head(&uri, None, Some((2, &planned, 1))) + .await + .unwrap(); + assert_eq!(observation.version, 3); + assert_eq!( + observation.effect_ownership, + EffectOwnership::OwnCompensatedAtHead + ); + + // A later/foreign Restore is not proof that recovery compensated to + // the manifest pin. Exact target identity matters: with manifest still + // pinned to v1, a HEAD Restore(v2) must remain interleaved rather than + // being promoted to resumable compensation. + restore_table_to_version(&uri, None, 2).await.unwrap(); + let foreign_restore = open_lance_head(&uri, None, Some((2, &planned, 1))) + .await + .unwrap(); + assert_eq!(foreign_restore.version, 4); + assert_eq!( + foreign_restore.effect_ownership, + EffectOwnership::OwnBeforeHead + ); + } + + #[tokio::test] + async fn exact_preflight_rebase_recovery_preserves_published_winner() { + // Lance may preflight-rebase an Append even when CommitBuilder's retry + // budget is zero. Reproduce the exact availability hazard: + // + // v1 prepared attempt's expected version + // v2 a competing writer commits and publishes + // v3 the stale prepared transaction lands at HEAD with its own UUID + // + // Full recovery must compensate v3 back to the CURRENT manifest pin + // (v2), not reject the sidecar merely because v2 != its original v1 + // expectation and not restore all the way to v1 (which loses winner). + const SCHEMA: &str = r#" +node Person { + age: I32? +} +"#; + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_str().unwrap(); + let db = crate::db::Omnigraph::init(root, SCHEMA).await.unwrap(); + + let txn = db.open_write_txn(None).await.unwrap(); + let lineage = db.new_lineage_intent_for_branch(None, None).await.unwrap(); + let entry = txn.base.entry("node:Person").unwrap().clone(); + let table_uri = format!("{}/{}", root.trim_end_matches('/'), entry.table_path); + let stale_dataset = txn.base.open("node:Person").await.unwrap(); + let store = TableStore::new(root, Arc::new(lance::session::Session::default())); + let staged = store + .stage_append( + &stale_dataset, + person_batch(&[("stale-attempt", Some(99))]), + &[], + ) + .await + .unwrap(); + let planned = staged.transaction_identity(); + assert_eq!(planned.read_version, entry.table_version); + + db.load( + "main", + r#"{"type":"Person","data":{"id":"winner","age":22}}"#, + crate::loader::LoadMode::Append, + ) + .await + .unwrap(); + let winner_snapshot = db.snapshot_of("main").await.unwrap(); + let winner_version = winner_snapshot.entry("node:Person").unwrap().table_version; + assert_eq!(winner_version, planned.read_version + 1); + + let authority = RecoveryAuthorityToken { + branch_identifier: txn.authority.branch_identifier.clone(), + graph_head: txn.authority.graph_head.clone(), + schema_ir_hash: txn.authority.schema_ir_hash.clone(), + schema_identity_version: txn.authority.schema_identity_version, + }; + let recovery_lineage = RecoveryLineageIntent { + graph_commit_id: lineage.graph_commit_id, + branch: lineage.branch, + actor_id: lineage.actor_id, + merged_parent_commit_id: lineage.merged_parent_commit_id, + created_at: lineage.created_at, + }; + let sidecar = new_occ_sidecar( + SidecarKind::Mutation, + None, + None, + vec![SidecarTablePin { + table_key: "node:Person".to_string(), + table_path: table_uri.clone(), + expected_version: planned.read_version, + post_commit_pin: planned.read_version + 1, + confirmed_version: None, + table_branch: entry.table_branch, + }], + authority, + recovery_lineage, + HashMap::from([("node:Person".to_string(), planned.clone())]), + ) + .unwrap(); + write_sidecar(root, db.storage_adapter(), &sidecar) + .await + .unwrap(); + + let (rebased, observed) = store + .commit_staged_exact(Arc::new(stale_dataset), staged) + .await + .unwrap(); + assert_eq!(observed, planned); + assert_eq!( + rebased.version().version, + winner_version + 1, + "the stale transaction must reproduce Lance's expected+2 preflight rebase" + ); + drop(rebased); + + // Model a first Full recovery pass that correctly compensates the + // stale v3 effect to the CURRENT manifest pin v2, then crashes before + // publishing the Restore commit. HEAD is now v4 / Restore(v2), while + // __manifest still selects the winner at v2 and the sidecar remains. + // Reopen must recognize that exact compensation and publish v4 instead + // of treating the owned v3 effect as buried beneath a foreign write. + restore_table_to_version(&table_uri, None, winner_version) + .await + .unwrap(); + let compensated = open_lance_head( + &table_uri, + None, + Some((planned.read_version + 1, &planned, winner_version)), + ) + .await + .unwrap(); + assert_eq!(compensated.version, winner_version + 2); + assert_eq!( + compensated.effect_ownership, + EffectOwnership::OwnCompensatedAtHead + ); + + drop(txn); + drop(db); + + let recovered = crate::db::Omnigraph::open(root).await.unwrap(); + let snapshot = recovered.snapshot_of("main").await.unwrap(); + let visible = snapshot.open("node:Person").await.unwrap(); + let batches: Vec = + futures::TryStreamExt::try_collect(visible.scan().try_into_stream().await.unwrap()) + .await + .unwrap(); + let ids: Vec = batches + .iter() + .flat_map(|batch| { + let ids = batch + .column_by_name("id") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + (0..ids.len()) + .map(|row| ids.value(row).to_string()) + .collect::>() + }) + .collect(); + assert_eq!( + ids, + vec!["winner"], + "recovery must preserve the published winner while removing only the stale effect" + ); + + let recovered_entry = snapshot.entry("node:Person").unwrap(); + let lance_head = Dataset::open(&table_uri).await.unwrap().version().version; + assert_eq!( + recovered_entry.table_version, lance_head, + "recovery must publish its compensation so manifest and Lance HEAD converge" + ); + assert!( + list_sidecars(root, recovered.storage_adapter()) + .await + .unwrap() + .is_empty(), + "the converged sidecar must be removed" + ); + } + + #[cfg(feature = "failpoints")] + #[tokio::test] + #[serial_test::serial] + async fn fixed_rollback_replays_partial_multi_table_audit_plan_after_crash() { + const SCHEMA: &str = r#" +node Person { age: I32? } +node Company { age: I32? } +"#; + let _scenario = fail::FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_str().unwrap(); + let db = crate::db::Omnigraph::init(root, SCHEMA).await.unwrap(); + let txn = db.open_write_txn(None).await.unwrap(); + let lineage = db.new_lineage_intent_for_branch(None, None).await.unwrap(); + let store = TableStore::new(root, Arc::new(lance::session::Session::default())); + + let person_entry = txn.base.entry("node:Person").unwrap().clone(); + let company_entry = txn.base.entry("node:Company").unwrap().clone(); + let person_uri = format!("{}/{}", root.trim_end_matches('/'), person_entry.table_path); + let company_uri = format!( + "{}/{}", + root.trim_end_matches('/'), + company_entry.table_path + ); + let person_ds = txn.base.open("node:Person").await.unwrap(); + let company_ds = txn.base.open("node:Company").await.unwrap(); + let person_staged = store + .stage_append(&person_ds, person_batch(&[("partial", Some(10))]), &[]) + .await + .unwrap(); + let company_staged = store + .stage_append(&company_ds, person_batch(&[("untouched", Some(20))]), &[]) + .await + .unwrap(); + let person_planned = person_staged.transaction_identity(); + let company_planned = company_staged.transaction_identity(); + + let authority = RecoveryAuthorityToken { + branch_identifier: txn.authority.branch_identifier.clone(), + graph_head: txn.authority.graph_head.clone(), + schema_ir_hash: txn.authority.schema_ir_hash.clone(), + schema_identity_version: txn.authority.schema_identity_version, + }; + let recovery_lineage = RecoveryLineageIntent { + graph_commit_id: lineage.graph_commit_id, + branch: lineage.branch, + actor_id: lineage.actor_id, + merged_parent_commit_id: lineage.merged_parent_commit_id, + created_at: lineage.created_at, + }; + let sidecar = new_occ_sidecar( + SidecarKind::Mutation, + None, + None, + vec![ + SidecarTablePin { + table_key: "node:Person".to_string(), + table_path: person_uri, + expected_version: person_entry.table_version, + post_commit_pin: person_entry.table_version + 1, + confirmed_version: None, + table_branch: person_entry.table_branch, + }, + SidecarTablePin { + table_key: "node:Company".to_string(), + table_path: company_uri, + expected_version: company_entry.table_version, + post_commit_pin: company_entry.table_version + 1, + confirmed_version: None, + table_branch: company_entry.table_branch, + }, + ], + authority, + recovery_lineage, + HashMap::from([ + ("node:Person".to_string(), person_planned.clone()), + ("node:Company".to_string(), company_planned), + ]), + ) + .unwrap(); + let operation_id = sidecar.operation_id.clone(); + write_sidecar(root, db.storage_adapter(), &sidecar) + .await + .unwrap(); + + let (person_committed, observed) = store + .commit_staged_exact(Arc::new(person_ds), person_staged) + .await + .unwrap(); + assert_eq!(observed, person_planned); + let person_effect_version = person_committed.version().version; + assert_eq!(person_effect_version, person_entry.table_version + 1); + drop(person_committed); + drop(company_staged); + drop(txn); + drop(db); + + let failpoint = crate::failpoints::ScopedFailPoint::new( + crate::failpoints::names::RECOVERY_POST_ROLLBACK_PUBLISH_PRE_AUDIT, + "return", + ); + let error = crate::db::Omnigraph::open(root) + .await + .err() + .expect("post-rollback failpoint must fail the first open"); + assert!( + error + .to_string() + .contains("recovery.post_rollback_publish_pre_audit") + ); + drop(failpoint); + + let persisted = list_sidecars(root, &ObjectStorageAdapter::local()) + .await + .unwrap(); + assert_eq!(persisted.len(), 1); + let planned_outcomes = persisted[0] + .protocol_v3 + .as_ref() + .unwrap() + .rollback_audit_outcomes + .as_ref() + .expect("rollback audit plan must be durable before the publish"); + assert_eq!( + planned_outcomes, + &[TableOutcome { + table_key: "node:Person".to_string(), + from_version: person_effect_version, + to_version: person_entry.table_version, + }], + "the untouched Company pin must not be fabricated into the audit plan" + ); + + let recovered = crate::db::Omnigraph::open(root).await.unwrap(); + let audit = RecoveryAudit::open(root).await.unwrap(); + let records = audit.list().await.unwrap(); + let record = records + .iter() + .find(|record| record.operation_id == operation_id) + .expect("fixed rollback retry must append its audit row"); + assert_eq!(record.recovery_kind, RecoveryKind::RolledBack); + assert_eq!(record.per_table_outcomes, planned_outcomes.clone()); + assert_eq!( + recovered + .snapshot_of("main") + .await + .unwrap() + .open("node:Person") + .await + .unwrap() + .count_rows(None) + .await + .unwrap(), + 0 + ); + assert!( + list_sidecars(root, recovered.storage_adapter()) + .await + .unwrap() + .is_empty() + ); + } + + #[cfg(feature = "failpoints")] + #[tokio::test] + #[serial_test::serial] + async fn fixed_rollback_replays_preflight_rebase_audit_plan_after_crash() { + const SCHEMA: &str = r#" +node Person { age: I32? } +"#; + let _scenario = fail::FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_str().unwrap(); + let db = crate::db::Omnigraph::init(root, SCHEMA).await.unwrap(); + + let txn = db.open_write_txn(None).await.unwrap(); + let lineage = db.new_lineage_intent_for_branch(None, None).await.unwrap(); + let entry = txn.base.entry("node:Person").unwrap().clone(); + let table_uri = format!("{}/{}", root.trim_end_matches('/'), entry.table_path); + let stale_dataset = txn.base.open("node:Person").await.unwrap(); + let store = TableStore::new(root, Arc::new(lance::session::Session::default())); + let staged = store + .stage_append( + &stale_dataset, + person_batch(&[("stale-attempt", Some(99))]), + &[], + ) + .await + .unwrap(); + let planned = staged.transaction_identity(); + + db.load( + "main", + r#"{"type":"Person","data":{"id":"winner","age":22}}"#, + crate::loader::LoadMode::Append, + ) + .await + .unwrap(); + let winner_version = db + .snapshot_of("main") + .await + .unwrap() + .entry("node:Person") + .unwrap() + .table_version; + assert_eq!(winner_version, planned.read_version + 1); + + let sidecar = new_occ_sidecar( + SidecarKind::Mutation, + None, + None, + vec![SidecarTablePin { + table_key: "node:Person".to_string(), + table_path: table_uri.clone(), + expected_version: planned.read_version, + post_commit_pin: planned.read_version + 1, + confirmed_version: None, + table_branch: entry.table_branch, + }], + RecoveryAuthorityToken { + branch_identifier: txn.authority.branch_identifier.clone(), + graph_head: txn.authority.graph_head.clone(), + schema_ir_hash: txn.authority.schema_ir_hash.clone(), + schema_identity_version: txn.authority.schema_identity_version, + }, + RecoveryLineageIntent { + graph_commit_id: lineage.graph_commit_id, + branch: lineage.branch, + actor_id: lineage.actor_id, + merged_parent_commit_id: lineage.merged_parent_commit_id, + created_at: lineage.created_at, + }, + HashMap::from([("node:Person".to_string(), planned.clone())]), + ) + .unwrap(); + let operation_id = sidecar.operation_id.clone(); + write_sidecar(root, db.storage_adapter(), &sidecar) + .await + .unwrap(); + + let (rebased, observed) = store + .commit_staged_exact(Arc::new(stale_dataset), staged) + .await + .unwrap(); + assert_eq!(observed, planned); + let own_head = rebased.version().version; + assert_eq!(own_head, winner_version + 1); + drop(rebased); + drop(txn); + drop(db); + + let failpoint = crate::failpoints::ScopedFailPoint::new( + crate::failpoints::names::RECOVERY_POST_ROLLBACK_PUBLISH_PRE_AUDIT, + "return", + ); + let error = crate::db::Omnigraph::open(root) + .await + .err() + .expect("post-rollback failpoint must fail the first open"); + assert!( + error + .to_string() + .contains("recovery.post_rollback_publish_pre_audit") + ); + drop(failpoint); + + let persisted = list_sidecars(root, &ObjectStorageAdapter::local()) + .await + .unwrap(); + assert_eq!(persisted.len(), 1); + let planned_outcomes = persisted[0] + .protocol_v3 + .as_ref() + .unwrap() + .rollback_audit_outcomes + .clone() + .expect("preflight compensation must persist its exact audit plan"); + assert_eq!( + planned_outcomes, + vec![TableOutcome { + table_key: "node:Person".to_string(), + from_version: own_head, + to_version: winner_version, + }], + "audit must report the actual v3 -> v2 compensation, never infer v2 -> v1 from pins" + ); + + let recovered = crate::db::Omnigraph::open(root).await.unwrap(); + let snapshot = recovered.snapshot_of("main").await.unwrap(); + let visible = snapshot.open("node:Person").await.unwrap(); + let batches: Vec = visible + .scan() + .try_into_stream() + .await + .unwrap() + .try_collect() + .await + .unwrap(); + let ids = batches + .iter() + .flat_map(|batch| { + let ids = batch + .column_by_name("id") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + (0..ids.len()) + .map(|row| ids.value(row).to_string()) + .collect::>() + }) + .collect::>(); + assert_eq!(ids, vec!["winner"]); + + let audit = RecoveryAudit::open(root).await.unwrap(); + let records = audit.list().await.unwrap(); + let record = records + .iter() + .find(|record| record.operation_id == operation_id) + .expect("fixed rollback retry must append its audit row"); + assert_eq!(record.recovery_kind, RecoveryKind::RolledBack); + assert_eq!(record.per_table_outcomes, planned_outcomes); + } + #[tokio::test] async fn list_sidecars_returns_empty_when_dir_missing() { let dir = tempfile::tempdir().unwrap(); @@ -2415,7 +4836,8 @@ mod tests { .await .expect_err("a pinned table with no achieved version must be a loud error"); assert!( - err.to_string().contains("pins and publish updates diverged"), + err.to_string() + .contains("pins and publish updates diverged"), "expected a pin/updates divergence error, got: {err}", ); } diff --git a/crates/omnigraph/src/db/manifest/state.rs b/crates/omnigraph/src/db/manifest/state.rs index b546d2e2..2fd66599 100644 --- a/crates/omnigraph/src/db/manifest/state.rs +++ b/crates/omnigraph/src/db/manifest/state.rs @@ -29,6 +29,11 @@ pub struct SubTableEntry { pub(super) struct ManifestState { pub(super) version: u64, pub(super) entries: Vec, + /// Exact materialized `graph_head:` values from this SAME manifest + /// version. Keeping the head beside the table snapshot prevents a + /// manifest-only refresh from leaving coarse write authority split between + /// a fresh table view and the commit graph's older derived cache. + pub(super) graph_heads: HashMap, } #[derive(Debug, Clone)] @@ -91,10 +96,13 @@ struct ManifestScan { /// the caller asked (`collect_lineage`). Empty on the table-state read hot /// path so it never pays the O(commits) lineage JSON decode; populated on the /// publish path, where `load_publish_state` already needs the parent and would - /// otherwise scan `__manifest` a second time via `read_graph_lineage`. `graph_head` - /// rows are not collected here — parent resolution uses the head-over-commits - /// computation, not the denormalized head pointer (see `resolve_lineage_rows`). + /// otherwise scan `__manifest` a second time via `read_graph_lineage`. lineage_rows: Vec, + /// Exact materialized `graph_head:` values, collected on every + /// table-state/publish pass. This is bounded by branch count; unlike + /// `lineage_rows`, it does not grow with commit history. OCC must distinguish + /// a present head from an absent one (notably on a fresh named branch). + graph_heads: HashMap, } pub(super) fn manifest_schema() -> SchemaRef { @@ -137,6 +145,7 @@ pub(super) async fn read_manifest_state(dataset: &Dataset) -> Result, tombstones: impl IntoIterator, + graph_heads: HashMap, ) -> ManifestState { let mut latest_versions = HashMap::::new(); for entry in version_entries { @@ -183,7 +193,11 @@ pub(super) fn assemble_manifest_state( }) .collect(); entries.sort_by(|a, b| a.table_key.cmp(&b.table_key)); - ManifestState { version, entries } + ManifestState { + version, + entries, + graph_heads, + } } // After RFC-013 P2 folded the publish path off this accessor (it now projects @@ -208,6 +222,9 @@ pub(super) struct PublishScan { pub(super) version_entries: Vec, pub(super) tombstones: Vec<((String, u64), ())>, pub(super) lineage_rows: Vec, + /// Exact `graph_head:` rows keyed by the branch suffix (`main` for + /// main). Absence is meaningful and is preserved by a missing map entry. + pub(super) graph_heads: HashMap, } /// One-scan read of everything the publish path needs. `collect_lineage` is @@ -224,6 +241,7 @@ pub(super) async fn read_publish_scan(dataset: &Dataset) -> Result .map(|tombstone| ((tombstone.table_key, tombstone.tombstone_version), ())) .collect(), lineage_rows: scan.lineage_rows, + graph_heads: scan.graph_heads, }) } @@ -264,14 +282,43 @@ fn decode_graph_commit_row( }) } +/// Decode one `graph_head` row into its exact branch-key / commit-id pair. +/// Shared by the dedicated lineage reader and the publisher's folded one-scan +/// path so presence, absence, and malformed-row handling cannot drift. +fn decode_graph_head_row( + object_ids: &StringArray, + metadata: &StringArray, + row: usize, +) -> Result<(String, String)> { + if metadata.is_null(row) { + return Err(OmniError::manifest_internal(format!( + "manifest graph_head row missing metadata for {}", + object_ids.value(row) + ))); + } + let head_meta: GraphHeadMetadata = serde_json::from_str(metadata.value(row)).map_err(|e| { + OmniError::manifest_internal(format!("failed to decode graph_head metadata: {e}")) + })?; + let branch_key = object_ids + .value(row) + .strip_prefix("graph_head:") + .ok_or_else(|| { + OmniError::manifest_internal(format!( + "invalid graph_head object id {}", + object_ids.value(row) + )) + })? + .to_string(); + Ok((branch_key, head_meta.head_commit_id)) +} + async fn read_manifest_scan(dataset: &Dataset, collect_lineage: bool) -> Result { // Project only the columns the assembly below reads (RFC-013 PR2 #1c). The - // table-state hot path never touches `object_id` (lineage decode only) or - // `base_objects` (reserved/unused — never read on any path), so reading them - // is wasted bytes on every `__manifest` scan — write publish AND every - // branch-op open. Mirrors Lance's own directory-catalog `__manifest` reads, - // which project to the needed columns rather than scanning all of them. - let mut projection: Vec<&str> = vec![ + // `object_id` is needed for the bounded graph-head authority decode on every + // path; `base_objects` remains reserved/unused. Mirrors Lance's own + // directory-catalog `__manifest` reads, which project only needed columns. + let projection: Vec<&str> = vec![ + "object_id", "object_type", "location", "metadata", @@ -280,9 +327,6 @@ async fn read_manifest_scan(dataset: &Dataset, collect_lineage: bool) -> Result< "table_branch", "row_count", ]; - if collect_lineage { - projection.push("object_id"); - } let mut scanner = dataset.scan(); scanner .project(&projection) @@ -299,6 +343,7 @@ async fn read_manifest_scan(dataset: &Dataset, collect_lineage: bool) -> Result< let mut version_entries = Vec::new(); let mut tombstones = Vec::new(); let mut lineage_rows = Vec::new(); + let mut graph_heads = HashMap::new(); for batch in &batches { let object_types = string_column(batch, "object_type")?; @@ -308,13 +353,11 @@ async fn read_manifest_scan(dataset: &Dataset, collect_lineage: bool) -> Result< let versions = u64_column(batch, "table_version")?; let branches = string_column(batch, "table_branch")?; let row_counts = u64_column(batch, "row_count")?; - // `object_id` is only needed for lineage decoding; skip the lookup - // entirely on the table-state hot path (`collect_lineage == false`). - let object_ids = if collect_lineage { - Some(string_column(batch, "object_id")?) - } else { - None - }; + // `object_id` is needed for the exact graph-head authority even on the + // table-state path. We still skip every `graph_commit` decode there, so + // the added work is bounded by the number of branch-head rows rather + // than growing with commit history. + let object_ids = string_column(batch, "object_id")?; for row in 0..batch.num_rows() { let table_key = table_keys.value(row).to_string(); @@ -358,21 +401,22 @@ async fn read_manifest_scan(dataset: &Dataset, collect_lineage: bool) -> Result< tombstone_version, }); } - // `graph_commit` rows (RFC-013) are decoded into the scan ONLY - // when `collect_lineage` is set (the publish path, which resolves - // a parent). The table-state hot path leaves them — and - // `graph_head` + any future object type — in the `_` arm so it - // never pays the O(commits) lineage JSON decode. When NOT - // collecting, `object_ids` is `None`, so this arm is the same - // forward-compat skip as the `_` arm. + // `graph_commit` rows (RFC-013) are decoded ONLY for the publish + // path, which resolves a parent. The table-state hot path skips + // them, while still decoding the bounded graph-head authority + // rows below. OBJECT_TYPE_GRAPH_COMMIT if collect_lineage => { - let object_ids = object_ids.expect("object_ids read when collect_lineage"); lineage_rows.push(decode_graph_commit_row( object_ids, metadata, versions, branches, row, )?); } - // Skipped on the table-state path (and for `graph_head` / unknown - // future object types on every path): no table snapshot needs them. + OBJECT_TYPE_GRAPH_HEAD => { + let (branch_key, head_commit_id) = + decode_graph_head_row(object_ids, metadata, row)?; + graph_heads.insert(branch_key, head_commit_id); + } + // Commit rows are skipped on the table-state path; unknown future + // object types are skipped on every path. _ => {} } } @@ -404,6 +448,7 @@ async fn read_manifest_scan(dataset: &Dataset, collect_lineage: bool) -> Result< version_entries: entries, tombstones, lineage_rows, + graph_heads, }) } @@ -447,26 +492,9 @@ pub(crate) async fn read_graph_lineage( )?); } OBJECT_TYPE_GRAPH_HEAD => { - if metadata.is_null(row) { - return Err(OmniError::manifest_internal(format!( - "manifest graph_head row missing metadata for {}", - object_ids.value(row) - ))); - } - let head_meta: GraphHeadMetadata = serde_json::from_str(metadata.value(row)) - .map_err(|e| { - OmniError::manifest_internal(format!( - "failed to decode graph_head metadata: {e}" - )) - })?; - // `object_id` is `graph_head:`; the branch key after - // the prefix is the projection's map key (`main` for main). - let branch_key = object_ids - .value(row) - .strip_prefix("graph_head:") - .unwrap_or_default() - .to_string(); - graph_heads.insert(branch_key, head_meta.head_commit_id); + let (branch_key, head_commit_id) = + decode_graph_head_row(object_ids, metadata, row)?; + graph_heads.insert(branch_key, head_commit_id); } _ => {} } @@ -623,7 +651,6 @@ pub(super) fn entries_to_batch( ) } - pub(super) fn manifest_rows_batch( object_ids: Vec, object_types: Vec, diff --git a/crates/omnigraph/src/db/manifest/tests.rs b/crates/omnigraph/src/db/manifest/tests.rs index 1d6e4b33..fc3e1a74 100644 --- a/crates/omnigraph/src/db/manifest/tests.rs +++ b/crates/omnigraph/src/db/manifest/tests.rs @@ -12,7 +12,11 @@ use lance_namespace::models::{ use lance_namespace_impls::DirectoryNamespaceBuilder; use tokio::sync::Mutex; -use super::publisher::{LineageIntent, ManifestBatchPublisher, PublishOutcome}; +use super::publisher::{ + GraphHeadExpectation, LineageIntent, ManifestBatchPublisher, PublishOutcome, + PublishPrecondition, +}; +use super::state::read_publish_scan; use super::*; use omnigraph_compiler::catalog::build_catalog; use omnigraph_compiler::schema::parser::parse_schema; @@ -154,10 +158,11 @@ async fn test_commit_changes_can_register_new_table_and_tombstone_old_one() { let ds = crate::table_store::TableStore::create_empty_dataset(&dataset_uri, &schema) .await .unwrap(); - let state = crate::table_store::TableStore::new(uri, Arc::new(lance::session::Session::default())) - .table_state(&dataset_uri, &ds) - .await - .unwrap(); + let state = + crate::table_store::TableStore::new(uri, Arc::new(lance::session::Session::default())) + .table_state(&dataset_uri, &ds) + .await + .unwrap(); mc.commit_changes(&[ ManifestChange::RegisterTable(TableRegistration { @@ -611,7 +616,10 @@ async fn test_branch_manifest_namespace_uses_entry_owner_branch_for_latest_table .with_branch("feature", None) .load() .await; - let err = format!("{:?}", mismatched.expect_err("branch-mismatched open must fail on v9")); + let err = format!( + "{:?}", + mismatched.expect_err("branch-mismatched open must fail on v9") + ); assert!( err.contains("belonging to branch"), "expected the v9 branch-consistency error, got: {err}" @@ -1148,11 +1156,12 @@ impl RecordingPublisher { #[async_trait] impl ManifestBatchPublisher for RecordingPublisher { - async fn publish( + async fn publish_with_precondition( &self, changes: &[ManifestChange], expected_table_versions: &HashMap, lineage: Option<&LineageIntent>, + precondition: &PublishPrecondition, ) -> Result { let requests: Vec = changes .iter() @@ -1163,7 +1172,7 @@ impl ManifestBatchPublisher for RecordingPublisher { .collect(); self.requests.lock().await.extend_from_slice(&requests); self.inner - .publish(changes, expected_table_versions, lineage) + .publish_with_precondition(changes, expected_table_versions, lineage, precondition) .await } } @@ -1172,11 +1181,12 @@ struct FailingPublisher; #[async_trait] impl ManifestBatchPublisher for FailingPublisher { - async fn publish( + async fn publish_with_precondition( &self, _changes: &[ManifestChange], _expected_table_versions: &HashMap, _lineage: Option<&LineageIntent>, + _precondition: &PublishPrecondition, ) -> Result { Err(OmniError::manifest( "injected batch publisher failure".to_string(), @@ -1760,7 +1770,9 @@ async fn sub_current_graph_is_refused_on_open_with_rebuild_hint() { // (v4) cannot open, since `MIN_SUPPORTED == CURRENT == 4`. { let mut ds = open_manifest_dataset(uri, None).await.unwrap(); - super::migrations::set_stamp_for_test(&mut ds, 3).await.unwrap(); + super::migrations::set_stamp_for_test(&mut ds, 3) + .await + .unwrap(); } // Read-write open is refused with the rebuild hint. @@ -1820,7 +1832,9 @@ async fn sub_current_graph_is_refused_then_rebuilt_via_export_import() { // Make it look like a graph from an older release: rewind the stamp below CURRENT. { let mut ds = open_manifest_dataset(uri_old, None).await.unwrap(); - super::migrations::set_stamp_for_test(&mut ds, 3).await.unwrap(); + super::migrations::set_stamp_for_test(&mut ds, 3) + .await + .unwrap(); } let err = match Omnigraph::open(uri_old).await { Ok(_) => panic!("a sub-CURRENT graph must be refused on open"), @@ -1866,12 +1880,11 @@ async fn sub_current_graph_is_refused_then_rebuilt_via_export_import() { // Two (or N) writers committing DISJOINT tables on the same branch still share // one mutable `graph_head:main` row (one `object_id`, `WhenMatched::UpdateAll`). // Their table-version rows never collide (distinct `object_id`s), so the *only* -// row-level CAS contention is on `graph_head:main`. The contract under test: -// exactly one writer wins each CAS round; the loser retries, re-resolves its -// parent off the freshly-advanced head (inside the publisher's retry loop), and -// re-commits — so every writer commits and the resulting graph_commit DAG is a -// single LINEAR chain (no fork), not a tree. This is the cross-process -// disjoint-table fork closed by the shared head row (invariants.md §7.1). +// row-level CAS contention is on `graph_head:main`. Two contracts coexist: +// legacy `Any` publishers retry, re-parent, and eventually form one linear DAG; +// RFC-022 `ExactGraphHead` publishers instead reject the stale prepared write +// after the first winner changes authority. Both rely on the same shared row to +// prevent a fork; the tests below pin both behaviors explicitly. /// A microsecond UNIX timestamp for a `LineageIntent`, matching the genesis / /// commit-graph `created_at` unit. @@ -1882,6 +1895,180 @@ fn lineage_now_micros() -> i64 { .as_micros() as i64 } +/// Race two lineage-only publishes against the same exact named-branch head. +/// Returns after proving exactly one committed and the loser surfaced a typed +/// read-set change. `establish_head=false` exercises the load-bearing absent-row +/// case on a fresh branch; `true` exercises ordinary head advancement. +async fn assert_exact_named_head_race(establish_head: bool) { + use crate::error::ManifestConflictDetails; + + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap(); + let catalog = build_test_catalog(); + let mut mc = ManifestCoordinator::init(uri, &catalog).await.unwrap(); + mc.create_branch("feature").await.unwrap(); + + let publisher = GraphNamespacePublisher::new(uri, Some("feature")); + let empty = HashMap::new(); + let expected_head = if establish_head { + let intent = LineageIntent { + graph_commit_id: ulid::Ulid::new().to_string(), + branch: Some("feature".to_string()), + actor_id: None, + merged_parent_commit_id: None, + created_at: lineage_now_micros(), + }; + publisher + .publish(&[], &empty, Some(&intent)) + .await + .expect("establish named-branch graph head"); + Some(intent.graph_commit_id) + } else { + None + }; + + // The folded publish scan must preserve exact absence. In particular, the + // inferred lineage head inherited from main must not masquerade as a + // materialized `graph_head:feature` row. + let branch_manifest = open_manifest_dataset(uri, Some("feature")).await.unwrap(); + let scan = read_publish_scan(&branch_manifest).await.unwrap(); + assert_eq!(scan.graph_heads.get("feature").cloned(), expected_head); + let branch_identifier = branch_manifest.branch_identifier().await.unwrap(); + + let precondition = PublishPrecondition::ExactGraphHead(GraphHeadExpectation::new( + Some("feature"), + branch_identifier, + expected_head.clone(), + )); + let intent_a = LineageIntent { + graph_commit_id: ulid::Ulid::new().to_string(), + branch: Some("feature".to_string()), + actor_id: Some("act-a".to_string()), + merged_parent_commit_id: None, + created_at: lineage_now_micros(), + }; + let intent_b = LineageIntent { + graph_commit_id: ulid::Ulid::new().to_string(), + branch: Some("feature".to_string()), + actor_id: Some("act-b".to_string()), + merged_parent_commit_id: None, + created_at: lineage_now_micros(), + }; + let publisher_a = GraphNamespacePublisher::new(uri, Some("feature")); + let publisher_b = GraphNamespacePublisher::new(uri, Some("feature")); + let precondition_a = precondition.clone(); + let precondition_b = precondition; + + let (result_a, result_b) = tokio::join!( + async { + publisher_a + .publish_with_precondition(&[], &empty, Some(&intent_a), &precondition_a) + .await + }, + async { + publisher_b + .publish_with_precondition(&[], &empty, Some(&intent_b), &precondition_b) + .await + } + ); + + let (winner_id, loser_error) = match (result_a, result_b) { + (Ok(_), Err(err)) => (intent_a.graph_commit_id.clone(), err), + (Err(err), Ok(_)) => (intent_b.graph_commit_id.clone(), err), + (Ok(_), Ok(_)) => panic!("exact-head race silently re-parented both writers"), + (Err(a), Err(b)) => panic!("both exact-head writers failed: {a:?} / {b:?}"), + }; + + let OmniError::Manifest(error) = loser_error else { + panic!("expected typed manifest conflict, got {loser_error:?}"); + }; + assert_eq!( + error.details, + Some(ManifestConflictDetails::ReadSetChanged { + member: "graph_head:feature".to_string(), + expected: expected_head, + actual: Some(winner_id.clone()), + }) + ); + + let branch_manifest = open_manifest_dataset(uri, Some("feature")).await.unwrap(); + let (commits, heads) = read_graph_lineage(&branch_manifest).await.unwrap(); + assert_eq!(heads.get("feature"), Some(&winner_id)); + assert_eq!( + commits + .iter() + .filter(|commit| { + commit.graph_commit_id == intent_a.graph_commit_id + || commit.graph_commit_id == intent_b.graph_commit_id + }) + .count(), + 1, + "the rejected intent must not leave an immutable graph_commit row" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn exact_head_publish_rejects_reparent_on_fresh_and_established_named_branch() { + assert_exact_named_head_race(false).await; + assert_exact_named_head_race(true).await; +} + +#[tokio::test] +async fn exact_publish_rejects_named_branch_delete_recreate_aba() { + use crate::error::ManifestConflictDetails; + + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap(); + let catalog = build_test_catalog(); + let mut mc = ManifestCoordinator::init(uri, &catalog).await.unwrap(); + mc.create_branch("feature").await.unwrap(); + + let old_branch = open_manifest_dataset(uri, Some("feature")).await.unwrap(); + let old_identifier = old_branch.branch_identifier().await.unwrap(); + assert!( + read_publish_scan(&old_branch) + .await + .unwrap() + .graph_heads + .get("feature") + .is_none(), + "fresh named branch starts without its own graph_head row" + ); + + // Recreate the same name at the same logical fork point. Numeric manifest + // version and exact graph-head absence can repeat; only Lance's native + // branch identifier distinguishes the incarnation. + mc.delete_branch("feature").await.unwrap(); + mc.create_branch("feature").await.unwrap(); + let recreated = open_manifest_dataset(uri, Some("feature")).await.unwrap(); + let recreated_identifier = recreated.branch_identifier().await.unwrap(); + assert_ne!(old_identifier, recreated_identifier); + + let precondition = PublishPrecondition::ExactGraphHead(GraphHeadExpectation::new( + Some("feature"), + old_identifier, + None, + )); + let err = GraphNamespacePublisher::new(uri, Some("feature")) + .publish_with_precondition(&[], &HashMap::new(), None, &precondition) + .await + .expect_err("recreated branch must reject the old incarnation token"); + let OmniError::Manifest(error) = err else { + panic!("expected typed manifest conflict, got {err:?}"); + }; + match error.details { + Some(ManifestConflictDetails::ReadSetChanged { + member, + expected: Some(expected), + actual: Some(actual), + }) => { + assert_eq!(member, "branch_identifier:feature"); + assert_ne!(expected, actual); + } + other => panic!("expected branch-identifier ReadSetChanged, got {other:?}"), + } +} + /// Append one row to a two-column NODE table (`id`, `name`) and return the /// resulting `SubTableUpdate` at the new on-disk version. Generalizes /// `append_person_and_make_update` to any node table whose schema is `(id: @@ -1943,8 +2130,10 @@ async fn assert_linear_chain(uri: &str, expected_total: usize) -> String { ); // (1) exactly one genesis. - let genesis: Vec<&GraphLineageRow> = - rows.iter().filter(|r| r.parent_commit_id.is_none()).collect(); + let genesis: Vec<&GraphLineageRow> = rows + .iter() + .filter(|r| r.parent_commit_id.is_none()) + .collect(); assert_eq!( genesis.len(), 1, @@ -2038,8 +2227,16 @@ async fn concurrent_disjoint_writes_share_head_and_form_linear_chain() { // version on the other's table; contention is purely the shared head row. let empty = HashMap::new(); let (res_a, res_b) = tokio::join!( - async { publisher_a.publish(&changes_a, &empty, Some(&intent_a)).await }, - async { publisher_b.publish(&changes_b, &empty, Some(&intent_b)).await } + async { + publisher_a + .publish(&changes_a, &empty, Some(&intent_a)) + .await + }, + async { + publisher_b + .publish(&changes_b, &empty, Some(&intent_b)) + .await + } ); // BOTH commit: disjoint tables → the head-row CAS loser retries within @@ -2114,8 +2311,16 @@ async fn concurrent_disjoint_writes_form_linear_chain_on_s3() { }; let empty = HashMap::new(); let (res_a, res_b) = tokio::join!( - async { publisher_a.publish(&changes_a, &empty, Some(&intent_a)).await }, - async { publisher_b.publish(&changes_b, &empty, Some(&intent_b)).await } + async { + publisher_a + .publish(&changes_a, &empty, Some(&intent_a)) + .await + }, + async { + publisher_b + .publish(&changes_b, &empty, Some(&intent_b)) + .await + } ); res_a.expect("writer A must commit on S3"); res_b.expect("writer B must commit on S3"); @@ -2161,8 +2366,13 @@ async fn n_concurrent_disjoint_writers_converge_to_one_linear_chain() { .await, ); updates.push( - append_node_row_and_make_update(uri, &company_entry, &format!("c{i}"), &format!("C{i}")) - .await, + append_node_row_and_make_update( + uri, + &company_entry, + &format!("c{i}"), + &format!("C{i}"), + ) + .await, ); } assert_eq!(updates.len(), N); @@ -2216,7 +2426,11 @@ async fn n_concurrent_disjoint_writers_converge_to_one_linear_chain() { // All 8 distinct writer ids committed (no lost commit, no duplicate id). committed_ids.sort(); committed_ids.dedup(); - assert_eq!(committed_ids.len(), N, "every writer must commit exactly once"); + assert_eq!( + committed_ids.len(), + N, + "every writer must commit exactly once" + ); // The final DAG is a single linear chain of genesis + 8 = 9, no fork. assert_linear_chain(uri, N + 1).await; diff --git a/crates/omnigraph/src/db/mod.rs b/crates/omnigraph/src/db/mod.rs index 2ce3e291..12ee44a3 100644 --- a/crates/omnigraph/src/db/mod.rs +++ b/crates/omnigraph/src/db/mod.rs @@ -10,7 +10,7 @@ pub use commit_graph::GraphCommit; pub use graph_coordinator::{GraphCoordinator, ReadTarget, ResolvedTarget, SnapshotId}; pub use manifest::{Snapshot, SubTableEntry, SubTableUpdate}; pub(crate) use omnigraph::ensure_public_branch_ref; -pub(crate) use omnigraph::WriteTxn; +pub(crate) use omnigraph::{DeferredTableFork, WriteTxn}; pub use omnigraph::{ CleanupPolicyOptions, InitOptions, MergeOutcome, Omnigraph, OpenMode, PendingIndex, RepairAction, RepairClassification, RepairOptions, RepairStats, SchemaApplyOptions, @@ -21,30 +21,27 @@ use crate::error::{OmniError, Result}; pub(crate) const SCHEMA_APPLY_LOCK_BRANCH: &str = "__schema_apply_lock__"; -/// Mutation kind, threaded through the version-check call sites so the -/// engine can apply an op-kind-aware policy: +/// Mutation kind, threaded through the early table-version checks so the +/// engine can apply an op-kind-aware staging policy. This check is not the +/// RFC-022 publish authority: enrolled mutation/load attempts additionally +/// capture an exact branch-wide `WriteTxn`, then revalidate it while holding +/// the root-shared schema → branch → sorted-table gates. /// /// - `Insert` / `Merge`: skip the strict pre-stage `ensure_expected_version` -/// check. Lance's `MergeInsertBuilder` rebases concurrent appends; the -/// per-(table, branch) writer queue serializes `commit_staged`; the -/// publisher's CAS (refreshed under the queue via -/// `MutationStaging::commit_all`'s `snapshot_for_branch` call) catches -/// genuine cross-process drift as `ManifestConflictDetails::ExpectedVersionMismatch`. -/// The pre-stage strict check would over-reject in-process concurrent -/// inserts, which is exactly the case PR 2 / MR-686 designed the -/// per-table queue to allow. +/// check because their staged files are reclaimable and the complete prepared +/// attempt is checked later against the exact branch authority. On a +/// pre-effect `ReadSetChanged`, mutation Insert and load Append/Merge discard +/// the whole attempt and reprepare with a bounded retry; they never patch +/// table pins beneath an already-validated plan. /// -/// - `Update` / `Delete`: keep the strict check. These have read-modify-write -/// semantics; Lance moving between the read at stage time and the write -/// at commit time means the staged batch is computed against stale state. -/// The strict check guards the per-query SI invariant. SERIALIZABLE -/// opt-in (§VI.36 future seam) is the long-term answer for tighter -/// semantics; today, in-process update-update races on the same key -/// stay rejected as 409 — acceptable. +/// - `Update` / `Delete`: keep the strict early check because these are +/// read-modify-write effects computed from a pinned image. Enrolled attempts +/// still perform the later branch-wide revalidation; a mismatch is strict +/// `ReadSetChanged` (HTTP 409), not a transparent replay. /// -/// - `SchemaRewrite`: keep the strict check. Schema apply runs under the -/// graph-wide `__schema_apply_lock__` AND per-table queues; the strict -/// check is uncontested at that point. +/// - `SchemaRewrite`: keep the strict early check for overwrite/rewrite effects. +/// An enrolled load Overwrite also uses the exact branch-wide gate and +/// surfaces `ReadSetChanged`; schema apply has its own schema/table protocol. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(crate) enum MutationOpKind { Insert, diff --git a/crates/omnigraph/src/db/omnigraph.rs b/crates/omnigraph/src/db/omnigraph.rs index c3bfc2da..c2e0630e 100644 --- a/crates/omnigraph/src/db/omnigraph.rs +++ b/crates/omnigraph/src/db/omnigraph.rs @@ -25,7 +25,9 @@ use omnigraph_compiler::{ use crate::db::graph_coordinator::{GraphCoordinator, PublishedSnapshot}; use crate::error::{OmniError, Result}; use crate::runtime_cache::RuntimeCache; -use crate::storage::{StorageAdapter, join_uri, normalize_root_uri, storage_for_uri}; +use crate::storage::{ + StorageAdapter, join_uri, normalize_root_uri, storage_for_uri, write_queue_root_identity, +}; use crate::storage_layer::SnapshotHandle; use crate::table_store::TableStore; @@ -41,7 +43,7 @@ pub use repair::{ }; pub use schema_apply::SchemaApplyOptions; pub use table_ops::PendingIndex; -pub(crate) use table_ops::OpenedForMutation; +pub(crate) use table_ops::{DeferredTableFork, OpenedForMutation}; use super::commit_graph::GraphCommit; use super::manifest::{ @@ -49,10 +51,11 @@ use super::manifest::{ table_path_for_table_key, }; use super::schema_state::{ - SCHEMA_SOURCE_FILENAME, load_or_bootstrap_schema_contract, read_accepted_schema_ir, - recover_schema_state_files, schema_ir_staging_uri, schema_ir_uri, schema_source_staging_uri, - schema_source_uri, schema_state_staging_uri, schema_state_uri, validate_schema_contract, - write_schema_contract, write_schema_contract_staging, + SCHEMA_SOURCE_FILENAME, load_or_bootstrap_schema_contract, load_validated_schema_contract, + read_accepted_schema_ir, read_schema_state_identity, recover_schema_state_files, + schema_ir_staging_uri, schema_ir_uri, schema_source_staging_uri, schema_source_uri, + schema_state_staging_uri, schema_state_uri, validate_schema_contract, write_schema_contract, + write_schema_contract_staging, }; use super::{ ReadTarget, ResolvedTarget, SCHEMA_APPLY_LOCK_BRANCH, SnapshotId, is_internal_system_branch, @@ -84,15 +87,18 @@ pub struct SchemaApplyPreview { /// base ONCE so the per-table opens reuse the pinned version instead of /// re-resolving / re-validating per table. The schema contract is validated once /// (when `base` is captured). NOT a general "no re-resolution" handle — the -/// commit-time OCC re-read, the live-HEAD drift probe, and the fork-authority reads -/// stay fresh (correctness machinery). Step 5 (PublishPlan unification) makes this +/// commit-time OCC re-read (including a second schema-identity validation), the +/// live-HEAD drift probe, and the fork-authority reads stay fresh (correctness +/// machinery). Step 5 (PublishPlan unification) makes this /// the non-optional publish carrier. (Write/maintenance opens attach the shared /// per-graph `Session` via the `TableStore`-held handle — the dataset-opener /// unification; the S3 cost gate for that term is still owed.) /// /// Threaded as `Option<&WriteTxn>` through the mutate/load write chain /// (`open_for_mutation_on_branch`, `commit_all`, `commit_updates_on_branch_with_expected`) -/// so a single write validates the schema contract EXACTLY ONCE — at capture. When +/// so a single write fully validates the schema contract once at capture and once +/// under the pre-effect gates, plus one cheap capture-fence marker read — never +/// once per table. When /// present, the per-table resolves source the pinned `base` entry instead of calling /// `resolved_branch_target` / `snapshot_for_branch` / `fresh_snapshot_for_branch` /// (each of which re-runs `ensure_schema_state_valid`). When absent (`None` — every @@ -102,11 +108,34 @@ pub struct SchemaApplyPreview { /// `ensure_expected_version`, and the commit-time OCC re-read still opens a fresh /// manifest snapshot (via `fresh_snapshot_for_branch_unchecked`) — only the redundant /// schema re-validation is dropped. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct WriteAuthorityToken { + /// Lance-native branch identity. Stable across commits, different after + /// delete+recreate even when the branch name and numeric version repeat. + pub(crate) branch_identifier: lance::dataset::refs::BranchIdentifier, + /// Exact materialized `graph_head:` value; absence is first-class + /// for a fresh named branch. + pub(crate) graph_head: Option, + /// Accepted schema identity used during preparation. Supported schema + /// transitions also advance `graph_head`, which is the atomically + /// contended authority row for this first coarse-OCC slice. + pub(crate) schema_ir_hash: String, + pub(crate) schema_identity_version: u32, +} + +#[derive(Debug, Clone)] pub(crate) struct WriteTxn { /// The resolved branch (`None` = main). pub(crate) branch: Option, /// The pinned base snapshot (per-table location + version + e_tag), captured once. pub(crate) base: Snapshot, + /// Complete coarse authority token for this prepared attempt. + pub(crate) authority: WriteAuthorityToken, + /// Catalog built from the exact accepted IR whose identity is recorded in + /// `authority`. Mutation/load planning and validation must use this snapshot, + /// never the handle-global catalog, which can lag a schema apply performed by + /// another long-lived handle. + pub(crate) catalog: Arc, } /// Top-level handle to an Omnigraph database. @@ -150,21 +179,25 @@ pub struct Omnigraph { /// Read-heavy on schema introspection paths, written only by /// `apply_schema`. Same ArcSwap rationale as `catalog`. schema_source: Arc>, - /// Per-`(table_key, branch)` writer queues — the engine's - /// write-serialization mechanism (the server holds the engine as a - /// lockless `Arc`). Reachable from engine internals - /// (mutation finalize, schema_apply, branch_merge, ensure_indices, - /// the fork path, recovery reconciler). + /// Root-scoped writer queues shared by every `Omnigraph` handle for this + /// canonical local root identity (or opaque remote URI) in the process. + /// Reachable from engine internals + /// (mutation finalize, schema_apply, branch_merge, ensure_indices, fork + /// paths, and both live/open-time recovery). Sharing across independently + /// opened handles is required because Restore/ref deletion is destructive. write_queue: Arc, - /// Process-wide mutex held across the swap → operate → restore window - /// in `branch_merge_impl`. Two concurrent merges with distinct targets + /// Handle-local mutex held across the swap → operate → restore window + /// in `branch_merge_impl`. Two concurrent merges through the same handle + /// with distinct targets /// would otherwise interleave their three separate /// `coordinator.write().await` acquisitions, leaving each merge's /// inner body running against the other's swapped coord. Pinned by /// `concurrent_branch_merges_distinct_targets_do_not_swap_into_each_other` /// in `crates/omnigraph-server/tests/server.rs`. /// - /// Cost: serializes ALL concurrent branch merges process-wide. + /// Cost: serializes all concurrent branch merges through one handle. + /// Independently opened handles own independent coordinators and mutexes; + /// their physical effects still meet at the root-scoped ordered gates. /// Acceptable because branch merges are heavy (table rewrites, index /// rebuilds), per-(table, branch) queues inside `commit_all` already /// serialize the data path, and merges are rare relative to /change @@ -273,6 +306,9 @@ impl Omnigraph { options: InitOptions, ) -> Result { let root = normalize_root_uri(uri)?; + let write_queue_identity = write_queue_root_identity(&root)?; + let write_queue = + crate::db::write_queue::WriteQueueManager::for_root(&write_queue_identity); // Preflight: refuse to clobber an existing graph unless the // operator passed `force`. This runs BEFORE any parse or @@ -316,7 +352,9 @@ impl Omnigraph { { return Err(OmniError::AlreadyInitialized { uri: root.clone() }); } - if let Err(err) = crate::failpoints::maybe_fail(crate::failpoints::names::INIT_AFTER_SCHEMA_PG_WRITTEN) { + if let Err(err) = crate::failpoints::maybe_fail( + crate::failpoints::names::INIT_AFTER_SCHEMA_PG_WRITTEN, + ) { best_effort_cleanup_init_artifacts(&root, storage.as_ref()).await; return Err(err); } @@ -375,7 +413,7 @@ impl Omnigraph { }), catalog: Arc::new(ArcSwap::from_pointee(catalog)), schema_source: Arc::new(ArcSwap::from_pointee(schema_source.to_string())), - write_queue: Arc::new(crate::db::write_queue::WriteQueueManager::new()), + write_queue, merge_exclusive: Arc::new(tokio::sync::Mutex::new(())), policy: None, embedding: Arc::new(tokio::sync::OnceCell::new()), @@ -410,6 +448,9 @@ impl Omnigraph { mode: OpenMode, ) -> Result { let root = normalize_root_uri(uri)?; + let write_queue_identity = write_queue_root_identity(&root)?; + let write_queue = + crate::db::write_queue::WriteQueueManager::for_root(&write_queue_identity); // Refuse a `__manifest` this binary cannot serve before the coordinator // reads any branch state — newer than CURRENT (an old binary must not // silently misread a newer graph) or below MIN_SUPPORTED (an older @@ -420,6 +461,20 @@ impl Omnigraph { // Open the coordinator first so the schema-staging recovery sweep can // compare its snapshot against any leftover staging files. let mut coordinator = GraphCoordinator::open(&root, Arc::clone(&storage)).await?; + // Schema publication is a three-file promotion plus an in-memory catalog + // swap. Every handle — including ReadOnly — must hold the root-scoped + // schema gate from its final coordinator refresh through the complete + // source/IR/state read and catalog construction. Otherwise an open can + // observe a partial promotion, or publish an old catalog after a live + // apply completed. ReadOnly still performs no recovery writes; this gate + // only serializes its read with an in-process publisher. + let schema_contract_guard = write_queue + .acquire(&crate::db::manifest::schema_apply_serial_queue_key()) + .await; + // `coordinator` was opened before the await above. A live handle may have + // published while this open blocked, so refresh under the continuously + // held gate before either recovery or contract capture. + coordinator.refresh().await?; // Both the schema-state recovery sweep AND the manifest-drift // recovery sweep are gated on `OpenMode::ReadWrite`. Read-only // consumers (NDJSON export, `commit list`, schema show) shouldn't @@ -430,6 +485,11 @@ impl Omnigraph { // the manifest pin is the consistent snapshot regardless of // drift on the per-table side or leftover schema-staging files. if matches!(mode, OpenMode::ReadWrite) { + // Schema staging is itself mutable recovery state. Hold the shared + // schema gate across BOTH its file pre-pass and the complete Full + // sidecar sweep, so `schema_state_recovery` cannot go stale in a + // release/reacquire gap. The sweep adds branch → sorted table gates + // per sidecar under this outer guard. let schema_state_recovery = recover_schema_state_files(&root, Arc::clone(&storage), &coordinator.snapshot()) .await?; @@ -445,9 +505,13 @@ impl Omnigraph { &mut coordinator, crate::db::manifest::RecoveryMode::Full, schema_state_recovery, + write_queue.as_ref(), ) .await?; } + crate::failpoints::maybe_fail( + crate::failpoints::names::OPEN_BEFORE_SCHEMA_CONTRACT_READ, + )?; // Read _schema.pg (post-recovery — may have just been renamed in). let schema_path = schema_source_uri(&root); let schema_source = storage.read_text(&schema_path).await?; @@ -464,7 +528,7 @@ impl Omnigraph { fixup_blob_schemas(&mut catalog); let session = Arc::new(lance::session::Session::default()); - Ok(Self { + let db = Self { root_uri: root.clone(), storage, coordinator: Arc::new(tokio::sync::RwLock::new(coordinator)), @@ -482,12 +546,16 @@ impl Omnigraph { }), catalog: Arc::new(ArcSwap::from_pointee(catalog)), schema_source: Arc::new(ArcSwap::from_pointee(schema_source)), - write_queue: Arc::new(crate::db::write_queue::WriteQueueManager::new()), + write_queue, merge_exclusive: Arc::new(tokio::sync::Mutex::new(())), policy: None, embedding: Arc::new(tokio::sync::OnceCell::new()), embedding_config: None, - }) + }; + // The returned handle now owns one coherent schema source/catalog view. + // Release only after both have been installed in the new object. + drop(schema_contract_guard); + Ok(db) } /// Returns an `Arc` snapshot. Cheap clone of the current @@ -590,7 +658,9 @@ impl Omnigraph { .map_err(|err| OmniError::Policy(err.to_string())) } - pub(crate) async fn ensure_schema_state_valid(&self) -> Result<()> { + pub(crate) async fn ensure_schema_state_valid( + &self, + ) -> Result { // Full per-call validation is intentional: a long-lived handle must // detect external drift of the schema source, IR, OR state on its next // operation (see lifecycle::long_lived_handle_rejects_schema_* tests). A @@ -600,6 +670,25 @@ impl Omnigraph { validate_schema_contract(self.uri(), Arc::clone(&self.storage)).await } + /// Load one operation-local catalog from the accepted schema contract while + /// the caller holds the root schema gate. + /// + /// Long-lived handles intentionally keep a warm ArcSwap catalog, so merely + /// validating the files does not make `self.catalog()` current after another + /// handle applies a schema. Control/legacy-adapter bridges use this capture + /// for planning and conservative table-gate enumeration. The caller MUST + /// already hold `schema_apply_serial_queue_key`; this helper does not acquire + /// it because the gate is a non-reentrant mutex. + pub(crate) async fn load_accepted_catalog_with_schema_gate_held( + &self, + ) -> Result> { + let (schema_ir, _) = + load_validated_schema_contract(self.uri(), Arc::clone(&self.storage)).await?; + let mut catalog = build_catalog_from_ir(&schema_ir)?; + fixup_blob_schemas(&mut catalog); + Ok(Arc::new(catalog)) + } + pub async fn plan_schema(&self, desired_schema_source: &str) -> Result { self.plan_schema_with_options(desired_schema_source, SchemaApplyOptions::default()) .await @@ -679,7 +768,7 @@ impl Omnigraph { schema_apply::ensure_schema_apply_idle(self, operation).await } - async fn ensure_schema_apply_not_locked(&self, operation: &str) -> Result<()> { + pub(crate) async fn ensure_schema_apply_not_locked(&self, operation: &str) -> Result<()> { schema_apply::ensure_schema_apply_not_locked(self, operation).await } @@ -718,11 +807,13 @@ impl Omnigraph { self.storage.as_ref() } - /// Per-`(table_key, branch)` writer queues. + /// Root-scoped writer queues (schema, branch, and `(table, branch)` gates). /// /// Engine-internal writers (mutation finalize, schema_apply, - /// branch_merge, ensure_indices) and the future MR-870 - /// recovery reconciler reach the queue manager via this accessor. + /// branch_merge, ensure_indices, maintenance, and live/open-time recovery) + /// reach the queue manager via this accessor. Independently + /// opened handles whose local paths resolve to the same canonical root (or + /// whose remote URIs match) return the same manager. /// Returns an `Arc` clone so callers can hold the manager across /// `&mut self` engine API boundaries. pub(crate) fn write_queue(&self) -> Arc { @@ -776,8 +867,10 @@ impl Omnigraph { /// schema) per table. Strict ops, the fork path, and the commit-time OCC re-read /// keep their fresh reads (those are correctness machinery — see the handoff doc). /// - /// "Once" covers the table-touch hot path captured here (proven by the node-insert - /// gate `write_validates_schema_contract_once`); it does NOT yet cover edge endpoint + /// "Once" covers the table-touch hot path captured here (the cost gate permits + /// one marker read plus one validation at pre-effect revalidation); it does + /// NOT yet + /// cover edge endpoint /// / cardinality RI validation (`ensure_node_id_exists`, the loader's RI/cardinality), /// which still resolve through `snapshot_for_branch` and re-validate. Those reads must /// observe LIVE committed state, so unifying them (validate-once + pinned + re-checked @@ -786,11 +879,54 @@ impl Omnigraph { /// per-graph `Session` (the dataset-opener unification); the S3 cost gate /// for that term is still owed (handoff §1d). pub(crate) async fn open_write_txn(&self, branch: Option<&str>) -> Result { - let resolved = self.resolved_branch_target(branch).await?; - Ok(WriteTxn { - branch: resolved.branch, - base: resolved.snapshot, - }) + const MAX_CAPTURE_RETRIES: usize = 8; + let branch = normalize_branch_name(branch.unwrap_or("main"))?; + + for _ in 0..MAX_CAPTURE_RETRIES { + // A schema apply publishes graph_head before promoting its staged + // contract. Read one fully validated IR/catalog, capture coherent + // manifest authority, then re-read the durable schema marker (the + // last promoted schema file) after a sentinel check. This accepts + // only (old head, old schema) or (new head, new schema), never the + // intermediate (new head, old schema) state, without paying for a + // second full schema parse during capture. + self.ensure_schema_apply_not_locked("write preparation") + .await?; + let (schema_ir, schema_state) = + load_validated_schema_contract(self.uri(), Arc::clone(&self.storage)).await?; + let (branch_identifier, graph_head, snapshot) = self + .write_authority_for_known_branch(branch.as_deref(), true) + .await?; + self.ensure_schema_apply_not_locked("write preparation") + .await?; + let trailing_schema_state = + read_schema_state_identity(self.uri(), self.storage.as_ref()).await?; + + if schema_state != trailing_schema_state { + tokio::task::yield_now().await; + continue; + } + + let mut catalog = build_catalog_from_ir(&schema_ir)?; + fixup_blob_schemas(&mut catalog); + return Ok(WriteTxn { + branch, + base: snapshot, + authority: WriteAuthorityToken { + branch_identifier, + graph_head, + schema_ir_hash: schema_state.schema_ir_hash, + schema_identity_version: schema_state.schema_identity_version, + }, + catalog: Arc::new(catalog), + }); + } + + Err(OmniError::manifest_read_set_changed( + format!("write_authority:{}", branch.as_deref().unwrap_or("main")), + None, + None, + )) } pub(crate) async fn resolved_branch_target( @@ -798,6 +934,13 @@ impl Omnigraph { branch: Option<&str>, ) -> Result { self.ensure_schema_state_valid().await?; + self.resolved_branch_target_unchecked(branch).await + } + + async fn resolved_branch_target_unchecked( + &self, + branch: Option<&str>, + ) -> Result { let requested = ReadTarget::Branch(branch.unwrap_or("main").to_string()); let normalized = normalize_branch_name(branch.unwrap_or("main"))?; let coord = self.coordinator.read().await; @@ -819,6 +962,99 @@ impl Omnigraph { coord.resolve_target(&requested).await } + /// Read the branch authority used by coarse OCC. When `fresh` is true the + /// warm coordinator is reused only after its cheap manifest-incarnation + /// probe proves it current; otherwise a fresh branch coordinator is opened. + async fn write_authority_for_known_branch( + &self, + branch: Option<&str>, + fresh: bool, + ) -> Result<( + lance::dataset::refs::BranchIdentifier, + Option, + Snapshot, + )> { + { + let coord = self.coordinator.read().await; + if branch == coord.current_branch() { + let current = if fresh { + let held = coord.manifest_incarnation(); + coord.probe_latest_incarnation().await?.matches(&held) + } else { + true + }; + if current { + return Ok(( + coord.branch_identifier().await?, + coord.exact_graph_head(), + coord.snapshot(), + )); + } + } + } + + let coord = self.open_coordinator_for_branch(branch).await?; + Ok(( + coord.branch_identifier().await?, + coord.exact_graph_head(), + coord.snapshot(), + )) + } + + /// Revalidate a prepared mutation/load attempt after its branch/table + /// gates are held and before recovery is armed or any Lance HEAD advances. + pub(crate) async fn revalidate_write_txn(&self, txn: &WriteTxn) -> Result { + // `commit_all` calls this while holding schema → branch → table gates. + // Recheck the durable sentinel inside that critical section so a schema + // apply observed after preparation cannot be followed by a table effect. + self.ensure_schema_apply_not_locked("write commit").await?; + let (branch_identifier, graph_head, snapshot) = self + .write_authority_for_known_branch(txn.branch.as_deref(), true) + .await?; + let schema_state = self.ensure_schema_state_valid().await?; + self.ensure_schema_apply_not_locked("write commit").await?; + if branch_identifier != txn.authority.branch_identifier { + return Err(OmniError::manifest_read_set_changed( + format!( + "branch_identifier:{}", + txn.branch.as_deref().unwrap_or("main") + ), + Some( + serde_json::to_string(&txn.authority.branch_identifier).map_err(|error| { + OmniError::manifest_internal(format!( + "serialize branch identifier: {error}" + )) + })?, + ), + Some(serde_json::to_string(&branch_identifier).map_err(|error| { + OmniError::manifest_internal(format!("serialize branch identifier: {error}")) + })?), + )); + } + if graph_head != txn.authority.graph_head { + return Err(OmniError::manifest_read_set_changed( + format!("graph_head:{}", txn.branch.as_deref().unwrap_or("main")), + txn.authority.graph_head.clone(), + graph_head, + )); + } + if schema_state.schema_ir_hash != txn.authority.schema_ir_hash { + return Err(OmniError::manifest_read_set_changed( + "schema_ir_hash".to_string(), + Some(txn.authority.schema_ir_hash.clone()), + Some(schema_state.schema_ir_hash), + )); + } + if schema_state.schema_identity_version != txn.authority.schema_identity_version { + return Err(OmniError::manifest_read_set_changed( + "schema_identity_version".to_string(), + Some(txn.authority.schema_identity_version.to_string()), + Some(schema_state.schema_identity_version.to_string()), + )); + } + Ok(snapshot) + } + pub(crate) async fn snapshot_for_branch(&self, branch: Option<&str>) -> Result { self.resolved_branch_target(branch) .await @@ -842,13 +1078,13 @@ impl Omnigraph { /// `resolve_target`. The OCC re-read uses only the returned `Snapshot` /// (per-table location + version), which `ManifestCoordinator::open().snapshot()` /// produces identically to `GraphCoordinator::open(...).snapshot()` — but - /// `resolve_target` additionally opens the commit graph (an extra - /// `_graph_commits.lance` probe) the OCC read never consults. Skipping that - /// load is a pure read-cost reduction, not a freshness change. The checked + /// `resolve_target` additionally assembles the lineage projection the OCC + /// read never consults. Skipping that work is a pure read-cost reduction, + /// not a freshness change. The checked /// `fresh_snapshot_for_branch` delegates here, so its no-`txn` callers /// (commit_all's None arm, optimize, repair, fork reclaim) get the same /// identical `Snapshot` via this lighter manifest-only read; they consume - /// only the snapshot and never relied on the commit-graph side load. + /// only the snapshot and never relied on the lineage projection. pub(crate) async fn fresh_snapshot_for_branch_unchecked( &self, branch: Option<&str>, @@ -862,33 +1098,6 @@ impl Omnigraph { Ok(manifest.snapshot()) } - /// Probe-gated OCC re-capture snapshot (RFC-013 PR2 #1a). The commit-time OCC - /// re-read MUST be as fresh as a cold re-scan (see `commit_all`), but a cold - /// `__manifest` full scan per write is O(fragments) and a dominant write-path - /// cost. When the warm coordinator is already bound to `branch` AND a cheap - /// incarnation probe (one object-store op, no row scan) proves it is current, - /// the warm snapshot IS byte-identical to a fresh re-read, so reuse it with - /// zero scans. On ANY mismatch — a concurrent in- or cross-process advance, or - /// a different bound branch — fall through to the cold - /// `fresh_snapshot_for_branch_unchecked` read, preserving the "must be fresh" - /// contract and cross-process drift detection. Mirrors the read path's - /// `resolve_target_inner` probe-and-reuse idiom; same-branch sequential writes - /// keep `coordinator` current (each commit refreshes its `known_state`), so the - /// common case reuses and skips the scan. The publish CAS - /// (`expected_table_versions`) remains the final arbiter. - pub(crate) async fn occ_snapshot_for_branch(&self, branch: Option<&str>) -> Result { - { - let coord = self.coordinator.read().await; - if branch == coord.current_branch() { - let held = coord.manifest_incarnation(); - if coord.probe_latest_incarnation().await?.matches(&held) { - return Ok(coord.snapshot()); - } - } - } - self.fresh_snapshot_for_branch_unchecked(branch).await - } - pub(crate) async fn version(&self) -> u64 { self.coordinator.read().await.version() } @@ -912,10 +1121,7 @@ impl Omnigraph { /// The on-disk internal-schema version of `target`'s branch (the storage-format /// version this graph is stamped at). Surfaced via `omnigraph snapshot`. - pub async fn internal_schema_version_of( - &self, - target: impl Into, - ) -> Result { + pub async fn internal_schema_version_of(&self, target: impl Into) -> Result { let branch = self.resolved_branch_of(target).await?; crate::db::manifest::internal_schema_stamp_at(self.uri(), branch.as_deref()).await } @@ -931,6 +1137,16 @@ impl Omnigraph { /// Synchronize this handle's write base to the latest head of the named branch. pub async fn sync_branch(&self, branch: &str) -> Result<()> { + // Coordinator selection is handle-local, but branch merge temporarily + // swaps this same coordinator and native controls derive authority from + // its active branch. Join their root-shared schema gate so sync cannot + // replace the coordinator during either authority window. This also + // captures the schema contract and target coordinator coherently across + // a concurrent schema apply. Lock order remains schema -> coordinator. + let _schema_guard = self + .write_queue() + .acquire(&crate::db::manifest::schema_apply_serial_queue_key()) + .await; self.ensure_schema_state_valid().await?; let branch = normalize_branch_name(branch)?; let next = self.open_coordinator_for_branch(branch.as_deref()).await?; @@ -966,13 +1182,13 @@ impl Omnigraph { /// 3. `heal_pending_sidecars_roll_forward` — close the /// finalize→publisher residual via roll-forward; defer rollback /// work to next ReadWrite open. Serializes against live writers - /// by acquiring each sidecar's per-(table_key, branch) write - /// queues, so refresh never rolls forward an in-flight writer's - /// sidecar from under it. + /// by acquiring each sidecar's root-scoped schema → branch → sorted + /// table gates, so refresh never rolls forward an in-flight writer's + /// sidecar from under it, even from another handle. /// 4. `runtime_cache.invalidate_all` — drop stale per-snapshot caches. /// - /// Steady state cost: one `list_dir` of `__recovery/` (typically - /// returns empty → early return for both passes). No additional + /// Steady-state cost: two empty `list_dir` probes of `__recovery/` (the + /// standalone schema-staging guard and the sidecar healer). No additional /// Lance reads. /// /// The staged-write entry points (`load_as`, `mutate_as`) run the @@ -1000,8 +1216,8 @@ impl Omnigraph { // has drifted from the cached `schema_source`. Tokio's RwLock is // not reentrant, so holding the write across that call deadlocks. // Pinned by `composite_flow_schema_apply_then_branch_ops_no_deadlock_in_refresh`. - // The heal also takes the lock itself (queues → coordinator - // order), so it must run after this guard is released. + // The heal also takes the locks itself (schema → branch → tables → + // coordinator), so it must run after this guard is released. { // Hold the schema-apply serialization key across the // list-then-reconcile pair: without it, a live apply can @@ -1035,7 +1251,7 @@ impl Omnigraph { .await?; } } // ← guards released before the heal's queue acquisition - crate::db::manifest::heal_pending_sidecars_roll_forward( + let _outcome = crate::db::manifest::heal_pending_sidecars_roll_forward( &self.root_uri, Arc::clone(&self.storage), &self.coordinator, @@ -1047,28 +1263,168 @@ impl Omnigraph { Ok(()) } - /// Write-entry heal: converge any pending recovery sidecars (a - /// previously failed writer's Phase B → Phase C residual) before - /// starting a new staged write, so a long-lived process (the HTTP - /// server, an embedded handle) recovers on its next write instead - /// of wedging every write on the commit-time drift guard until - /// restart. Roll-forward only; rollback-eligible sidecars defer to - /// the next ReadWrite open exactly as [`refresh`](Self::refresh) - /// does. + /// Legacy write-entry heal: converge any roll-forward-eligible recovery + /// sidecars and leave rollback-eligible intents for the next ReadWrite open. /// - /// Steady-state cost: one `list_dir` of `__recovery/` (typically - /// empty → immediate return). See + /// This preserves the pre-RFC-022 behavior for adapters that have not yet + /// enrolled in the closed recovery barrier (schema apply and maintenance + /// adapters in this slice). Mutation/load and branch merge use + /// [`heal_pending_recovery_sidecars_for_write`](Self::heal_pending_recovery_sidecars_for_write) + /// instead and reject relevant unresolved intents before capturing a base. + /// + /// Steady-state cost here is one `list_dir` of `__recovery/` (typically + /// empty → immediate return). Enrolled mutation/load and branch-merge + /// attempts perform a second check under their effect gates to close the + /// post-prepare race. See /// `recovery::heal_pending_sidecars_roll_forward` for the - /// concurrency contract (per-table write-queue acquisition). + /// concurrency contract (root-scoped ordered gate acquisition). pub(crate) async fn heal_pending_recovery_sidecars(&self) -> Result<()> { - let processed = crate::db::manifest::heal_pending_sidecars_roll_forward( + let _outcome = self.heal_pending_recovery_sidecars_outcome().await?; + Ok(()) + } + + /// RFC-022 synchronous write/control recovery barrier. Run the live-safe + /// healer, then reject any guarded unresolved intent on a relevant graph branch. A + /// SchemaApply intent is graph-global because it changes the accepted schema + /// identity used by every branch. + /// + /// `relevant_branches` must use the engine convention (`None` = main), but + /// this helper defensively folds `Some("main")` as well so load's explicit + /// base representation cannot create a second main identity. + pub(crate) async fn heal_pending_recovery_sidecars_for_write( + &self, + relevant_branches: &[Option<&str>], + ) -> Result<()> { + let outcome = self.heal_pending_recovery_sidecars_outcome().await?; + let blocking = outcome.unresolved.iter().find(|intent| { + intent.writer_kind == crate::db::manifest::SidecarKind::SchemaApply + || relevant_branches + .iter() + .any(|branch| branch.filter(|name| *name != "main") == intent.branch.as_deref()) + }); + if let Some(intent) = blocking { + let table_scope = if intent.table_keys.is_empty() { + "no table pins".to_string() + } else { + format!("tables {}", intent.table_keys.join(", ")) + }; + return Err(OmniError::recovery_required( + intent.operation_id.clone(), + format!( + "pending {:?} recovery operation on branch '{}' blocks the synchronous \ + write/control recovery barrier ({table_scope}); reopen the graph \ + read-write before retrying", + intent.writer_kind, + intent.branch.as_deref().unwrap_or("main"), + ), + )); + } + Ok(()) + } + + /// Branch-delete recovery barrier. + /// + /// An unresolved sidecar on the branch being deleted is not a reason to + /// wedge deletion forever: once the native manifest ref is removed, that + /// sidecar's table effects are unreachable and the recovery sweep records an + /// orphan-discard audit. Safety comes from branch_delete subsequently taking + /// schema -> target branch -> every accepted-catalog table gate before the + /// ref mutation, which waits out any live in-process owner. SchemaApply + /// remains graph-global and must still block deletion. + async fn heal_pending_recovery_sidecars_for_branch_delete( + &self, + branch: &str, + ) -> Result<()> { + let outcome = self.heal_pending_recovery_sidecars_outcome().await?; + if let Some(intent) = outcome + .unresolved + .iter() + .find(|intent| intent.writer_kind == crate::db::manifest::SidecarKind::SchemaApply) + { + return Err(OmniError::recovery_required( + intent.operation_id.clone(), + format!( + "pending SchemaApply recovery operation blocks deletion of branch '{branch}'; \ + reopen the graph read-write before retrying" + ), + )); + } + Ok(()) + } + + /// Gate-aware half of the recovery barrier. + /// + /// Callers run the healer before preparation, then acquire schema/branch + /// control gates. A different writer may arm an intent in that gap, so native + /// branch-control operations and RFC-022 data writers must list once more + /// under their complete gate set before any independently durable effect. + /// This helper deliberately does not invoke recovery: recovery acquires the + /// same gates and would self-deadlock. + pub(crate) async fn ensure_no_pending_recovery_sidecars_under_gates( + &self, + relevant_branches: &[Option<&str>], + operation: &str, + ) -> Result<()> { + let sidecars = + crate::db::manifest::list_sidecars(&self.root_uri, self.storage.as_ref()).await?; + let blocking = sidecars.iter().find(|sidecar| { + let sidecar_branch = sidecar + .branch + .as_deref() + .filter(|branch| *branch != "main"); + sidecar.writer_kind == crate::db::manifest::SidecarKind::SchemaApply + || relevant_branches.iter().any(|branch| { + branch.filter(|name| *name != "main") == sidecar_branch + }) + }); + if let Some(sidecar) = blocking { + return Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + format!( + "pending {:?} recovery operation on branch '{}' blocks {operation}", + sidecar.writer_kind, + sidecar.branch.as_deref().unwrap_or("main"), + ), + )); + } + Ok(()) + } + + /// Final under-gate check for branch deletion. Target-branch sidecars are + /// intentionally allowed: the held complete table envelope proves no live + /// in-process owner can still be applying them, and deleting the branch + /// makes their effects unreachable. Only graph-global schema recovery can + /// still invalidate the operation. + async fn ensure_branch_delete_recovery_safe_under_gates( + &self, + branch: &str, + ) -> Result<()> { + let sidecars = + crate::db::manifest::list_sidecars(&self.root_uri, self.storage.as_ref()).await?; + if let Some(sidecar) = sidecars.iter().find(|sidecar| { + sidecar.writer_kind == crate::db::manifest::SidecarKind::SchemaApply + }) { + return Err(OmniError::recovery_required( + sidecar.operation_id.clone(), + format!( + "pending SchemaApply recovery operation blocks deletion of branch '{branch}'" + ), + )); + } + Ok(()) + } + + async fn heal_pending_recovery_sidecars_outcome( + &self, + ) -> Result { + let outcome = crate::db::manifest::heal_pending_sidecars_roll_forward( &self.root_uri, Arc::clone(&self.storage), &self.coordinator, &self.write_queue, ) .await?; - if processed { + if outcome.processed_any { // A rolled-forward SchemaApply sidecar moved disk + manifest // to the new schema (staging promoted, registrations // published); the in-memory catalog must follow or the very @@ -1077,10 +1433,21 @@ impl Omnigraph { self.reload_schema_if_source_changed().await?; self.invalidate_read_caches().await; } - Ok(()) + Ok(outcome) } async fn reload_schema_if_source_changed(&self) -> Result<()> { + // Recovery gates are intentionally scoped per sidecar and are released + // before this call. Reacquire the schema gate across the complete + // source/IR/state read and ArcSwap publication so a concurrent apply + // cannot interleave its sequential file promotions with this reload. + let _schema_guard = self + .write_queue + .acquire(&crate::db::manifest::schema_apply_serial_queue_key()) + .await; + crate::failpoints::maybe_fail( + crate::failpoints::names::SCHEMA_RELOAD_BEFORE_CONTRACT_READ, + )?; let schema_path = schema_source_uri(&self.root_uri); let schema_source = self.storage.read_text(&schema_path).await?; if schema_source == *self.schema_source.load_full() { @@ -1436,6 +1803,28 @@ impl Omnigraph { .map(str::to_string) } + /// Conservative table-gate envelope for graph-level control/maintenance. + /// + /// Current legacy sidecar writers acquire `(table_key, target_branch)` gates + /// but do not all acquire the coarse schema/branch gates yet. A native branch + /// operation or version-GC barrier therefore takes every catalog table key on + /// each graph branch it can affect before its final sidecar recheck. This is + /// intentionally broader than mutation/load's exact touched-table set. + pub(crate) fn table_queue_keys_for_branches( + &self, + branches: &[Option], + catalog: &Catalog, + ) -> Vec { + let table_keys = optimize::all_table_keys(catalog); + let mut queue_keys = Vec::with_capacity(table_keys.len() * branches.len()); + for branch in branches { + for table_key in &table_keys { + queue_keys.push((table_key.clone(), branch.clone())); + } + } + queue_keys + } + async fn ensure_branch_delete_safe(&self, branch: &str, branches: &[String]) -> Result<()> { let descendants = self .coordinator @@ -1489,8 +1878,9 @@ impl Omnigraph { for (table_key, table_path) in cleanup_targets { let dataset_uri = self.storage().dataset_uri(&table_path); - let outcome = match crate::failpoints::maybe_fail(crate::failpoints::names::BRANCH_DELETE_BEFORE_TABLE_CLEANUP) - { + let outcome = match crate::failpoints::maybe_fail( + crate::failpoints::names::BRANCH_DELETE_BEFORE_TABLE_CLEANUP, + ) { Ok(()) => { self.storage() .force_delete_branch(&dataset_uri, branch) @@ -1571,9 +1961,57 @@ impl Omnigraph { &omnigraph_policy::ResourceScope::TargetBranch(name.to_string()), actor, )?; - self.ensure_schema_state_valid().await?; - self.ensure_schema_apply_idle("branch_create").await?; ensure_public_branch_ref(name, "branch_create")?; + let target = normalize_branch_name(name)? + .ok_or_else(|| OmniError::manifest("cannot create branch 'main'".to_string()))?; + self.ensure_schema_state_valid().await?; + let source = self.active_branch().await; + let relevant = [source.as_deref(), Some(target.as_str())]; + // Native ref control follows the same closed barrier shape as data + // writes: heal before accepting authority, then re-check under + // schema -> source/target branch gates. + self.heal_pending_recovery_sidecars_for_write(&relevant) + .await?; + crate::failpoints::maybe_fail( + crate::failpoints::names::BRANCH_CONTROL_POST_RECOVERY_BARRIER, + )?; + let _schema_guard = self + .write_queue() + .acquire(&crate::db::manifest::schema_apply_serial_queue_key()) + .await; + let _branch_guards = self + .write_queue() + .acquire_branches(&[source.clone(), Some(target.clone())]) + .await; + self.refresh_coordinator_only().await?; + self.ensure_schema_apply_not_locked("branch_create").await?; + let control_catalog = self + .load_accepted_catalog_with_schema_gate_held() + .await?; + let table_queue_keys = self.table_queue_keys_for_branches( + &[source.clone(), Some(target.clone())], + &control_catalog, + ); + let _table_guards = self.write_queue().acquire_many(&table_queue_keys).await; + self.ensure_no_pending_recovery_sidecars_under_gates(&relevant, "branch_create") + .await?; + self.refresh_coordinator_only().await?; + self.ensure_schema_apply_not_locked("branch_create").await?; + self.ensure_schema_state_valid().await?; + if self + .coordinator + .read() + .await + .all_branches() + .await? + .iter() + .any(|branch| branch == &target) + { + return Err(OmniError::manifest_conflict(format!( + "branch '{}' already exists", + target + ))); + } self.coordinator.write().await.branch_create(name).await } @@ -1611,7 +2049,6 @@ impl Omnigraph { }, actor, )?; - self.ensure_schema_apply_idle("branch_create_from").await?; self.branch_create_from_impl(target, name, false).await } @@ -1632,6 +2069,58 @@ impl Omnigraph { ensure_public_branch_ref(name, "branch_create_from")?; } let branch = normalize_branch_name(&branch_name)?; + let target_branch = normalize_branch_name(name)? + .ok_or_else(|| OmniError::manifest("cannot create branch 'main'".to_string()))?; + self.ensure_schema_state_valid().await?; + let relevant = [branch.as_deref(), Some(target_branch.as_str())]; + self.heal_pending_recovery_sidecars_for_write(&relevant) + .await?; + crate::failpoints::maybe_fail( + crate::failpoints::names::BRANCH_CONTROL_POST_RECOVERY_BARRIER, + )?; + let _schema_guard = self + .write_queue() + .acquire(&crate::db::manifest::schema_apply_serial_queue_key()) + .await; + let _branch_guards = self + .write_queue() + .acquire_branches(&[branch.clone(), Some(target_branch.clone())]) + .await; + self.refresh_coordinator_only().await?; + self.ensure_schema_apply_not_locked("branch_create_from") + .await?; + let control_catalog = self + .load_accepted_catalog_with_schema_gate_held() + .await?; + let table_queue_keys = self + .table_queue_keys_for_branches( + &[branch.clone(), Some(target_branch.clone())], + &control_catalog, + ); + let _table_guards = self.write_queue().acquire_many(&table_queue_keys).await; + self.ensure_no_pending_recovery_sidecars_under_gates( + &relevant, + "branch_create_from", + ) + .await?; + self.refresh_coordinator_only().await?; + self.ensure_schema_apply_not_locked("branch_create_from") + .await?; + self.ensure_schema_state_valid().await?; + if self + .coordinator + .read() + .await + .all_branches() + .await? + .iter() + .any(|candidate| candidate == &target_branch) + { + return Err(OmniError::manifest_conflict(format!( + "branch '{}' already exists", + target_branch + ))); + } // Operate on a freshly-opened source coordinator that's owned locally // — never touch `self.coordinator`. The pre-fix implementation used // `swap_coordinator_for_branch` + operate + `restore_coordinator` as @@ -1674,12 +2163,36 @@ impl Omnigraph { &omnigraph_policy::ResourceScope::TargetBranch(name.to_string()), actor, )?; - self.ensure_schema_state_valid().await?; - self.ensure_schema_apply_idle("branch_delete").await?; ensure_public_branch_ref(name, "branch_delete")?; - self.refresh().await?; let branch = normalize_branch_name(name)? .ok_or_else(|| OmniError::manifest("cannot delete branch 'main'".to_string()))?; + self.ensure_schema_state_valid().await?; + self.heal_pending_recovery_sidecars_for_branch_delete(&branch) + .await?; + crate::failpoints::maybe_fail( + crate::failpoints::names::BRANCH_CONTROL_POST_RECOVERY_BARRIER, + )?; + let _schema_guard = self + .write_queue() + .acquire(&crate::db::manifest::schema_apply_serial_queue_key()) + .await; + let _branch_guard = self.write_queue().acquire_branch(Some(&branch)).await; + self.refresh_coordinator_only().await?; + self.ensure_schema_apply_not_locked("branch_delete").await?; + let control_catalog = self + .load_accepted_catalog_with_schema_gate_held() + .await?; + let table_queue_keys = + self.table_queue_keys_for_branches(&[Some(branch.clone())], &control_catalog); + let _table_guards = self.write_queue().acquire_many(&table_queue_keys).await; + self.ensure_branch_delete_recovery_safe_under_gates(&branch) + .await?; + crate::failpoints::maybe_fail( + crate::failpoints::names::BRANCH_DELETE_POST_TABLE_GATES, + )?; + self.refresh_coordinator_only().await?; + self.ensure_schema_apply_not_locked("branch_delete").await?; + self.ensure_schema_state_valid().await?; let branches = self.coordinator.read().await.branch_list().await?; if !branches.iter().any(|candidate| candidate == &branch) { return Err(OmniError::manifest_not_found(format!( @@ -1748,6 +2261,26 @@ impl Omnigraph { source_branch: Option<&str>, source_version: u64, active_branch: &str, + ) -> Result { + self.fork_dataset_from_entry_state_under_intent( + table_key, + full_path, + source_branch, + source_version, + active_branch, + None, + ) + .await + } + + pub(crate) async fn fork_dataset_from_entry_state_under_intent( + &self, + table_key: &str, + full_path: &str, + source_branch: Option<&str>, + source_version: u64, + active_branch: &str, + operation_id: Option<&str>, ) -> Result { match table_ops::fork_dataset_from_entry_state( self, @@ -1768,6 +2301,7 @@ impl Omnigraph { source_branch, source_version, active_branch, + operation_id, ) .await } @@ -1830,8 +2364,8 @@ impl Omnigraph { updates: &[crate::db::SubTableUpdate], expected_table_versions: &std::collections::HashMap, actor_id: Option<&str>, - txn: Option<&crate::db::WriteTxn>, - committed_handles: std::collections::HashMap, + txn: &crate::db::WriteTxn, + lineage_intent: crate::db::manifest::LineageIntent, ) -> Result { table_ops::commit_updates_on_branch_with_expected( self, @@ -1840,11 +2374,36 @@ impl Omnigraph { expected_table_versions, actor_id, txn, - committed_handles, + lineage_intent, ) .await } + /// Mint the immutable lineage identity before recovery is armed. Parentage + /// remains publisher-resolved under the exact branch-head precondition. + pub(crate) async fn new_lineage_intent_for_branch( + &self, + branch: Option<&str>, + actor_id: Option<&str>, + ) -> Result { + let current_branch = self + .coordinator + .read() + .await + .current_branch() + .map(str::to_string); + if branch.map(str::to_string) == current_branch { + return self + .coordinator + .read() + .await + .new_lineage_intent(actor_id, None); + } + self.open_coordinator_for_branch(branch) + .await? + .new_lineage_intent(actor_id, None) + } + /// Invalidate the cached graph index. Called after edge mutations. pub(crate) async fn invalidate_graph_index(&self) { table_ops::invalidate_graph_index(self).await @@ -2505,6 +3064,34 @@ edge WorksAt: Person -> Company ); } + #[cfg(unix)] + #[tokio::test] + async fn local_symlink_alias_handles_share_write_queue_manager() { + use std::os::unix::fs::symlink; + + let parent = tempfile::tempdir().unwrap(); + let real_parent = parent.path().join("real"); + let alias_parent = parent.path().join("alias"); + std::fs::create_dir(&real_parent).unwrap(); + symlink(&real_parent, &alias_parent).unwrap(); + + // `init` computes the manager identity while this suffix is absent; + // `open_read_only` computes it again through the symlink after creation. + let real_root = real_parent.join("graph.omni"); + let alias_root = alias_parent.join("graph.omni"); + let initialized = Omnigraph::init(real_root.to_str().unwrap(), TEST_SCHEMA) + .await + .unwrap(); + let reopened = Omnigraph::open_read_only(alias_root.to_str().unwrap()) + .await + .unwrap(); + + assert!(Arc::ptr_eq( + &initialized.write_queue(), + &reopened.write_queue() + )); + } + #[tokio::test] async fn test_init_and_open_route_graph_metadata_through_storage_adapter() { let dir = tempfile::tempdir().unwrap(); diff --git a/crates/omnigraph/src/db/omnigraph/optimize.rs b/crates/omnigraph/src/db/omnigraph/optimize.rs index b768ac59..241edd99 100644 --- a/crates/omnigraph/src/db/omnigraph/optimize.rs +++ b/crates/omnigraph/src/db/omnigraph/optimize.rs @@ -21,8 +21,9 @@ //! retention. Destructive to version history — callers should gate this //! behind an explicit confirm flag at the CLI layer. //! -//! Both walk every node + edge table on the `main` branch. Run branches -//! are ephemeral by design so we do not optimize them. +//! Both orchestrate the graph's node + edge datasets from main authority; +//! cleanup preserves Lance-referenced named-branch history according to its +//! retention policy. use std::time::Duration; @@ -113,10 +114,10 @@ pub struct TableOptimizeStats { /// `fragments_removed == 0`, `fragments_added == 0`, and `!committed`. pub skipped: Option, /// Manifest table version observed by optimize for drift skips. `None` for - /// normal compaction/no-op/blob skips. + /// normal compaction/no-op outcomes. pub manifest_version: Option, /// Lance HEAD version observed by optimize for drift skips. `None` for - /// normal compaction/no-op/blob skips. + /// normal compaction/no-op outcomes. pub lance_head_version: Option, /// Declared `@index` columns on this table the reconciler could not build /// this run, each with the `reason` (today: a vector column with no @@ -141,20 +142,6 @@ impl TableOptimizeStats { } } - /// Stat for a table that was deliberately skipped (compaction not attempted). - fn skipped(table_key: String, reason: SkipReason) -> Self { - Self { - table_key, - fragments_removed: 0, - fragments_added: 0, - committed: false, - skipped: Some(reason), - manifest_version: None, - lance_head_version: None, - pending_indexes: Vec::new(), - } - } - /// Stat for a table skipped because the manifest and Lance HEAD disagree. fn skipped_for_drift( table_key: String, @@ -207,7 +194,6 @@ pub async fn optimize_all_tables(db: &Omnigraph) -> Result { - return Err(OmniError::Lance(format!("optimize_indices on {}: {}", table_key, e))); + return Err(OmniError::Lance(format!( + "optimize_indices on {}: {}", + table_key, e + ))); } } @@ -528,7 +518,9 @@ async fn optimize_one_table( // Pin the per-writer Phase B → Phase C residual: Lance HEAD has advanced but the // manifest publish below hasn't run. - crate::failpoints::maybe_fail(crate::failpoints::names::OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT)?; + crate::failpoints::maybe_fail( + crate::failpoints::names::OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT, + )?; // Phase C: monotonic fast-forward publish. The compaction is committed at Lance // HEAD `N`; publish a manifest pointer that includes it. If a concurrent writer @@ -734,10 +726,7 @@ async fn compact_internal_table( // conflict we re-open at the new HEAD and rerun — the canonical Lance-consumer // pattern. Each attempt opens fresh because the conflict means the version moved. for attempt in 0..COMPACTION_RETRY_BUDGET { - let handle = db - .storage() - .open_dataset_head(&uri, None) - .await?; + let handle = db.storage().open_dataset_head(&uri, None).await?; let mut ds = handle.into_dataset(); // Keep optimize non-destructive by construction (see clear_stale_auto_cleanup_config). @@ -745,8 +734,7 @@ async fn compact_internal_table( let cleared_config = match clear_stale_auto_cleanup_config(&mut ds).await { Ok(cleared) => cleared, Err(e) => { - if attempt + 1 < COMPACTION_RETRY_BUDGET && is_retryable_lance_conflict(&e) - { + if attempt + 1 < COMPACTION_RETRY_BUDGET && is_retryable_lance_conflict(&e) { continue; } return Err(OmniError::Lance(e.to_string())); @@ -784,10 +772,7 @@ async fn compact_internal_table( true, )); } - Err(e) - if attempt + 1 < COMPACTION_RETRY_BUDGET - && is_retryable_lance_conflict(&e) => - { + Err(e) if attempt + 1 < COMPACTION_RETRY_BUDGET && is_retryable_lance_conflict(&e) => { continue; } Err(e) => return Err(OmniError::Lance(e.to_string())), @@ -815,11 +800,46 @@ pub async fn cleanup_all_tables( db.ensure_schema_state_valid().await?; db.ensure_schema_apply_idle("cleanup").await?; + // Version GC must never run while recovery still needs exact Lance + // transaction/version history to prove effect ownership or resume an + // interrupted compensation. Refuse before orphan reconciliation or any + // per-table cleanup so this operation is all-or-nothing with respect to the + // recovery-history floor. A read-write reopen resolves the sidecar first. + if !crate::db::manifest::list_sidecars(db.root_uri(), db.storage_adapter()) + .await? + .is_empty() + { + return Err(OmniError::manifest_conflict( + "cleanup requires a clean recovery state; reopen the graph to run the \ + recovery sweep before garbage-collecting versions", + )); + } + crate::failpoints::maybe_fail( + crate::failpoints::names::CLEANUP_POST_RECOVERY_CHECK_PRE_GATES, + )?; + + // Close the empty-check -> GC race. Mutation/load take schema then branch + // then table gates; current legacy sidecar writers take at least their table + // gates. Cleanup takes the conservative superset and holds it through every + // `cleanup_old_versions` call, then performs the authoritative sidecar check + // under those gates. Without this envelope a writer can arm+commit+fail after + // the fast check and GC can delete the exact transaction/version history + // Full recovery needs to prove ownership or Restore. + let _cleanup_schema_guard = db + .write_queue() + .acquire(&crate::db::manifest::schema_apply_serial_queue_key()) + .await; + db.refresh_coordinator_only().await?; + db.ensure_schema_apply_not_locked("cleanup").await?; + let cleanup_catalog = db + .load_accepted_catalog_with_schema_gate_held() + .await?; + // Reclaim orphaned branch forks (from an incomplete prior `branch_delete`) // before version GC. Authority-derived and idempotent; the eager // best-effort reclaim in `branch_delete` covers the common case, this is // the guaranteed backstop. Logged for observability. - let reconciled = reconcile_orphaned_branches(db).await?; + let reconciled = reconcile_orphaned_branches_with_catalog(db, &cleanup_catalog).await?; if !reconciled.reclaimed.is_empty() { tracing::info!( count = reconciled.reclaimed.len(), @@ -835,13 +855,10 @@ pub async fn cleanup_all_tables( ); } - let before_timestamp = options.older_than.map(|d| Utc::now() - d); - let keep_versions = options.keep_versions; - let resolved = db.resolved_branch_target(None).await?; let snapshot = resolved.snapshot; - let table_tasks: Vec<_> = all_table_keys(&db.catalog()) + let table_tasks: Vec<_> = all_table_keys(&cleanup_catalog) .into_iter() .filter_map(|table_key| { let entry = snapshot.entry(&table_key)?; @@ -850,6 +867,38 @@ pub async fn cleanup_all_tables( }) .collect(); + // Schema gate stability means no native branch create/delete can change this + // set between enumeration and acquisition. Include main canonically as None; + // `all_branches` returns the user-facing "main" spelling. + let mut graph_branches = db + .coordinator + .read() + .await + .all_branches() + .await? + .into_iter() + .map(|branch| if branch == "main" { None } else { Some(branch) }) + .collect::>(); + graph_branches.push(None); + graph_branches.sort(); + graph_branches.dedup(); + let _cleanup_branch_guards = db.write_queue().acquire_branches(&graph_branches).await; + let gc_queue_keys = db.table_queue_keys_for_branches(&graph_branches, &cleanup_catalog); + let _cleanup_table_guards = db.write_queue().acquire_many(&gc_queue_keys).await; + + if !crate::db::manifest::list_sidecars(db.root_uri(), db.storage_adapter()) + .await? + .is_empty() + { + return Err(OmniError::manifest_conflict( + "cleanup observed a recovery sidecar after acquiring its GC gates; reopen the graph \ + read-write to recover before garbage-collecting versions", + )); + } + + let before_timestamp = options.older_than.map(|d| Utc::now() - d); + let keep_versions = options.keep_versions; + if table_tasks.is_empty() { return Ok(Vec::new()); } @@ -868,9 +917,7 @@ pub async fn cleanup_all_tables( // `cleanup_old_versions` is a Lance-only maintenance API not // surfaced through `TableStorage` — see the optimize path // above for the same rationale. Unwrap via `into_dataset()`. - let handle = storage - .open_dataset_head(&full_path, None) - .await?; + let handle = storage.open_dataset_head(&full_path, None).await?; let ds = handle.into_dataset(); let before_version = keep_versions .map(|n| ds.version().version.saturating_sub(n as u64)) @@ -928,8 +975,9 @@ pub struct BranchReconcileStats { pub failures: Vec<(String, String)>, } -/// Drop every per-table and commit-graph Lance branch fork the manifest does -/// not reference. +/// Drop every per-table Lance branch fork the manifest does not reference. +/// Graph lineage lives in `__manifest`; the retired standalone commit datasets +/// have no branch-ref cleanup path here. /// /// Two origins produce a manifest-unreferenced fork: /// 1. A `branch_delete` flips the manifest authority (atomic) but a @@ -953,7 +1001,16 @@ pub struct BranchReconcileStats { /// are dropped before parents (longest name first). Idempotent and authority- /// derived: no-ops once reconciled, and degrades to finding nothing if a future /// Lance atomic multi-dataset branch op prevents orphans from forming. +#[cfg(all(test, feature = "failpoints"))] pub async fn reconcile_orphaned_branches(db: &Omnigraph) -> Result { + let catalog = db.catalog(); + reconcile_orphaned_branches_with_catalog(db, &catalog).await +} + +async fn reconcile_orphaned_branches_with_catalog( + db: &Omnigraph, + catalog: &omnigraph_compiler::catalog::Catalog, +) -> Result { use std::collections::{HashMap, HashSet}; // Live manifest branches: the set whose per-table placements are @@ -969,7 +1026,7 @@ pub async fn reconcile_orphaned_branches(db: &Omnigraph) -> Result = all_table_keys(&db.catalog()) + let table_targets: Vec<(String, String)> = all_table_keys(catalog) .into_iter() .filter_map(|table_key| { let entry = snapshot.entry(&table_key)?; @@ -1021,11 +1078,12 @@ pub async fn reconcile_orphaned_branches(db: &Omnigraph) -> Result db.snapshot_for_branch(Some(&branch)).await, - Err(injected) => Err(injected), - }; + let branch_snapshot = match crate::failpoints::maybe_fail( + crate::failpoints::names::CLEANUP_RESOLVE_BRANCH_SNAPSHOT, + ) { + Ok(()) => db.snapshot_for_branch(Some(&branch)).await, + Err(injected) => Err(injected), + }; match branch_snapshot { Ok(snap) => { branch_snapshots.insert(branch.clone(), snap); @@ -1083,7 +1141,7 @@ pub async fn reconcile_orphaned_branches(db: &Omnigraph) -> Result {} super::table_ops::ForkRefStatus::Legitimate => continue, super::table_ops::ForkRefStatus::Indeterminate => { @@ -1101,7 +1159,9 @@ pub async fn reconcile_orphaned_branches(db: &Omnigraph) -> Result storage.force_delete_branch(&full_path, &branch).await, Err(injected) => Err(injected), }; diff --git a/crates/omnigraph/src/db/omnigraph/schema_apply.rs b/crates/omnigraph/src/db/omnigraph/schema_apply.rs index 7fde3565..d860713d 100644 --- a/crates/omnigraph/src/db/omnigraph/schema_apply.rs +++ b/crates/omnigraph/src/db/omnigraph/schema_apply.rs @@ -152,6 +152,15 @@ where // exists, so the heal can never observe it. db.heal_pending_recovery_sidecars().await?; + // Process-local schema-control gate. RFC-022 mutation/load commit paths + // acquire this before their branch/table gates and retain it through + // publication. Taking it before the durable sentinel closes the old race in + // which schema apply could create the sentinel while a mutation already held + // a table queue, causing that mutation to advance Lance HEAD and only then + // discover the schema lock. The native sentinel remains the cross-handle / + // crash-visible authority; this queue removes the avoidable same-handle race. + let schema_gate_key = crate::db::manifest::schema_apply_serial_queue_key(); + let _schema_gate = db.write_queue().acquire(&schema_gate_key).await; acquire_schema_apply_lock(db).await?; let result = apply_schema_with_lock(db, desired_schema_source, options, validate_catalog).await; let release_result = release_schema_apply_lock(db).await; @@ -440,23 +449,17 @@ where // practice. They exist for symmetry with the recovery reconciler, which // acquires the same queues before any `Dataset::restore` it issues for // SchemaApply sidecars. - let mut schema_apply_queue_keys: Vec<(String, Option)> = recovery_pins + let schema_apply_queue_keys: Vec<(String, Option)> = recovery_pins .iter() .map(|pin| (pin.table_key.clone(), pin.table_branch.clone())) .collect(); - // The serialization key the write-entry heal acquires before touching - // schema staging or a SchemaApply sidecar. Per-table keys alone don't - // cover a registration-only migration (no pins, but a sidecar and - // staging files on disk) — without this, a concurrent write's heal can - // promote this apply's staging files and publish its registrations out - // from under it. Acquired whenever a sidecar will be written, held - // through Phase D (the guards live to the end of this function). + // The outer `apply_schema` holds the schema-control serialization key from + // before sentinel creation through sentinel release. Per-table guards here + // therefore cover only the concrete table effects; acquiring the schema key + // again would deadlock because these queues are intentionally non-reentrant. let writes_sidecar = !(recovery_pins.is_empty() && sidecar_registrations.is_empty() && sidecar_tombstones.is_empty()); - if writes_sidecar { - schema_apply_queue_keys.push(crate::db::manifest::schema_apply_serial_queue_key()); - } let _schema_apply_queue_guards = db .write_queue() .acquire_many(&schema_apply_queue_keys) @@ -862,10 +865,7 @@ pub(super) async fn ensure_snapshot_entry_head_matches( let dataset_uri = db.storage().dataset_uri(&entry.table_path); let ds = db .storage() - .open_dataset_head( - &dataset_uri, - entry.table_branch.as_deref(), - ) + .open_dataset_head(&dataset_uri, entry.table_branch.as_deref()) .await?; db.storage() .ensure_expected_version(&ds, &entry.table_key, entry.table_version) diff --git a/crates/omnigraph/src/db/omnigraph/table_ops.rs b/crates/omnigraph/src/db/omnigraph/table_ops.rs index b3c3a69a..7fb85cd3 100644 --- a/crates/omnigraph/src/db/omnigraph/table_ops.rs +++ b/crates/omnigraph/src/db/omnigraph/table_ops.rs @@ -118,8 +118,7 @@ pub(super) async fn ensure_indices_for_branch( continue; } let full_path = format!("{}/{}", db.root_uri, entry.table_path); - if needs_index_work_node(db, type_name, &full_path, entry.table_branch.as_deref()).await? - { + if needs_index_work_node(db, type_name, &full_path, entry.table_branch.as_deref()).await? { recovery_pins.push(crate::db::manifest::SidecarTablePin { table_key, table_path: full_path, @@ -213,14 +212,13 @@ pub(super) async fn ensure_indices_for_branch( entry.table_version, active_branch, crate::db::MutationOpKind::SchemaRewrite, + false, ) .await? } }, None => ( - db.storage() - .open_dataset_head(&full_path, None) - .await?, + db.storage().open_dataset_head(&full_path, None).await?, None, ), }; @@ -261,14 +259,13 @@ pub(super) async fn ensure_indices_for_branch( entry.table_version, active_branch, crate::db::MutationOpKind::SchemaRewrite, + false, ) .await? } }, None => ( - db.storage() - .open_dataset_head(&full_path, None) - .await?, + db.storage().open_dataset_head(&full_path, None).await?, None, ), }; @@ -296,7 +293,9 @@ pub(super) async fn ensure_indices_for_branch( // (one commit_staged per index built) but the manifest publish below // hasn't run. Used by // `tests/failpoints.rs::ensure_indices_phase_b_failure_recovered_on_next_open`. - crate::failpoints::maybe_fail(crate::failpoints::names::ENSURE_INDICES_POST_PHASE_B_PRE_MANIFEST_COMMIT)?; + crate::failpoints::maybe_fail( + crate::failpoints::names::ENSURE_INDICES_POST_PHASE_B_PRE_MANIFEST_COMMIT, + )?; if !updates.is_empty() { commit_prepared_updates_on_branch(db, branch, &updates, None).await?; @@ -500,6 +499,16 @@ pub(crate) struct OpenedForMutation { pub(crate) expected_version: u64, pub(crate) full_path: String, pub(crate) table_branch: Option, + /// RFC-022 first-touch named-branch writes stage against the inherited + /// source snapshot and defer the durable Lance ref creation until after + /// their v3 recovery intent is armed in `StagedMutation::commit_all`. + pub(crate) deferred_fork: Option, +} + +#[derive(Debug, Clone)] +pub(crate) struct DeferredTableFork { + pub(crate) source_entry: crate::db::SubTableEntry, + pub(crate) target_branch: String, } impl OpenedForMutation { @@ -590,6 +599,7 @@ pub(super) async fn open_for_mutation_on_branch( expected_version: entry.table_version, full_path, table_branch: Some(active_branch.to_string()), + deferred_fork: None, }); } // Main branch, non-strict → no open. (Main never forks.) @@ -599,6 +609,7 @@ pub(super) async fn open_for_mutation_on_branch( expected_version: entry.table_version, full_path, table_branch: None, + deferred_fork: None, }); } // Non-strict but the table isn't on the active branch yet — falls @@ -609,13 +620,19 @@ pub(super) async fn open_for_mutation_on_branch( match resolved_branch.as_deref() { None => { - let ds = db - .storage() - .open_dataset_head(&full_path, None) - .await?; + let ds = db.storage().open_dataset_head(&full_path, None).await?; if op_kind.strict_pre_stage_version_check() { - db.storage() - .ensure_expected_version(&ds, table_key, entry.table_version)?; + if txn.is_some() && ds.version() != entry.table_version { + return Err(OmniError::manifest_read_set_changed( + format!("table_head:{table_key}"), + Some(entry.table_version.to_string()), + Some(ds.version().to_string()), + )); + } + if txn.is_none() { + db.storage() + .ensure_expected_version(&ds, table_key, entry.table_version)?; + } } let version = ds.version(); Ok(OpenedForMutation { @@ -623,9 +640,28 @@ pub(super) async fn open_for_mutation_on_branch( expected_version: version, full_path, table_branch: None, + deferred_fork: None, }) } Some(active_branch) => { + // RFC-022-enrolled mutation/load adapters must arm durable intent + // before creating a per-table Lance branch ref. Read and stage from + // the inherited source entry now; `commit_all` creates the target + // ref after its v3 sidecar is durable, then commits this transaction + // onto the new ref. Legacy writers retain the eager fork path below. + if txn.is_some() && entry.table_branch.as_deref() != Some(active_branch) { + let ds = db.storage().open_snapshot_at_entry(entry).await?; + return Ok(OpenedForMutation { + handle: Some(ds), + expected_version: entry.table_version, + full_path, + table_branch: Some(active_branch.to_string()), + deferred_fork: Some(DeferredTableFork { + source_entry: entry.clone(), + target_branch: active_branch.to_string(), + }), + }); + } let (ds, table_branch) = open_owned_dataset_for_branch_write( db, table_key, @@ -634,6 +670,7 @@ pub(super) async fn open_for_mutation_on_branch( entry.table_version, active_branch, op_kind, + txn.is_some(), ) .await?; let version = ds.version(); @@ -642,6 +679,7 @@ pub(super) async fn open_for_mutation_on_branch( expected_version: version, full_path, table_branch, + deferred_fork: None, }) } } @@ -655,6 +693,7 @@ pub(super) async fn open_owned_dataset_for_branch_write( entry_version: u64, active_branch: &str, op_kind: crate::db::MutationOpKind, + occ_enrolled: bool, ) -> Result<(SnapshotHandle, Option)> { match entry_branch { Some(branch) if branch == active_branch => { @@ -663,8 +702,17 @@ pub(super) async fn open_owned_dataset_for_branch_write( .open_dataset_head(full_path, Some(active_branch)) .await?; if op_kind.strict_pre_stage_version_check() { - db.storage() - .ensure_expected_version(&ds, table_key, entry_version)?; + if occ_enrolled && ds.version() != entry_version { + return Err(OmniError::manifest_read_set_changed( + format!("table_head:{table_key}"), + Some(entry_version.to_string()), + Some(ds.version().to_string()), + )); + } + if !occ_enrolled { + db.storage() + .ensure_expected_version(&ds, table_key, entry_version)?; + } } Ok((ds, Some(active_branch.to_string()))) } @@ -678,11 +726,19 @@ pub(super) async fn open_owned_dataset_for_branch_write( let live = db.snapshot_for_branch(Some(active_branch)).await?; if let Some(entry) = live.entry(table_key) { if entry.table_branch.as_deref() == Some(active_branch) { - return Err(OmniError::manifest_expected_version_mismatch( - table_key, - entry_version, - entry.table_version, - )); + return if occ_enrolled { + Err(OmniError::manifest_read_set_changed( + format!("table_head:{table_key}"), + Some(entry_version.to_string()), + Some(entry.table_version.to_string()), + )) + } else { + Err(OmniError::manifest_expected_version_mismatch( + table_key, + entry_version, + entry.table_version, + )) + }; } } // The fork advances Lance state before the manifest publish. The @@ -760,7 +816,32 @@ pub(crate) async fn classify_fork_ref( db: &Omnigraph, table_key: &str, branch: &str, + excluding_operation_id: Option<&str>, ) -> ForkRefStatus { + // Deferred mutation/load forks are created only after their v3 sidecar is + // durable. Until the manifest publish places this table on `branch`, that + // sidecar is the only durable ownership record for the ref. Treat a + // matching pending intent as indeterminate rather than an orphan so neither + // destructive caller can steal a live writer's fork. The writer that owns + // the intent may exclude itself while reclaiming a genuinely stale ref it + // collided with; every other sidecar remains a hard stop. A list failure is + // likewise indeterminate -- cleanup must never turn missing authority into + // permission to delete. + let sidecars = + match crate::db::manifest::list_sidecars(db.root_uri(), db.storage_adapter()).await { + Ok(sidecars) => sidecars, + Err(_) => return ForkRefStatus::Indeterminate, + }; + if sidecars.iter().any(|sidecar| { + Some(sidecar.operation_id.as_str()) != excluding_operation_id + && sidecar.branch.as_deref() == Some(branch) + && sidecar.tables.iter().any(|pin| { + pin.table_key == table_key && pin.table_branch.as_deref() == Some(branch) + }) + }) { + return ForkRefStatus::Indeterminate; + } + // `classify.fresh_read` failpoint: simulate a transient failure of the // fresh-authority read (no-op without the `failpoints` feature). Lets a // test exercise the Indeterminate path — a read failure on a live branch @@ -819,13 +900,34 @@ pub(super) async fn reclaim_orphaned_fork_and_refork( source_branch: Option<&str>, source_version: u64, active_branch: &str, + current_operation_id: Option<&str>, ) -> Result { + // A v3 mutation/load sidecar is written before its deferred fork. A + // manifest-unreferenced ref claimed by another pending operation is live, + // not an orphan: never force-delete it. Excluding our own operation lets a + // writer reclaim a genuinely stale pre-existing ref after its own intent is + // durable. A sidecar-list failure is indeterminate and therefore loud. + let sidecars = crate::db::manifest::list_sidecars(db.root_uri(), db.storage_adapter()).await?; + if let Some(owner) = sidecars.iter().find(|sidecar| { + Some(sidecar.operation_id.as_str()) != current_operation_id + && sidecar.branch.as_deref() == Some(active_branch) + && sidecar.tables.iter().any(|pin| { + pin.table_key == table_key && pin.table_branch.as_deref() == Some(active_branch) + }) + }) { + return Err(OmniError::manifest_read_set_changed( + format!("fork_intent:{active_branch}:{table_key}"), + None, + Some(owner.operation_id.clone()), + )); + } + // Self-validate against FRESH authority before destroying anything. Only an // Orphan is reclaimable; a Legitimate status (a concurrent writer published // a real fork despite the caller's possibly-cached proof) or an // Indeterminate one (transient read) surfaces a retryable conflict rather // than stranding the manifest at a version the recreated ref won't have. - match classify_fork_ref(db, table_key, active_branch).await { + match classify_fork_ref(db, table_key, active_branch, current_operation_id).await { ForkRefStatus::Orphan => {} ForkRefStatus::Legitimate => { let actual = db @@ -834,6 +936,13 @@ pub(super) async fn reclaim_orphaned_fork_and_refork( .ok() .and_then(|s| s.entry(table_key).map(|e| e.table_version)) .unwrap_or(source_version); + if current_operation_id.is_some() { + return Err(OmniError::manifest_read_set_changed( + format!("table_head:{table_key}"), + Some(source_version.to_string()), + Some(actual.to_string()), + )); + } return Err(OmniError::manifest_expected_version_mismatch( table_key, source_version, @@ -1101,7 +1210,9 @@ async fn stage_and_commit_btree( // to demonstrate that a stage-step failure in the staged-index // path (`stage_create_btree_index` succeeded; `commit_staged` not // yet called) leaves no Lance-HEAD drift on the touched table. - crate::failpoints::maybe_fail(crate::failpoints::names::ENSURE_INDICES_POST_STAGE_PRE_COMMIT_BTREE)?; + crate::failpoints::maybe_fail( + crate::failpoints::names::ENSURE_INDICES_POST_STAGE_PRE_COMMIT_BTREE, + )?; let new_ds = db .storage() .commit_staged(ds.clone(), staged) @@ -1153,30 +1264,28 @@ async fn prepare_updates_for_commit( branch: Option<&str>, updates: &[crate::db::SubTableUpdate], txn: Option<&crate::db::WriteTxn>, - // Post-`commit_staged` handles handed out by `StagedMutation::commit_all` - // (RFC-013 step 3b, collapse #4): table_key → the handle already open at - // its just-committed version. When a table's handle is present, the index - // build below reuses it and SKIPS the `reopen_for_mutation` open. Absent - // entries (other writers — schema apply, merge, ensure_indices, tests — - // pass `HashMap::new()`) keep the byte-identical `reopen_for_mutation` - // path. Delete tables ARE staged now (MR-A), so their handle is present - // like any other staged write. - mut committed_handles: std::collections::HashMap, ) -> Result> { if updates.is_empty() { return Ok(Vec::new()); } - // With a `WriteTxn` the schema contract was validated once at capture, so - // reuse the pinned base entries (same per-branch manifest snapshot) instead - // of `snapshot_for_branch` (which re-runs `ensure_schema_state_valid`). Only - // the `entry(table_key).table_path` is read out of it here, identical to the - // no-txn path; the post-`commit_staged` index build below still reopens the - // dataset at its just-committed version. Without a txn, byte-identical. - let snapshot = match txn { - Some(txn) => txn.base.clone(), - None => db.snapshot_for_branch(branch).await?, - }; + // RFC-022 mutation/load adapter: the physical effect envelope must be + // closed before its recovery sidecar is armed. Building indexes here can + // add one or several extra Lance commits (and vector-index creation is an + // inline-commit residual), so those commits cannot be represented as the + // staged data transaction promised by the sidecar. Indexes are derived + // state: enrolled mutation/load writes publish the exact data-table result + // and leave declared-index materialization to the existing + // ensure_indices/optimize reconciler. Other writers keep their historical + // behavior until their own effect adapters migrate. + if txn.is_some() { + return Ok(updates.to_vec()); + } + + // Enrolled mutation/load returned above: derived-index work is deliberately + // outside their exact physical effect envelope. Legacy callers retain the + // historical reopen/build path. + let snapshot = db.snapshot_for_branch(branch).await?; let mut prepared = Vec::with_capacity(updates.len()); for update in updates { @@ -1190,38 +1299,26 @@ async fn prepare_updates_for_commit( let mut prepared_update = update.clone(); if prepared_update.row_count > 0 { let full_path = format!("{}/{}", db.root_uri, entry.table_path); - // Reuse the post-`commit_staged` handle when the caller handed one - // out (collapse #4): it is already open at exactly - // `prepared_update.table_version`, so the defense-in-depth strict - // re-check `reopen_for_mutation` would run is trivially satisfied - // and the open is redundant. When no handle is present (other - // writers, or any non-staged table), fall back to the byte-identical - // `reopen_for_mutation` path. - // - // Strict version check is correct on the fallback: this runs INSIDE + // Strict version check is correct here: this runs INSIDE // the publisher commit path, after `commit_staged` already // advanced Lance HEAD to `prepared_update.table_version`. // The check is a defense-in-depth assertion that the // dataset state matches what we just committed; not the // pre-stage race the op-kind policy targets. - let mut ds = match committed_handles.remove(&prepared_update.table_key) { - Some(ds) => ds, - None => { - reopen_for_mutation( - db, - &prepared_update.table_key, - &full_path, - prepared_update.table_branch.as_deref(), - prepared_update.table_version, - crate::db::MutationOpKind::SchemaRewrite, - ) - .await? - } - }; + let mut ds = reopen_for_mutation( + db, + &prepared_update.table_key, + &full_path, + prepared_update.table_branch.as_deref(), + prepared_update.table_version, + crate::db::MutationOpKind::SchemaRewrite, + ) + .await?; // Any column not yet buildable (e.g. a vector column whose rows // have null embeddings) is deferred and logged inside // build_indices; a later ensure_indices/optimize materializes it. - // The load/mutate/merge commit must not fail on it. + // Legacy merge/test callers must not fail on it; enrolled + // mutation/load callers returned before this block. let _pending = build_indices_on_dataset(db, &prepared_update.table_key, &mut ds).await?; let state = db.storage().table_state(&full_path, &ds).await?; @@ -1253,6 +1350,7 @@ async fn commit_prepared_updates( Ok(manifest_version) } +#[cfg(feature = "failpoints")] async fn commit_prepared_updates_with_expected( db: &Omnigraph, updates: &[crate::db::SubTableUpdate], @@ -1303,6 +1401,7 @@ pub(super) async fn commit_prepared_updates_on_branch( Ok(manifest_version) } +#[cfg(feature = "failpoints")] pub(super) async fn commit_prepared_updates_on_branch_with_expected( db: &Omnigraph, branch: Option<&str>, @@ -1356,14 +1455,8 @@ pub(super) async fn commit_updates( .await .current_branch() .map(str::to_string); - let prepared = prepare_updates_for_commit( - db, - current_branch.as_deref(), - updates, - None, - std::collections::HashMap::new(), - ) - .await?; + let prepared = + prepare_updates_for_commit(db, current_branch.as_deref(), updates, None).await?; commit_prepared_updates(db, &prepared, None).await } @@ -1390,20 +1483,60 @@ pub(super) async fn commit_updates_on_branch_with_expected( updates: &[crate::db::SubTableUpdate], expected_table_versions: &std::collections::HashMap, actor_id: Option<&str>, - txn: Option<&crate::db::WriteTxn>, - committed_handles: std::collections::HashMap, + txn: &crate::db::WriteTxn, + lineage_intent: crate::db::manifest::LineageIntent, ) -> Result { db.ensure_schema_apply_not_locked("write commit").await?; - let prepared = - prepare_updates_for_commit(db, branch, updates, txn, committed_handles).await?; - commit_prepared_updates_on_branch_with_expected( - db, + let prepared = prepare_updates_for_commit(db, branch, updates, Some(txn)).await?; + + debug_assert_eq!(lineage_intent.actor_id.as_deref(), actor_id); + let changes = prepared + .iter() + .cloned() + .map(ManifestChange::Update) + .collect::>(); + let expectation = crate::db::manifest::GraphHeadExpectation::new( branch, - &prepared, - expected_table_versions, - actor_id, - ) - .await + txn.authority.branch_identifier.clone(), + txn.authority.graph_head.clone(), + ); + let precondition = crate::db::manifest::PublishPrecondition::ExactGraphHead(expectation); + + let current_branch = db + .coordinator + .read() + .await + .current_branch() + .map(str::to_string); + let requested_branch = branch.map(str::to_string); + let published = if requested_branch == current_branch { + db.coordinator + .write() + .await + .commit_changes_with_intent_and_expected( + &changes, + expected_table_versions, + lineage_intent, + &precondition, + ) + .await? + } else { + let mut coordinator = match requested_branch.as_deref() { + Some(branch) => { + GraphCoordinator::open_branch(db.uri(), branch, Arc::clone(&db.storage)).await? + } + None => GraphCoordinator::open(db.uri(), Arc::clone(&db.storage)).await?, + }; + coordinator + .commit_changes_with_intent_and_expected( + &changes, + expected_table_versions, + lineage_intent, + &precondition, + ) + .await? + }; + Ok(published.manifest_version) } pub(super) async fn invalidate_graph_index(db: &Omnigraph) { @@ -1452,7 +1585,7 @@ mod classify_fork_ref_tests { .await .unwrap(); assert_eq!( - classify_fork_ref(&db, "node:Company", "feature").await, + classify_fork_ref(&db, "node:Company", "feature", None).await, ForkRefStatus::Legitimate, "a manifest-placed fork must classify as Legitimate (never destroyed)" ); @@ -1467,7 +1600,7 @@ mod classify_fork_ref_tests { ds.create_branch("feature", v, None).await.unwrap(); } assert_eq!( - classify_fork_ref(&db, "node:Person", "feature").await, + classify_fork_ref(&db, "node:Person", "feature", None).await, ForkRefStatus::Orphan, "a ref the manifest does not place on the branch must classify as Orphan" ); @@ -1480,7 +1613,7 @@ mod classify_fork_ref_tests { ds.create_branch("ghost", v, None).await.unwrap(); } assert_eq!( - classify_fork_ref(&db, "node:Person", "ghost").await, + classify_fork_ref(&db, "node:Person", "ghost", None).await, ForkRefStatus::Orphan, "a ref for a branch absent from the manifest must classify as Orphan" ); diff --git a/crates/omnigraph/src/db/recovery_audit.rs b/crates/omnigraph/src/db/recovery_audit.rs index 60137fe8..c8617b11 100644 --- a/crates/omnigraph/src/db/recovery_audit.rs +++ b/crates/omnigraph/src/db/recovery_audit.rs @@ -1,10 +1,11 @@ //! Recovery audit row storage in `_graph_commit_recoveries.lance`. //! -//! A standalone internal table (not catalog-tracked). Each successful -//! recovery sweep — roll-forward or roll-back — records one row here so -//! operators investigating a sidecar-attributed mutation can correlate -//! `omnigraph commit list --filter actor=omnigraph:recovery` with the -//! original actor whose mutation was rolled forward / back. +//! A standalone internal table (not catalog-tracked). Each completed recovery +//! action records one row here with the original actor and exact per-table +//! outcome. A v3 roll-forward preserves the interrupted writer's lineage and +//! actor, while rollback and legacy recovery lineage use +//! `omnigraph:recovery`; ordinary commit history is therefore not a complete +//! recovery log. This table currently has no public CLI query surface. //! //! This standalone table is additive: it doesn't bump //! `INTERNAL_MANIFEST_SCHEMA_VERSION`. Folding `recovery_for_actor` and @@ -14,10 +15,10 @@ //! Atomicity caveat: append to `_graph_commit_recoveries.lance` is //! sequential w.r.t. the recovery commit, which RFC-013 Phase 7 records in //! `__manifest` (folded into the recovery publish CAS via `publish_recovery_commit`). -//! A crash between the publish and this audit append leaves a recovery commit -//! with no audit row. The recovery sweep tolerates it the same way (re-entry -//! sees `NoMovement` for already-restored / already-published tables; the audit -//! append is retried, minting a fresh recovery commit). +//! A crash between the publish and this audit append leaves a visible outcome +//! with no audit row. V3 sidecars carry fixed outcome ids and durable audit +//! payloads, so re-entry appends the missing row without minting another +//! commit; legacy sidecars retain their stale-sidecar cleanup behavior. use std::sync::Arc; diff --git a/crates/omnigraph/src/db/schema_state.rs b/crates/omnigraph/src/db/schema_state.rs index b131a166..9335d131 100644 --- a/crates/omnigraph/src/db/schema_state.rs +++ b/crates/omnigraph/src/db/schema_state.rs @@ -78,7 +78,21 @@ pub(crate) async fn load_or_bootstrap_schema_contract( pub(crate) async fn validate_schema_contract( root_uri: &str, storage: Arc, -) -> Result<()> { +) -> Result { + load_validated_schema_contract(root_uri, storage) + .await + .map(|(_, state)| state) +} + +/// Load the accepted IR and its schema identity from one validated contract +/// read. Mutation/load preparation carries the catalog built from this exact IR +/// beside the identity in its `WriteTxn`; consulting the handle-global catalog +/// would let a long-lived handle combine a newly observed schema token with an +/// older in-memory plan. +pub(crate) async fn load_validated_schema_contract( + root_uri: &str, + storage: Arc, +) -> Result<(SchemaIR, SchemaState)> { let current_source_ir = read_current_source_ir(root_uri, storage.as_ref()).await?; let (persisted_ir, state) = match read_schema_contract(root_uri, storage.as_ref()).await? { SchemaContractRead::Present { ir, state } => (ir, state), @@ -90,7 +104,33 @@ pub(crate) async fn validate_schema_contract( }; validate_persisted_schema_contract(&persisted_ir, &state)?; - validate_current_source_matches(&state, ¤t_source_ir) + validate_current_source_matches(&state, ¤t_source_ir)?; + Ok((persisted_ir, state)) +} + +/// Read only the durable schema-identity marker. Schema apply promotes this +/// file after `_schema.pg` and `_schema.ir.json`, then releases its sentinel. +/// A capture path that already performed one full contract validation can use a +/// trailing marker read to detect the publish-before-promotion window without +/// paying for a second full source+IR parse. +pub(crate) async fn read_schema_state_identity( + root_uri: &str, + storage: &dyn StorageAdapter, +) -> Result { + let text = storage.read_text(&schema_state_uri(root_uri)).await?; + let state = serde_json::from_str::(&text).map_err(|err| { + schema_lock_conflict(format!( + "graph schema state in {} is invalid: {}", + SCHEMA_STATE_FILENAME, err + )) + })?; + if state.format_version != SCHEMA_STATE_FORMAT_VERSION { + return Err(schema_lock_conflict(format!( + "graph schema state format {} is unsupported", + state.format_version + ))); + } + Ok(state) } pub(crate) async fn write_schema_contract( diff --git a/crates/omnigraph/src/db/write_queue.rs b/crates/omnigraph/src/db/write_queue.rs index 07605662..0b24e578 100644 --- a/crates/omnigraph/src/db/write_queue.rs +++ b/crates/omnigraph/src/db/write_queue.rs @@ -1,15 +1,18 @@ //! Per-`(table_key, branch)` writer queues. //! -//! These queues are the engine's write-serialization mechanism: the server -//! holds the engine as a lockless `Arc` (writes are `&self`), so -//! disjoint-key writes proceed concurrently and only writes to the same -//! `(table_key, branch_ref)` serialize here. This module owns the queue -//! data structure; callers in `MutationStaging::commit_all`, `branch_merge`, -//! `schema_apply`, `ensure_indices`, the fork path (first write to a table on -//! a branch — acquired before the fork, held through the manifest publish), -//! and the recovery reconciler acquire guards before any per-table Lance -//! commit. Serialization is in-process only; cross-process -//! writers on one graph remain one-winner-CAS at the manifest publish. +//! These queues are the engine's process-local, root-scoped write-serialization +//! mechanism. The server normally holds one lockless `Arc`, but +//! independently opened handles for the same canonical local root identity +//! (or the same opaque object-store URI) share this manager too. Legacy +//! writers serialize only on `(table_key, branch_ref)` so +//! disjoint keys can proceed concurrently. RFC-022-enrolled mutation/load +//! attempts additionally take a coarse branch effect gate because validation +//! may depend on tables they do not write. This module owns both queue classes; +//! callers in `MutationStaging::commit_all`, branch controls, `branch_merge`, +//! `schema_apply`, `ensure_indices`, cleanup, branch forking, and recovery acquire the applicable guards +//! before a Lance HEAD advance or destructive recovery action. Serialization +//! remains in-process only; cross-process writers on one graph remain +//! one-winner-CAS at publish. //! //! ## Why exclusive `tokio::sync::Mutex<()>` per key //! @@ -32,13 +35,13 @@ //! ## Sorted-order acquisition //! //! `acquire_many` accepts a slice of keys and acquires them in -//! lexicographic order. Multi-table writers (mutation finalize, -//! branch_merge, future recovery reconciler) MUST go through +//! lexicographic order. Multi-table writers and control paths (mutation +//! finalize, branch merge, schema apply, maintenance, and recovery) MUST go through //! `acquire_many` so all callers agree on acquisition order — this is //! how lock-order inversion deadlock is prevented. use std::collections::HashMap; -use std::sync::{Arc, Mutex}; +use std::sync::{Arc, Mutex, OnceLock, Weak}; use tokio::sync::{Mutex as AsyncMutex, OwnedMutexGuard}; @@ -52,14 +55,26 @@ pub(crate) type TableQueueKey = (String, Option); /// Per-`(table_key, branch)` writer queue manager. /// -/// Lives on `Omnigraph` as `Arc` so HTTP handlers, -/// engine internals, the CLI binary, and future background reconcilers -/// (MR-870 recovery, MR-848 index) all reach it via the engine handle. +/// Every `Omnigraph` handle for one canonical root identity shares the same +/// manager via a process-global weak registry. This matters beyond HTTP's usual +/// `Arc` shape: a separately-opened handle can run recovery, and +/// Lance Restore/ref deletion must serialize with a live writer owned by the +/// first handle. The registry deliberately keys only by the queue root +/// identity; custom storage adapters for the same URI conservatively serialize +/// too. #[derive(Default)] pub(crate) struct WriteQueueManager { /// Held only briefly per `acquire` call: clone out the per-key Arc, /// release the std mutex, then await the per-key tokio Mutex. queues: Mutex>>>, + /// Coarse per-branch effect gate used by RFC-022-enrolled writers. + /// + /// This is deliberately separate from `queues`: a branch is authority, + /// not a synthetic table key. Enrolled writers acquire this gate before + /// any table queue and hold it through manifest publication. Legacy + /// writers continue to use only the table queues until their adapters are + /// migrated. + branch_queues: Mutex, Arc>>>, } impl WriteQueueManager { @@ -67,6 +82,24 @@ impl WriteQueueManager { Self::default() } + /// Return the process-wide queue manager for one canonical graph-root + /// identity. + /// Weak values avoid retaining every graph URI ever opened by a long-lived + /// multi-tenant process; lookup opportunistically removes dead entries. + pub(crate) fn for_root(root_identity: &str) -> Arc { + static REGISTRY: OnceLock>>> = + OnceLock::new(); + let registry = REGISTRY.get_or_init(|| Mutex::new(HashMap::new())); + let mut roots = registry.lock().expect("root write queue registry poisoned"); + if let Some(existing) = roots.get(root_identity).and_then(Weak::upgrade) { + return existing; + } + roots.retain(|_, manager| manager.strong_count() > 0); + let manager = Arc::new(Self::new()); + roots.insert(root_identity.to_string(), Arc::downgrade(&manager)); + manager + } + /// Get-or-create the per-key queue and clone its Arc. fn slot(&self, key: &TableQueueKey) -> Arc> { let mut map = self.queues.lock().expect("write queue map poisoned"); @@ -78,6 +111,55 @@ impl WriteQueueManager { fresh } + fn branch_slot(&self, branch: &Option) -> Arc> { + let mut map = self + .branch_queues + .lock() + .expect("branch write queue map poisoned"); + if let Some(existing) = map.get(branch) { + return Arc::clone(existing); + } + let fresh = Arc::new(AsyncMutex::new(())); + map.insert(branch.clone(), Arc::clone(&fresh)); + fresh + } + + /// Acquire the coarse effect gate for one graph branch. + /// + /// RFC-022-enrolled callers MUST acquire this before any per-table queue. + /// It is an in-process contention optimization only; publisher OCC and + /// recovery remain the correctness authorities. + pub(crate) async fn acquire_branch( + &self, + branch: Option<&str>, + ) -> OwnedMutexGuard<()> { + let key = branch.map(str::to_string); + self.branch_slot(&key).lock_owned().await + } + + /// Acquire several graph-branch control gates in one deterministic order. + /// + /// Native branch create-from reads a source ref and mutates a target ref, so + /// both incarnations must remain stable across its fresh revalidation and + /// visibility point. Sorting/deduping gives branch control the same + /// deadlock-free acquisition rule as [`Self::acquire_many`] gives tables. + pub(crate) async fn acquire_branches( + &self, + branches: &[Option], + ) -> Vec> { + if branches.is_empty() { + return Vec::new(); + } + let mut sorted = branches.to_vec(); + sorted.sort(); + sorted.dedup(); + let mut guards = Vec::with_capacity(sorted.len()); + for branch in sorted { + guards.push(self.branch_slot(&branch).lock_owned().await); + } + guards + } + /// Acquire exclusive access to the queue for one `(table_key, branch)`. /// /// Blocks until the lock is available. Drop the returned guard to @@ -112,6 +194,8 @@ impl WriteQueueManager { #[cfg(test)] mod tests { use super::*; + use crate::storage::write_queue_root_identity; + use std::path::PathBuf; use std::time::{Duration, Instant}; use tokio::time::timeout; @@ -140,6 +224,23 @@ mod tests { assert_eq!(guards.len(), 1); } + #[tokio::test] + async fn acquire_branches_dedupes_main_and_named_keys() { + let qm = WriteQueueManager::new(); + let guards = timeout( + Duration::from_secs(2), + qm.acquire_branches(&[ + Some("feature".to_string()), + None, + Some("feature".to_string()), + None, + ]), + ) + .await + .expect("duplicate branch keys must not self-deadlock"); + assert_eq!(guards.len(), 2); + } + #[tokio::test] async fn acquire_many_sorts_keys_deterministically() { // Two callers passing keys in different orders must acquire in @@ -232,4 +333,55 @@ mod tests { .await .expect("same-table-different-branch should not serialize"); } + + #[test] + fn opaque_root_registry_shares_manager_across_handles() { + let root = format!("memory://write-queue-registry/{}", ulid::Ulid::new()); + let first = WriteQueueManager::for_root(&root); + let second = WriteQueueManager::for_root(&root); + assert!(Arc::ptr_eq(&first, &second)); + + let other = WriteQueueManager::for_root(&format!("{root}/other")); + assert!(!Arc::ptr_eq(&first, &other)); + } + + #[test] + fn relative_and_absolute_local_roots_share_manager() { + let relative = PathBuf::from("target") + .join("write-queue-identities") + .join(ulid::Ulid::new().to_string()) + .join("graph.omni"); + let absolute = std::env::current_dir().unwrap().join(&relative); + let relative_identity = write_queue_root_identity(relative.to_str().unwrap()).unwrap(); + let absolute_identity = write_queue_root_identity(absolute.to_str().unwrap()).unwrap(); + + assert_eq!(relative_identity, absolute_identity); + let first = WriteQueueManager::for_root(&relative_identity); + let second = WriteQueueManager::for_root(&absolute_identity); + assert!(Arc::ptr_eq(&first, &second)); + } + + #[cfg(unix)] + #[test] + fn real_and_symlinked_local_roots_share_manager_before_init() { + use std::os::unix::fs::symlink; + + let parent = tempfile::tempdir().unwrap(); + let real_parent = parent.path().join("real"); + let alias_parent = parent.path().join("alias"); + std::fs::create_dir(&real_parent).unwrap(); + symlink(&real_parent, &alias_parent).unwrap(); + + // The graph suffix deliberately does not exist: init computes its + // queue identity before creating the graph directory. + let real_root = real_parent.join("future").join("graph.omni"); + let alias_root = alias_parent.join("future").join("graph.omni"); + let real_identity = write_queue_root_identity(real_root.to_str().unwrap()).unwrap(); + let alias_identity = write_queue_root_identity(alias_root.to_str().unwrap()).unwrap(); + + assert_eq!(real_identity, alias_identity); + let first = WriteQueueManager::for_root(&real_identity); + let second = WriteQueueManager::for_root(&alias_identity); + assert!(Arc::ptr_eq(&first, &second)); + } } diff --git a/crates/omnigraph/src/error.rs b/crates/omnigraph/src/error.rs index a24f1532..54bf30cf 100644 --- a/crates/omnigraph/src/error.rs +++ b/crates/omnigraph/src/error.rs @@ -22,6 +22,15 @@ pub enum ManifestConflictDetails { expected: u64, actual: u64, }, + /// A logical authority value captured during write preparation changed + /// before the manifest visibility decision. Unlike a touched-table + /// version mismatch, this may name a read-only dependency such as the + /// target branch's graph head or schema identity. + ReadSetChanged { + member: String, + expected: Option, + actual: Option, + }, /// Lance's row-level CAS rejected the publish because a concurrent writer /// landed a row with the same `object_id`. Distinct from /// `ExpectedVersionMismatch`: the caller's expectations (if any) still @@ -85,6 +94,16 @@ pub enum OmniError { Manifest(ManifestError), #[error("merge conflicts: {0:?}")] MergeConflicts(Vec), + /// A durable recovery intent overlaps this write. Its physical effects may + /// already have landed, or it may still be armed before its first effect; + /// either way the sidecar named by `operation_id` must be resolved before + /// the caller retries. Treating this as ordinary OCC would let a writer + /// advance around unresolved commit ownership. + #[error("recovery required for operation {operation_id}: {reason}")] + RecoveryRequired { + operation_id: String, + reason: String, + }, /// Engine-layer policy enforcement (MR-722). Wraps either a policy /// denial ("you can't do that") or a policy-evaluation failure /// ("the policy engine itself blew up"). The HTTP layer maps @@ -103,6 +122,16 @@ pub enum OmniError { } impl OmniError { + pub(crate) fn is_read_set_changed(&self) -> bool { + matches!( + self, + Self::Manifest(ManifestError { + details: Some(ManifestConflictDetails::ReadSetChanged { .. }), + .. + }) + ) + } + pub fn manifest(message: impl Into) -> Self { Self::Manifest(ManifestError::new(ManifestErrorKind::BadRequest, message)) } @@ -146,4 +175,37 @@ impl OmniError { .with_details(ManifestConflictDetails::RowLevelCasContention), ) } + + pub fn manifest_read_set_changed( + member: impl Into, + expected: Option, + actual: Option, + ) -> Self { + let member = member.into(); + let message = format!( + "write authority '{}' changed during preparation (expected {}, current {}) — reprepare from the current branch state", + member, + expected.as_deref().unwrap_or(""), + actual.as_deref().unwrap_or(""), + ); + Self::Manifest( + ManifestError::new(ManifestErrorKind::Conflict, message).with_details( + ManifestConflictDetails::ReadSetChanged { + member, + expected, + actual, + }, + ), + ) + } + + pub fn recovery_required( + operation_id: impl Into, + reason: impl Into, + ) -> Self { + Self::RecoveryRequired { + operation_id: operation_id.into(), + reason: reason.into(), + } + } } diff --git a/crates/omnigraph/src/exec/merge.rs b/crates/omnigraph/src/exec/merge.rs index 81fb4850..338047aa 100644 --- a/crates/omnigraph/src/exec/merge.rs +++ b/crates/omnigraph/src/exec/merge.rs @@ -635,16 +635,16 @@ fn row_signature(batch: &RecordBatch, row: usize) -> Result { /// it is validated like `AdoptWithDelta`; only an empty-delta adopt is skipped. async fn build_merge_changeset( db: &Omnigraph, + catalog: &Catalog, candidates: &HashMap, ) -> Result { - let catalog = db.catalog(); let mut changeset = crate::validate::ChangeSet::new(); for (table_key, candidate) in candidates { // Validation reads only id/src/dst + scalar constraint columns; project // out Vector/Blob so the change-set never holds embeddings (holding the // delta with embeddings would re-introduce the memory pressure the // streaming append exists to avoid). - let projection = validation_projection(&catalog, table_key); + let projection = validation_projection(catalog, table_key); let projection: Vec<&str> = projection.iter().map(String::as_str).collect(); let mut change = crate::validate::TableChange::default(); match candidate { @@ -736,7 +736,7 @@ async fn scan_staged_for_validation( } async fn validate_merge_candidates( - db: &Omnigraph, + catalog: &Catalog, target_snapshot: &Snapshot, changeset: &crate::validate::ChangeSet, ) -> Result<()> { @@ -746,9 +746,9 @@ async fn validate_merge_candidates( // uniqueness, edge-RI, and cardinality all route through one evaluator shared // with (eventually) the write path — closing the merge-vs-write drift. let committed = crate::validate::CommittedState::merge(target_snapshot); - let constraints = crate::validate::constraints_for(&db.catalog()); + let constraints = crate::validate::constraints_for(catalog); let violations = - crate::validate::evaluate(&constraints, changeset, &committed, &db.catalog()).await?; + crate::validate::evaluate(&constraints, changeset, &committed, catalog).await?; if violations.is_empty() { Ok(()) } else { @@ -1233,13 +1233,6 @@ impl Omnigraph { actor_id, )?; self.ensure_schema_apply_idle("branch_merge").await?; - // Converge any pending recovery sidecar before the merge - // captures its target snapshot: the merge's publish would - // otherwise make the drifted Phase-B commit visible as an - // unattributed side effect (manifest catches up to HEAD with no - // recovery audit row) and leave the stale sidecar behind. Runs - // before the merge's own sidecar exists. - self.heal_pending_recovery_sidecars().await?; self.branch_merge_impl(source, target, actor_id).await } @@ -1263,6 +1256,34 @@ impl Omnigraph { )); } + let relevant_branches = [source_branch.as_deref(), target_branch.as_deref()]; + // Branch merge is still a legacy per-table publisher, but its graph-ref + // authority must be stable for the complete prepare -> publish window. + // First converge or reject relevant recovery intent, then join the same + // root-shared schema -> branch order used by native branch controls. + // Holding both branch gates through publication prevents a target + // delete/recreate from reusing the branch name underneath a plan (ABA). + self.heal_pending_recovery_sidecars_for_write(&relevant_branches) + .await?; + let _schema_guard = self + .write_queue() + .acquire(&crate::db::manifest::schema_apply_serial_queue_key()) + .await; + let _branch_guards = self + .write_queue() + .acquire_branches(&[source_branch.clone(), target_branch.clone()]) + .await; + self.ensure_no_pending_recovery_sidecars_under_gates( + &relevant_branches, + "branch_merge", + ) + .await?; + self.refresh_coordinator_only().await?; + self.ensure_schema_apply_not_locked("branch_merge").await?; + let merge_catalog = self + .load_accepted_catalog_with_schema_gate_held() + .await?; + let source_head_commit_id = self .head_commit_id_for_branch(source_branch.as_deref()) .await? @@ -1298,6 +1319,15 @@ impl Omnigraph { )) .await? .snapshot; + let target_snapshot = self + .resolved_target(ReadTarget::Branch( + target_branch.clone().unwrap_or_else(|| "main".to_string()), + )) + .await? + .snapshot; + crate::failpoints::maybe_fail( + crate::failpoints::names::BRANCH_MERGE_POST_AUTHORITY_CAPTURE, + )?; // Hold the merge-exclusive mutex across the full swap → operate // → restore window. Two concurrent branch_merge calls would // otherwise interleave their three separate `coordinator.write()` @@ -1316,6 +1346,10 @@ impl Omnigraph { .branch_merge_on_current_target( &base_snapshot, &source_snapshot, + &target_snapshot, + merge_catalog.as_ref(), + source_branch.as_deref(), + target_branch.as_deref(), &target_head_commit_id, &source_head_commit_id, is_fast_forward, @@ -1353,8 +1387,8 @@ impl Omnigraph { // `crates/omnigraph/tests/failpoints.rs`. // // Err-path refresh is best-effort: the merge body's error - // (typically the structured `manifest_conflict` from the - // post_queue_snapshot drift check) is the value the caller + // (typically the structured read-set conflict from the fresh + // post-table-gate manifest check) is the value the caller // needs to see. A refresh-time storage error would replace // that with a less informative error; the next op or the next // `Omnigraph::open` will re-sync the coord anyway. @@ -1378,13 +1412,15 @@ impl Omnigraph { &self, base_snapshot: &Snapshot, source_snapshot: &Snapshot, + target_snapshot: &Snapshot, + catalog: &Catalog, + source_branch: Option<&str>, + target_branch: Option<&str>, target_head_commit_id: &str, source_head_commit_id: &str, is_fast_forward: bool, actor_id: Option<&str>, ) -> Result { - let target_snapshot = self.snapshot().await; - let mut table_keys = HashSet::new(); for entry in base_snapshot.entries() { table_keys.insert(entry.table_key.clone()); @@ -1415,10 +1451,10 @@ impl Omnigraph { if same_manifest_state(base_entry, target_entry) { let candidate = classify_adopt( self, - &self.catalog(), + catalog, base_snapshot, source_snapshot, - &target_snapshot, + target_snapshot, table_key, ) .await?; @@ -1428,10 +1464,10 @@ impl Omnigraph { if let Some(staged) = stage_streaming_table_merge( table_key, - &self.catalog(), + catalog, base_snapshot, source_snapshot, - &target_snapshot, + target_snapshot, &mut conflicts, ) .await? @@ -1447,8 +1483,8 @@ impl Omnigraph { return Err(OmniError::MergeConflicts(conflicts)); } - let changeset = build_merge_changeset(self, &candidates).await?; - validate_merge_candidates(self, &target_snapshot, &changeset).await?; + let changeset = build_merge_changeset(self, catalog, &candidates).await?; + validate_merge_candidates(catalog, target_snapshot, &changeset).await?; // Recovery sidecar: protect the per-table commit_staged loop. // Pin `RewriteMerged` and `AdoptWithDelta` candidates — both advance @@ -1469,54 +1505,48 @@ impl Omnigraph { // HEAD unpinned — is closed: `classify_adopt` pre-computes the delta, so a // HEAD-advancing adopt is `AdoptWithDelta` (pinned here) and an empty-delta // adopt stays `AdoptSourceState`. - // Acquire per-(table_key, target_branch) queues for every table - // touched by the merge plan. Sorted-order acquisition prevents - // lock-order inversion against concurrent multi-table writers. - // The active branch (set by the caller's `swap_coordinator_for_branch`) - // is the merge target; queue keys are scoped to it because a - // branch_merge writes only to the target branch. // - // Held across the per-table publish loop and the manifest - // commit + record_merge_commit calls below, so no concurrent - // writer to a touched (table, target_branch) can interleave - // between our commit_staged and our publish. - let active_branch_for_keys = self.active_branch().await; - let merge_queue_keys: Vec<(String, Option)> = ordered_table_keys - .iter() - .filter(|table_key| { - matches!( - candidates.get(*table_key), - Some(CandidateTableState::RewriteMerged(_)) - | Some(CandidateTableState::AdoptSourceState { .. }) - | Some(CandidateTableState::AdoptWithDelta(_)) - ) - }) - .map(|table_key| (table_key.clone(), active_branch_for_keys.clone())) - .collect(); + // This bridge still coexists with legacy maintenance writers that take + // only `(table, branch)` queues. Acquire the conservative all-catalog + // envelope for BOTH source and target, in the global sorted order, then + // re-run the sidecar barrier and re-read both manifest branches before + // Phase A. Planning remains outside table queues, but no plan derived + // from a stale source/target snapshot can cross into physical effects. + let active_branch_for_keys = target_branch.map(str::to_string); + let merge_branches = [ + source_branch.map(str::to_string), + active_branch_for_keys.clone(), + ]; + let merge_queue_keys = self.table_queue_keys_for_branches(&merge_branches, catalog); let _merge_queue_guards = self.write_queue().acquire_many(&merge_queue_keys).await; - let post_queue_snapshot = self.snapshot().await; - for table_key in &ordered_table_keys { - let Some(candidate) = candidates.get(table_key) else { - continue; - }; - if !matches!( - candidate, - CandidateTableState::RewriteMerged(_) - | CandidateTableState::AdoptSourceState { .. } - | CandidateTableState::AdoptWithDelta(_) - ) { - continue; - } - let expected = target_snapshot.entry(table_key).map(|e| e.table_version); - let current = post_queue_snapshot - .entry(table_key) - .map(|e| e.table_version); - if expected != current { - return Err(OmniError::manifest_expected_version_mismatch( - table_key.clone(), - expected.unwrap_or(0), - current.unwrap_or(0), + self.ensure_no_pending_recovery_sidecars_under_gates( + &[source_branch, target_branch], + "branch_merge after acquiring source/target table gates", + ) + .await?; + let fresh_source_snapshot = self + .fresh_snapshot_for_branch_unchecked(source_branch) + .await?; + let fresh_target_snapshot = self + .fresh_snapshot_for_branch_unchecked(target_branch) + .await?; + for (member, prepared, current) in [ + ("source", source_snapshot, &fresh_source_snapshot), + ("target", target_snapshot, &fresh_target_snapshot), + ] { + if prepared.version() != current.version() { + return Err(OmniError::manifest_read_set_changed( + format!( + "branch_merge_{member}:{}", + if member == "source" { + source_branch.unwrap_or("main") + } else { + target_branch.unwrap_or("main") + } + ), + Some(prepared.version().to_string()), + Some(current.version().to_string()), )); } } @@ -1607,7 +1637,7 @@ impl Omnigraph { }; let update = match candidate_state { CandidateTableState::AdoptSourceState { .. } => { - publish_adopted_source_state(self, source_snapshot, &target_snapshot, table_key) + publish_adopted_source_state(self, source_snapshot, target_snapshot, table_key) .await? } CandidateTableState::AdoptWithDelta(delta) => { @@ -1631,6 +1661,9 @@ impl Omnigraph { // `updates` carry the real per-table final versions (multiple // commit_staged calls per table, so not derivable from `post_commit_pin` // alone). A failure here leaves the unconfirmed sidecar → roll back. + crate::failpoints::maybe_fail( + crate::failpoints::names::BRANCH_MERGE_POST_EFFECTS_PRE_CONFIRM, + )?; if let Some((sidecar, _)) = recovery.as_mut() { let confirmed_versions: std::collections::HashMap = updates .iter() diff --git a/crates/omnigraph/src/exec/mutation.rs b/crates/omnigraph/src/exec/mutation.rs index c56897a8..ae820d08 100644 --- a/crates/omnigraph/src/exec/mutation.rs +++ b/crates/omnigraph/src/exec/mutation.rs @@ -382,21 +382,13 @@ fn predicate_to_sql( /// Replace specific columns in a RecordBatch with new literal values. /// -/// Blob columns may or may not be present in `batch` depending on the -/// caller's scan projection: -/// - If `batch` does NOT contain a blob column AND it has no assignment, -/// the column is OMITTED from the output. `merge_insert` leaves it -/// untouched. -/// - If `batch` DOES contain a blob column AND it has no assignment, the -/// column is COPIED to the output. This enables coalescing of -/// different-shape updates into a single full-schema merge batch (the -/// per-table accumulator in `MutationStaging` requires consistent -/// schemas across pending batches for `concat_batches`). The -/// round-tripping cost is acceptable for typical agent-driven -/// mutations; tables with large blobs and unassigned-blob updates may -/// want to be split into separate queries. -/// - If a blob column has a string-URI assignment, build the blob array -/// inline. +/// Blob-bearing updates always arrive with the full logical schema. Committed +/// blob payloads were materialized by the caller and rebuilt as logical +/// `Struct` arrays; pending batches already have that shape. An +/// unassigned blob is copied through, while an assigned string URI is rebuilt +/// with the same blob writer used by inserts. Consequently every update batch +/// has the catalog schema and can safely share one pending merge stream with +/// inserts and earlier updates. fn apply_assignments( full_schema: &SchemaRef, batch: &RecordBatch, @@ -404,8 +396,6 @@ fn apply_assignments( blob_properties: &HashSet, ) -> Result { let mut columns: Vec = Vec::with_capacity(full_schema.fields().len()); - let mut out_fields: Vec = Vec::with_capacity(full_schema.fields().len()); - for field in full_schema.fields().iter() { if blob_properties.contains(field.name()) { if let Some(Literal::String(uri)) = assignments.get(field.name()) { @@ -414,26 +404,25 @@ fn apply_assignments( for _ in 0..batch.num_rows() { crate::loader::append_blob_value(&mut builder, uri)?; } - let blob_field = lance::blob::blob_field(field.name(), true); - out_fields.push(blob_field); columns.push( builder .finish() .map_err(|e| OmniError::Lance(e.to_string()))?, ); - } else if let Some(col) = batch.column_by_name(field.name()) { - // Unassigned but scan included it: copy through (writes - // back the same blob, no observable change but uniform - // schema for the accumulator). - let blob_field = lance::blob::blob_field(field.name(), field.is_nullable()); - out_fields.push(blob_field); + } else { + // Unassigned: the materializing scan must have normalized the + // committed value (or pending value) to the logical blob + // schema, so copying it preserves both bytes and full-schema + // merge compatibility. + let col = batch.column_by_name(field.name()).ok_or_else(|| { + OmniError::Lance(format!( + "blob column '{}' not found in full-schema mutation scan", + field.name() + )) + })?; columns.push(col.clone()); } - // else: scan did not include this blob column and no - // assignment — omit. Caller's accumulator must accept the - // narrower schema (legacy single-merge_insert path). } else if let Some(lit) = assignments.get(field.name()) { - out_fields.push(field.as_ref().clone()); columns.push(literal_to_typed_array( lit, field.data_type(), @@ -446,13 +435,11 @@ fn apply_assignments( field.name() )) })?; - out_fields.push(field.as_ref().clone()); columns.push(col.clone()); } } - RecordBatch::try_new(Arc::new(Schema::new(out_fields)), columns) - .map_err(|e| OmniError::Lance(e.to_string())) + RecordBatch::try_new(full_schema.clone(), columns).map_err(|e| OmniError::Lance(e.to_string())) } // ─── Mutation execution ────────────────────────────────────────────────────── @@ -461,15 +448,15 @@ use super::staging::{MutationStaging, PendingMode}; /// Open a sub-table dataset for read or staged write within the current /// mutation query, capturing pre-write metadata in `staging` on first touch. -/// The captured version is the publisher's CAS fence at end-of-query -/// (per-table OCC). +/// The captured table version is the physical staging baseline. The publisher's +/// logical CAS fence is the enclosing `WriteTxn` authority: native branch +/// identity, exact optional graph head, and accepted schema identity. /// -/// On first touch, opens the dataset at HEAD on the requested branch -/// via `open_for_mutation_on_branch`, which compares Lance HEAD against -/// the manifest's pinned version — that fence is the engine's -/// publisher-style OCC catching cross-writer drift before we make any -/// changes. For delete-only queries, this strict open is also the uncovered -/// drift guard. +/// On first touch, resolves the table from the transaction's pinned base. +/// Strict read-modify-write operations open that exact version and retain the +/// early HEAD-vs-pin drift guard; Insert/Merge may skip the physical open and +/// carry only a reclaimable stage plan. Neither path substitutes for the final +/// branch-wide token check under the effect gates. /// /// On subsequent touches *within the same query*, Lance HEAD has not moved /// since first touch — inserts, updates AND deletes all stage their work and @@ -481,8 +468,7 @@ use super::staging::{MutationStaging, PendingMode}; /// touch records another predicate (`record_delete`), and `stage_all` combines /// them into one staged delete — there is no post-inline-commit reopen to /// special-case anymore. -impl Omnigraph { -} +impl Omnigraph {} async fn open_table_for_mutation( db: &Omnigraph, @@ -495,8 +481,8 @@ async fn open_table_for_mutation( // `open_for_mutation_on_branch` returns the expected version even when it // skips the open (collapse #1, the non-strict insert/merge path): the version // is the pinned base's, identical to the opened handle's `.version()`. Use it - // directly for `ensure_path` so the no-open path still captures the publisher - // CAS fence. + // directly for `ensure_path` so the no-open path retains its exact physical + // staging baseline; the branch-wide WriteTxn is the publisher CAS fence. let opened = db .open_for_mutation_on_branch(branch, table_key, op_kind, txn) .await?; @@ -512,6 +498,7 @@ async fn open_table_for_mutation( table_key, opened.full_path.clone(), opened.table_branch.clone(), + opened.deferred_fork.clone(), opened.expected_version, op_kind, ); @@ -648,18 +635,18 @@ impl Omnigraph { txn: &crate::db::WriteTxn, ) -> Result<()> { // RI/uniqueness read the write's already-validated pinned base (`txn.base`), - // NOT a fresh `snapshot_for_branch` — which would re-run the schema-contract - // validation the WriteTxn already did once (RFC-013 step 3b capture-once). - // Cardinality reads LIVE HEAD per edge table (the #298 stale-handle fix) via - // the live opener in `CommittedState::write`. + // NOT a fresh `snapshot_for_branch` — per-table resolution must not add a + // schema-contract validation beyond capture + the pre-effect gate. + // Cardinality reads a fresh manifest-visible branch snapshot (the #298 + // stale-handle fix) via `CommittedState::write`; it never follows an + // unpublished raw Lance HEAD or a not-yet-created first-touch ref. let committed = crate::validate::CommittedState::write(&txn.base, self, txn.branch.as_deref()); // `to_changeset` carries both constructive batches and the ids the delete // ops captured from their own scans (`deleted_ids`), so the evaluator // recounts the srcs a delete empties (`@card`) and sees removed rows for // RI — the faithful change-set the merge path also builds. - crate::validate::validate_changeset(&staging.to_changeset(), &committed, &self.catalog()) - .await + crate::validate::validate_changeset(&staging.to_changeset(), &committed, &txn.catalog).await } async fn mutate_with_current_actor( @@ -670,15 +657,51 @@ impl Omnigraph { params: &ParamMap, actor_id: Option<&str>, ) -> Result { - // Converge any pending recovery sidecar (a previously failed - // writer's Phase B → Phase C residual) before executing: the - // inline delete path advances Lance HEAD during execution and - // the staged path's commit-time drift guard refuses - // sidecar-covered drift, so a long-lived handle must heal here - // — not at restart. One `list_dir` when no sidecars exist (the - // steady state). MUST run before `open_write_txn` below — the heal - // may advance the manifest, so the pinned base must be captured after. - self.heal_pending_recovery_sidecars().await?; + const MAX_PRE_EFFECT_REPREPARES: usize = 32; + + // Resolve request-scoped values such as now() once so a safe + // pre-effect retry does not change the logical input. + let resolved_params = enrich_mutation_params(params)?; + for attempt in 0..=MAX_PRE_EFFECT_REPREPARES { + let mut retryable = false; + match self + .mutate_one_attempt( + branch, + query_source, + query_name, + &resolved_params, + actor_id, + &mut retryable, + ) + .await + { + Err(err) + if retryable + && err.is_read_set_changed() + && attempt < MAX_PRE_EFFECT_REPREPARES => + { + tracing::debug!( + attempt = attempt + 1, + branch, + "prepared mutation authority changed before effects; repreparing" + ); + self.refresh().await?; + } + result => return result, + } + } + unreachable!("bounded mutation retry loop always returns") + } + + async fn mutate_one_attempt( + &self, + branch: &str, + query_source: &str, + query_name: &str, + params: &ParamMap, + actor_id: Option<&str>, + retryable: &mut bool, + ) -> Result { let requested = Self::normalize_branch_name(branch)?; // Reject internal `__run__*` / system-prefixed branches at the // public write boundary. Direct-publish paths assert this @@ -687,62 +710,42 @@ impl Omnigraph { if let Some(name) = requested.as_deref() { crate::db::ensure_public_branch_ref(name, "mutate")?; } - // Capture-once write transaction (RFC-013 step 3b). `open_write_txn` - // validates the schema contract ONCE (it resolves the branch target, - // whose first line is `ensure_schema_state_valid`) and pins the base - // snapshot for this write. Threaded as `Some(&txn)` through execution, - // staging commit, and the manifest publish so the per-table opens and - // the commit-time OCC re-read reuse the pinned base instead of - // re-validating the contract at every resolve point. Captured AFTER the - // recovery heal (which may advance the manifest) and AFTER `requested` - // is known so it pins the post-heal snapshot for the correct branch. + // Stage A: converge any roll-forward-eligible sidecars, then close the + // barrier on every unresolved intent for this graph branch. This MUST + // run before `open_write_txn`: healing may advance the manifest, and a + // deferred Armed intent remains ownership even when no table HEAD moved. + self.heal_pending_recovery_sidecars_for_write(&[requested.as_deref()]) + .await?; + // Capture one branch-wide write authority after the recovery barrier: + // native branch identity, exact optional graph head, accepted schema + // identity/catalog, and the base table snapshot. Execution, validation, + // staging, and publication all use this immutable attempt. `commit_all` + // revalidates the complete token under the root-shared schema → branch → + // sorted-table gates before it arms recovery or advances Lance HEAD. let txn = self.open_write_txn(requested.as_deref()).await?; - let resolved_params = enrich_mutation_params(params)?; + let resolved_params = params.clone(); - // Per-query staging accumulator. Inserts and updates push batches - // into `pending`; deletes push predicates into `delete_predicates`. At - // end-of-query, `finalize` issues one `stage_*` + `commit_staged` per - // touched table (inserts/updates/deletes alike), then the publisher - // commits the manifest atomically across all touched tables. Branch is - // threaded explicitly — no coordinator swap. + // Per-query staging accumulator. Inserts and updates push batches into + // `pending`; deletes push predicates into `delete_predicates`. At the + // boundary, `stage_all` prepares one exact transaction per touched table + // and `commit_all` records those identities in a durable schema-v3 + // recovery intent before independently advancing the table HEADs. The + // publisher then makes the complete result graph-visible in one manifest + // CAS. Branch is threaded explicitly — no coordinator swap. let mut staging = MutationStaging::default(); // Lower + validate up front so the touched-table set is known before // execution. A lowering/validation error returns exactly as it did // when this happened inside execute_named_mutation. - let ir = self.lower_named_mutation(query_source, query_name)?; - - // Up-front fork-queue acquisition (see the loader for the full - // rationale): if this mutation will fork any touched table onto a - // non-main branch, acquire the per-(table, branch) write queues for - // every touched table before the first fork and hold them through the - // publish, so the orphan-fork reclaim can't race a concurrent - // in-process fork. The touched set is derived from the lowered IR. - let fork_queue_guards: Option<( - Vec<(String, Option)>, - Vec>, - )> = if let Some(active) = requested.as_deref() { - let snapshot = self.snapshot_for_branch(Some(active)).await?; - let touched: Vec<(String, Option)> = self - .touched_table_keys(&ir) - .into_iter() - .map(|k| (k, Some(active.to_string()))) - .collect(); - let needs_fork = touched.iter().any(|(table_key, _)| { - snapshot - .entry(table_key) - .map(|e| e.table_branch.as_deref() != Some(active)) - .unwrap_or(false) - }); - if needs_fork { - let guards = self.write_queue().acquire_many(&touched).await; - Some((touched, guards)) - } else { - None - } - } else { - None - }; + let ir = self.lower_named_mutation(&txn.catalog, query_source, query_name)?; + // Only an insert-only mutation is safe to replay automatically after a + // pre-effect authority mismatch. Update/Delete keep strict caller-visible + // `ReadSetChanged`; replaying their stale read-modify-write plan would be + // a semantic rebase rather than a fresh execution contract. + *retryable = ir + .ops + .iter() + .all(|op| matches!(op, MutationOpIR::Insert { .. })); let exec_result = self .execute_named_mutation( @@ -750,7 +753,7 @@ impl Omnigraph { &resolved_params, requested.as_deref(), &mut staging, - Some(&txn), + &txn, ) .await; @@ -760,62 +763,74 @@ impl Omnigraph { Ok(total) => { self.validate_staged_mutation(&staging, &txn).await?; let staged = staging.stage_all(self, requested.as_deref()).await?; - // `_queue_guards` holds per-(table_key, branch) write - // queues acquired inside `commit_all`. Held across the - // manifest publish below so no concurrent writer can - // interleave between our commit_staged and our publish - // (which would correctly fail our CAS but leave Lance - // HEAD advanced — the residual class MR-870 recovers). + crate::failpoints::maybe_fail( + crate::failpoints::names::MUTATION_POST_STAGE_PRE_EFFECT_GATE, + )?; + let lineage_intent = self + .new_lineage_intent_for_branch(requested.as_deref(), actor_id) + .await?; + // `_queue_guards` holds the root-shared schema gate, branch + // effect gate, and sorted table gates acquired by `commit_all`. + // They remain held through manifest publication, covering the + // complete same-process sidecar/effect lifetime. They are a + // local serialization aid; the exact publisher precondition and + // durable v3 recovery plan remain the correctness authorities. let super::staging::CommittedMutation { updates, expected_versions, sidecar_handle, guards: _queue_guards, - committed_handles, } = staged .commit_all( self, requested.as_deref(), crate::db::manifest::SidecarKind::Mutation, actor_id, - fork_queue_guards, - Some(&txn), + &txn, + &lineage_intent, ) .await?; - // Failpoint that wedges the documented finalize→publisher - // residual: per-table `commit_staged` calls already - // advanced Lance HEAD on every touched table; a failure - // injected here mirrors the production-rare case where - // the publisher's CAS pre-check rejects (or the manifest - // write throws) after staged commits succeeded. The - // sidecar written inside `staging.finalize()` persists - // across this failure so the next `Omnigraph::open`'s - // recovery sweep can roll forward — see + // Failpoint for the confirmed-effects → publisher boundary: + // table HEADs have advanced but graph visibility has not. The + // v3 sidecar already contains exact transaction identities, + // immutable manifest delta, and fixed lineage/rollback outcomes. + // Any failure from here is `RecoveryRequired`; synchronous heal + // or a read-write open converges the recorded outcome. See // `tests/failpoints.rs::recovery_rolls_forward_after_finalize_publisher_failure`. - crate::failpoints::maybe_fail(crate::failpoints::names::MUTATION_POST_FINALIZE_PRE_PUBLISHER)?; - self.commit_updates_on_branch_with_expected( - requested.as_deref(), - &updates, - &expected_versions, - actor_id, - Some(&txn), - committed_handles, - ) - .await?; - // Phase C succeeded — sidecar can be deleted. If this - // delete fails, the next open's sweep classifies every - // table as NoMovement (manifest pin == Lance HEAD == - // post_commit_pin) and the sidecar is treated as a - // stale artifact (cleaned up via the Phase 2 logic). + crate::failpoints::maybe_fail( + crate::failpoints::names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, + )?; + let publish_result = self + .commit_updates_on_branch_with_expected( + requested.as_deref(), + &updates, + &expected_versions, + actor_id, + &txn, + lineage_intent, + ) + .await; + if let Err(err) = publish_result { + // A sidecar exists iff at least one table effect was + // committed. Lineage-only / zero-row mutations have no + // physical residual to recover, so preserve their original + // publish error (notably ReadSetChanged) and let the normal + // retry/409 path handle it. + return match sidecar_handle.as_ref() { + Some(handle) => Err(OmniError::recovery_required( + handle.operation_id.clone(), + err.to_string(), + )), + None => Err(err), + }; + } if let Some(handle) = sidecar_handle { // Best-effort cleanup: the manifest publish already - // succeeded, so the user's mutation is durable. A - // failed delete leaves the sidecar on disk; the - // next open's recovery sweep classifies every table - // as `NoMovement` (manifest pin == Lance HEAD == - // post_commit_pin) and tidies up. Failing the user - // here would return an error for a write that - // already landed. + // succeeded, so the user's mutation is durable. A failed + // delete leaves a fixed, idempotent v3 outcome for the next + // synchronous heal or read-write open to audit and remove. + // Failing the user here would report an error for a write + // that already landed. if let Err(err) = crate::db::manifest::delete_sidecar(&handle, self.storage_adapter()).await { @@ -841,13 +856,14 @@ impl Omnigraph { /// is unchanged. fn lower_named_mutation( &self, + catalog: &omnigraph_compiler::catalog::Catalog, query_source: &str, query_name: &str, ) -> Result { let query_decl = omnigraph_compiler::find_named_query(query_source, query_name) .map_err(|e| OmniError::manifest(e.to_string()))?; - let checked = typecheck_query_decl(&self.catalog(), &query_decl)?; + let checked = typecheck_query_decl(catalog, &query_decl)?; match checked { CheckedQuery::Mutation(_) => {} CheckedQuery::Read(_) => { @@ -863,58 +879,13 @@ impl Omnigraph { Ok(ir) } - /// The COMPLETE set of `(node|edge):{type}` table keys a mutation IR can - /// touch at execution time, keyed as `MutationStaging`/`commit_all` key - /// them. Must be a superset of everything execution forks/commits, since - /// it drives the up-front fork-queue acquisition and `commit_all`'s - /// held-guard coverage check — a miss means an unserialized fork/commit. - /// - /// The set is a pure function of (IR ops + catalog). For each op it mirrors - /// the execute path's node-vs-edge dispatch (`node_types` first, then - /// `edge_types`). A `delete ` additionally **cascades** to every edge - /// type whose endpoint is that node (see `execute_delete_node`), forking - /// those edge tables during execution — so they are included here, derived - /// the same way the executor derives them (`from_type`/`to_type` match). - /// Unknown types are skipped (the execute path surfaces the error). - /// Sorted + deduped for one-shot `acquire_many`. - fn touched_table_keys(&self, ir: &omnigraph_compiler::ir::MutationIR) -> Vec { - use omnigraph_compiler::ir::MutationOpIR; - let catalog = self.catalog(); - let mut keys: Vec = Vec::new(); - for op in &ir.ops { - let type_name = match op { - MutationOpIR::Insert { type_name, .. } - | MutationOpIR::Update { type_name, .. } - | MutationOpIR::Delete { type_name, .. } => type_name, - }; - if catalog.node_types.contains_key(type_name) { - keys.push(format!("node:{type_name}")); - // A node delete cascades to every edge touching this node type, - // forking those edge tables. Include them so the up-front - // acquisition covers the cascade (mirrors execute_delete_node). - if matches!(op, MutationOpIR::Delete { .. }) { - for (edge_name, edge_type) in &catalog.edge_types { - if edge_type.from_type == *type_name || edge_type.to_type == *type_name { - keys.push(format!("edge:{edge_name}")); - } - } - } - } else if catalog.edge_types.contains_key(type_name) { - keys.push(format!("edge:{type_name}")); - } - } - keys.sort(); - keys.dedup(); - keys - } - async fn execute_named_mutation( &self, ir: &omnigraph_compiler::ir::MutationIR, params: &ParamMap, branch: Option<&str>, staging: &mut MutationStaging, - txn: Option<&crate::db::WriteTxn>, + txn: &crate::db::WriteTxn, ) -> Result { let mut total = MutationResult::default(); for op in &ir.ops { @@ -932,7 +903,13 @@ impl Omnigraph { predicate, } => { self.execute_update( - type_name, assignments, predicate, params, branch, staging, txn, + type_name, + assignments, + predicate, + params, + branch, + staging, + txn, ) .await? } @@ -957,18 +934,19 @@ impl Omnigraph { params: &ParamMap, branch: Option<&str>, staging: &mut MutationStaging, - txn: Option<&crate::db::WriteTxn>, + txn: &crate::db::WriteTxn, ) -> Result { let mut resolved: HashMap = HashMap::new(); for a in assignments { resolved.insert(a.property.clone(), resolve_expr_value(&a.value, params)?); } - let is_node = self.catalog().node_types.contains_key(type_name); - let is_edge = self.catalog().edge_types.contains_key(type_name); + let catalog = &txn.catalog; + let is_node = catalog.node_types.contains_key(type_name); + let is_edge = catalog.edge_types.contains_key(type_name); if is_node { - let node_type = &self.catalog().node_types[type_name]; + let node_type = &catalog.node_types[type_name]; let schema = node_type.arrow_schema.clone(); let blob_props = node_type.blob_properties.clone(); let id = if let Some(key_prop) = node_type.key_property() { @@ -1001,7 +979,8 @@ impl Omnigraph { // only `ensure_path`'s captured version (read inside // `open_table_for_mutation`) is used downstream. let (_ds, _full_path, _table_branch) = - open_table_for_mutation(self, staging, branch, &table_key, insert_kind, txn).await?; + open_table_for_mutation(self, staging, branch, &table_key, insert_kind, Some(txn)) + .await?; // Accumulate. @key inserts go into the Merge stream (so a // later update on the same id coalesces correctly); no-key // inserts go into the Append stream. @@ -1017,26 +996,28 @@ impl Omnigraph { affected_edges: 0, }) } else if is_edge { - let edge_type = &self.catalog().edge_types[type_name]; + let edge_type = &catalog.edge_types[type_name]; let schema = edge_type.arrow_schema.clone(); let blob_props = edge_type.blob_properties.clone(); let id = ulid::Ulid::new().to_string(); let batch = build_insert_batch(&schema, &id, &resolved, &blob_props)?; - // Validation (edge-RI, enum, unique, @card against LIVE HEAD) runs + // Validation (edge-RI, enum, unique, @card against the live + // manifest-visible branch snapshot) runs // end-of-query via the evaluator. let table_key = format!("edge:{}", type_name); // Capture pre-write metadata on first touch (ensure_path). Edge // inserts are non-strict, so with a `WriteTxn` this opens NOTHING // (collapse #1) and the handle is discarded — validation, including - // `@card` against LIVE HEAD, runs end-of-query via the evaluator. + // `@card` against the live committed branch snapshot, runs + // end-of-query via the evaluator. let (_handle, _full_path, _table_branch) = open_table_for_mutation( self, staging, branch, &table_key, crate::db::MutationOpKind::Insert, - txn, + Some(txn), ) .await?; // Accumulate the new edge row. Edge IDs are ULID-generated so @@ -1062,10 +1043,11 @@ impl Omnigraph { params: &ParamMap, branch: Option<&str>, staging: &mut MutationStaging, - txn: Option<&crate::db::WriteTxn>, + txn: &crate::db::WriteTxn, ) -> Result { + let catalog = &txn.catalog; // Defense in depth: ensure this is a node type - if !self.catalog().node_types.contains_key(type_name) { + if !catalog.node_types.contains_key(type_name) { return Err(OmniError::manifest(format!( "update is only supported for node types, not '{}'", type_name @@ -1073,7 +1055,7 @@ impl Omnigraph { } // Reject updates to @key properties — identity is immutable - if let Some(key_prop) = self.catalog().node_types[type_name].key_property() { + if let Some(key_prop) = catalog.node_types[type_name].key_property() { if assignments.iter().any(|a| a.property == key_prop) { return Err(OmniError::manifest(format!( "cannot update @key property '{}' — delete and re-insert instead", @@ -1083,8 +1065,8 @@ impl Omnigraph { } let pred_sql = predicate_to_sql(predicate, params, false)?; - let schema = self.catalog().node_types[type_name].arrow_schema.clone(); - let blob_props = self.catalog().node_types[type_name].blob_properties.clone(); + let schema = catalog.node_types[type_name].arrow_schema.clone(); + let blob_props = catalog.node_types[type_name].blob_properties.clone(); let table_key = format!("node:{}", type_name); let (handle, _full_path, _table_branch) = open_table_for_mutation( @@ -1093,7 +1075,7 @@ impl Omnigraph { branch, &table_key, crate::db::MutationOpKind::Update, - txn, + Some(txn), ) .await?; // Update is a STRICT op, so collapse #1 never skips its open — the @@ -1101,25 +1083,9 @@ impl Omnigraph { let ds = handle.expect("strict Update op always opens its dataset"); // Scan committed via Lance + apply the same predicate to pending - // batches via DataFusion `MemTable` (read-your-writes for prior - // ops in this query). The pending side may include rows from - // earlier `insert` / `update` ops on the same table. - // - // For blob tables we project away the blob columns: Lance's - // scanner doesn't accept the standard projection path on blob - // descriptors and would panic with a `Field::project` assertion. - // The downstream `apply_assignments` synthesizes blob columns - // from explicit assignments and omits unassigned blobs (Lance's - // merge_insert leaves them untouched). Tables without blob - // columns scan the full schema unprojected. - let non_blob_cols: Vec<&str> = schema - .fields() - .iter() - .filter(|f| !blob_props.contains(f.name())) - .map(|f| f.name().as_str()) - .collect(); - let projection: Option<&[&str]> = - (!blob_props.is_empty()).then_some(non_blob_cols.as_slice()); + // batches via DataFusion `MemTable` (read-your-writes for prior ops in + // this query). The pending side may include rows from earlier + // `insert` / `update` ops on the same table. let pending_batches = staging.pending_batches(&table_key); let pending_schema = staging.pending_schema(&table_key); // Use merge semantics on the union: a committed row whose `id` @@ -1128,17 +1094,34 @@ impl Omnigraph { // otherwise the predicate runs against stale committed values // and a chained `update where ` can match a row whose // pending value no longer satisfies . - let batches = self - .storage() - .scan_with_pending( - &ds, - pending_batches, - pending_schema, - projection, - Some(&pred_sql), - Some("id"), - ) - .await?; + // A blob-v2 scan normally yields physical descriptor structs, which + // cannot be fed back to the full-schema merge writer. Select matched + // committed row ids without projecting blobs, then take and rebuild + // only those payloads into the logical blob schema before unioning + // pending rows. This keeps correctness independent of whether an id + // index happens to steer Lance onto its legacy partial-column plan. + let batches = if blob_props.is_empty() { + self.storage() + .scan_with_pending( + &ds, + pending_batches, + pending_schema, + None, + Some(&pred_sql), + Some("id"), + ) + .await? + } else { + self.storage() + .scan_with_pending_materialized_blobs( + &ds, + pending_batches, + pending_schema, + Some(&pred_sql), + Some("id"), + ) + .await? + }; if batches.is_empty() || batches.iter().all(|b| b.num_rows() == 0) { return Ok(MutationResult { @@ -1148,14 +1131,9 @@ impl Omnigraph { } // Concat the matched batches (committed + pending) into one. The - // helper trusts that both sides share a schema — Lance returns - // dataset-schema-ordered columns and DataFusion returns - // MemTable-schema-ordered columns; both should match the catalog's - // arrow_schema when the projection is consistent. If they - // diverge (typically a blob-table mid-schema-shift), the helper - // surfaces a clear error directing the caller to split the - // mutation. - let matched = concat_match_batches_to_schema(&schema, &blob_props, batches)?; + // helper binds both sides to the catalog's full logical schema. Any + // divergence here is an internal scan/staging contract violation. + let matched = concat_match_batches_to_schema(&schema, batches)?; let affected_count = matched.num_rows(); @@ -1187,9 +1165,9 @@ impl Omnigraph { params: &ParamMap, branch: Option<&str>, staging: &mut MutationStaging, - txn: Option<&crate::db::WriteTxn>, + txn: &crate::db::WriteTxn, ) -> Result { - let is_node = self.catalog().node_types.contains_key(type_name); + let is_node = txn.catalog.node_types.contains_key(type_name); if is_node { self.execute_delete_node(type_name, predicate, params, branch, staging, txn) .await @@ -1206,7 +1184,7 @@ impl Omnigraph { params: &ParamMap, branch: Option<&str>, staging: &mut MutationStaging, - txn: Option<&crate::db::WriteTxn>, + txn: &crate::db::WriteTxn, ) -> Result { let pred_sql = predicate_to_sql(predicate, params, false)?; @@ -1217,7 +1195,7 @@ impl Omnigraph { branch, &table_key, crate::db::MutationOpKind::Delete, - txn, + Some(txn), ) .await?; // Delete is a STRICT op, so collapse #1 never skips its open. @@ -1256,7 +1234,9 @@ impl Omnigraph { // HEAD only at the unified end-of-query commit — no inline residual. // `open_table_for_mutation` above already captured the table's // path/version/op-kind via `ensure_path`. - crate::failpoints::maybe_fail(crate::failpoints::names::MUTATION_DELETE_NODE_PRE_PRIMARY_DELETE)?; + crate::failpoints::maybe_fail( + crate::failpoints::names::MUTATION_DELETE_NODE_PRE_PRIMARY_DELETE, + )?; staging.record_deleted_ids(&table_key, &deleted_ids); staging.record_delete(&table_key, pred_sql.clone()); @@ -1267,8 +1247,8 @@ impl Omnigraph { .collect(); let id_list = escaped.join(", "); - let edge_info: Vec<(String, String, String)> = self - .catalog() + let edge_info: Vec<(String, String, String)> = txn + .catalog .edge_types .iter() .map(|(name, et)| (name.clone(), et.from_type.clone(), et.to_type.clone())) @@ -1294,7 +1274,7 @@ impl Omnigraph { branch, &edge_table_key, crate::db::MutationOpKind::Delete, - txn, + Some(txn), ) .await?; // Delete is a STRICT op, so collapse #1 never skips its open. @@ -1310,8 +1290,10 @@ impl Omnigraph { // by both a cascade and an explicit `delete ` — is counted // once. Record the ORIGINAL cascade filter (the combined staged // delete removes the union); skip only when nothing NEW matches. - let count_filter = - dedup_delete_filter(&cascade_filter, staging.recorded_delete_predicates(&edge_table_key)); + let count_filter = dedup_delete_filter( + &cascade_filter, + staging.recorded_delete_predicates(&edge_table_key), + ); // Scan (not count) the cascade-removed edge ids so validation // recounts the OTHER endpoint's @card after the cascade; `len()` is // the affected count. @@ -1347,7 +1329,7 @@ impl Omnigraph { params: &ParamMap, branch: Option<&str>, staging: &mut MutationStaging, - txn: Option<&crate::db::WriteTxn>, + txn: &crate::db::WriteTxn, ) -> Result { let pred_sql = predicate_to_sql(predicate, params, true)?; @@ -1358,7 +1340,7 @@ impl Omnigraph { branch, &table_key, crate::db::MutationOpKind::Delete, - txn, + Some(txn), ) .await?; // Delete is a STRICT op, so collapse #1 never skips its open. @@ -1420,23 +1402,20 @@ fn ids_from_batches(batches: &[RecordBatch]) -> Vec { /// `scan_with_pending` returns committed-side and pending-side batches in /// order; both should share a schema if pending was produced through /// `apply_assignments` with full-schema scan input. If schemas drift, -/// surface a clear error so the user can split the query. +/// surface the internal contract failure at the mutation boundary. fn concat_match_batches_to_schema( - _schema: &SchemaRef, - _blob_properties: &HashSet, + schema: &SchemaRef, batches: Vec, ) -> Result { if batches.len() == 1 { - return Ok(batches.into_iter().next().unwrap()); + let batch = batches.into_iter().next().unwrap(); + return RecordBatch::try_new(schema.clone(), batch.columns().to_vec()) + .map_err(|e| OmniError::Lance(e.to_string())); } - let common = batches[0].schema(); - arrow_select::concat::concat_batches(&common, &batches).map_err(|e| { + arrow_select::concat::concat_batches(schema, &batches).map_err(|e| { OmniError::Lance(format!( - "scan_with_pending returned batches with mismatched schemas \ - across the committed/pending boundary; this typically indicates \ - a blob-column shape mismatch between the committed table and a \ - prior in-query insert/update. Split blob-touching mutations \ - into separate queries. ({})", + "mutation scan returned batches that violate the full logical schema \ + across the committed/pending boundary ({})", e )) }) diff --git a/crates/omnigraph/src/exec/staging.rs b/crates/omnigraph/src/exec/staging.rs index 2f865f37..b74962d3 100644 --- a/crates/omnigraph/src/exec/staging.rs +++ b/crates/omnigraph/src/exec/staging.rs @@ -1,10 +1,11 @@ //! Per-query staging accumulator for direct-publish writes. //! //! `MutationStaging` accumulates per-table input batches in memory during a -//! `mutate_as` or `load` query, then at end-of-query commits each touched -//! table via Lance's distributed-write API (one `stage_*` + `commit_staged` -//! per table) and returns the publisher inputs (`SubTableUpdate` list + -//! `expected_table_versions`). +//! `mutate_as` or `load` query. At end-of-query it prepares one staged Lance +//! transaction per touched existing ref (or a deferred first-touch plan), then +//! joins the RFC-022 protocol: acquire ordered gates, revalidate the complete +//! read set, arm durable recovery, apply the physical effects, and return the +//! exact publisher inputs while retaining the guards through manifest CAS. //! //! Read-your-writes within the same query is satisfied by the in-memory //! pending batches (see `pending_batches`) — read sites union the committed @@ -13,7 +14,8 @@ //! //! This module is shared by the engine's mutation path (`exec/mutation.rs`) //! and the bulk loader (`loader/mod.rs`); both feed insert/update batches -//! into `pending` and route end-of-query commits through `finalize`. +//! into `pending` and route end-of-query work through `stage_all` then +//! `commit_all`, followed by one exact manifest publish. //! Deletes accumulate as predicates in `delete_predicates` (via //! `record_delete`) and stage through the same `stage_* → commit_staged` //! path as writes — `stage_delete` produces a deletion-vector transaction @@ -30,7 +32,8 @@ use arrow_schema::SchemaRef; use futures::stream::StreamExt; use crate::db::manifest::{ - RecoverySidecarHandle, SidecarKind, SidecarTablePin, new_sidecar, write_sidecar, + RecoveryAuthorityToken, RecoveryLineageIntent, RecoverySidecarHandle, SidecarKind, + SidecarTablePin, confirm_occ_sidecar_phase_b, new_occ_sidecar, write_sidecar, }; use crate::db::{MutationOpKind, SubTableUpdate}; use crate::error::{OmniError, Result}; @@ -44,6 +47,20 @@ pub(crate) enum PendingMode { Overwrite, } +/// Work that must be staged only after a first-touch target ref exists. Lance +/// writes uncommitted fragment/deletion files under the opened branch's tree; +/// staging against the inherited source and later committing on the target +/// would publish paths that do not exist in the target tree. +enum DeferredStagePlan { + Pending { + mode: PendingMode, + batch: RecordBatch, + }, + Delete { + predicate: String, + }, +} + /// Per-table accumulator. Each insert/update op pushes a `RecordBatch` into /// `batches`; at end-of-query the accumulated batches concat into a single /// stage call. @@ -73,16 +90,21 @@ impl PendingTable { #[derive(Debug, Clone)] pub(crate) struct StagedTablePath { pub(crate) full_path: String, + /// Final Lance ref recorded in the manifest update. pub(crate) table_branch: Option, + /// First-touch named-branch fork deferred until the v3 recovery sidecar is + /// durable. Preparation reads the inherited `source_entry`; after arming, + /// commit creates `target_branch` and stages branch-local files there. + pub(crate) deferred_fork: Option, } /// Per-query staging state. /// -/// Replaces the legacy inline-commit `MutationStaging.latest` map with -/// an in-memory accumulator that defers all Lance HEAD advances to -/// end-of-query. Inserts, updates AND deletes all stage here, so the bug -/// class "Lance HEAD drifts ahead of `__manifest`" is unreachable in -/// `mutate_as` and `load` by construction. +/// Replaces the legacy per-statement inline-commit map with an in-memory +/// accumulator that defers every Lance HEAD advance to the query commit +/// boundary. Inserts, updates, and deletes share one effect set. Durable +/// recovery intent owns the remaining effect-to-manifest-CAS gap; statement +/// failures cannot create uncovered HEAD drift. #[derive(Default)] pub(crate) struct MutationStaging { /// Pre-write manifest version per table — the publisher's CAS fence at @@ -126,6 +148,7 @@ impl MutationStaging { table_key: &str, full_path: String, table_branch: Option, + deferred_fork: Option, expected_version: u64, op_kind: MutationOpKind, ) { @@ -134,6 +157,7 @@ impl MutationStaging { .or_insert(StagedTablePath { full_path, table_branch, + deferred_fork, }); self.expected_versions .entry(table_key.to_string()) @@ -351,7 +375,7 @@ impl MutationStaging { delete_predicates, // Validation-only; consumed before staging, nothing to commit here. deleted_ids: _, - op_kinds, + op_kinds: _, } = self; let mut stage_inputs: Vec<(String, PendingTable, StagedTablePath, u64)> = @@ -430,7 +454,6 @@ impl MutationStaging { Ok(StagedMutation { staged: staged_entries, expected_versions, - op_kinds, }) } } @@ -442,23 +465,27 @@ async fn stage_pending_table( path: StagedTablePath, expected: u64, ) -> Result> { - // Reopen the dataset for staging. Append/Merge can be rebased later by - // Lance + publisher CAS; Overwrite is a strict replacement and uses the - // same SchemaRewrite policy as schema apply. + // Reopen the pinned dataset. Existing-table effects stage on this handle + // now. A deferred first-touch effect uses it only as the inherited source + // pin; its files stage on the target handle after sidecar + fork. let stage_kind = match table.mode { PendingMode::Append => crate::db::MutationOpKind::Insert, PendingMode::Merge => crate::db::MutationOpKind::Merge, PendingMode::Overwrite => crate::db::MutationOpKind::SchemaRewrite, }; - let ds = db - .reopen_for_mutation( - &table_key, - &path.full_path, - path.table_branch.as_deref(), - expected, - stage_kind, - ) - .await?; + let ds = match path.deferred_fork.as_ref() { + Some(fork) => db.storage().open_snapshot_at_entry(&fork.source_entry).await?, + None => { + db.reopen_for_mutation( + &table_key, + &path.full_path, + path.table_branch.as_deref(), + expected, + stage_kind, + ) + .await? + } + }; if table.batches.is_empty() { return Ok(None); @@ -475,6 +502,25 @@ async fn stage_pending_table( } } }; + if path.deferred_fork.is_some() { + // The recovery identity is minted before the fork; the actual Lance + // transaction is staged on the new target ref after the sidecar is + // durable, then bound to this UUID. Its read version remains Lance's + // independently-derived value and is checked by the binder. + let planned_transaction = pre_minted_transaction_identity(expected); + return Ok(Some(StagedTableEntry { + table_key, + path, + expected_version: expected, + dataset: ds, + staged_write: None, + deferred_stage: Some(DeferredStagePlan::Pending { + mode: table.mode, + batch: combined, + }), + planned_transaction, + })); + } // Stage produces uncommitted fragments + transaction. No Lance HEAD // advance until `commit_all` runs `commit_staged`. @@ -493,12 +539,15 @@ async fn stage_pending_table( } PendingMode::Overwrite => db.storage().stage_overwrite(&ds, combined).await?, }; + let planned_transaction = staged.transaction_identity(); Ok(Some(StagedTableEntry { table_key, path, expected_version: expected, dataset: ds, - staged_write: staged, + staged_write: Some(staged), + deferred_stage: None, + planned_transaction, })) } @@ -517,136 +566,192 @@ async fn stage_delete_table( path: StagedTablePath, expected: u64, ) -> Result> { - let ds = db - .reopen_for_mutation( - &table_key, - &path.full_path, - path.table_branch.as_deref(), - expected, - crate::db::MutationOpKind::Delete, - ) - .await?; + let ds = match path.deferred_fork.as_ref() { + Some(fork) => db.storage().open_snapshot_at_entry(&fork.source_entry).await?, + None => { + db.reopen_for_mutation( + &table_key, + &path.full_path, + path.table_branch.as_deref(), + expected, + crate::db::MutationOpKind::Delete, + ) + .await? + } + }; + if path.deferred_fork.is_some() { + // Probe only. The actual deletion vector must be written under the + // target branch tree after the durable fork intent creates that ref. + if db + .storage() + .first_row_id_for_filter(&ds, &predicate) + .await? + .is_none() + { + return Ok(None); + } + return Ok(Some(StagedTableEntry { + table_key, + path, + expected_version: expected, + dataset: ds, + staged_write: None, + deferred_stage: Some(DeferredStagePlan::Delete { predicate }), + planned_transaction: pre_minted_transaction_identity(expected), + })); + } match db.storage().stage_delete(&ds, &predicate).await? { Some(staged) => Ok(Some(StagedTableEntry { table_key, path, expected_version: expected, dataset: ds, - staged_write: staged, + planned_transaction: staged.transaction_identity(), + staged_write: Some(staged), + deferred_stage: None, })), None => Ok(None), } } -/// Output of [`MutationStaging::stage_all`]. Carries the staged Lance -/// transactions (Phase A complete; uncommitted fragments written) plus -/// the per-table metadata needed to write the recovery sidecar, run -/// `commit_staged` (Phase B), and produce the publisher's input. +/// Output of [`MutationStaging::stage_all`]. Carries ready Lance transactions +/// for existing refs and complete deferred stage plans for first-touch named +/// refs, plus the metadata needed to arm recovery, apply Stage-F effects, and +/// produce the publisher's input. /// -/// Splitting `stage_all` and `commit_all` is the structural prerequisite -/// for MR-686: a future commit can drop queue acquisition + manifest-pin -/// revalidation between Phase A and Phase B without touching staging -/// logic. +/// Splitting `stage_all` and `commit_all` keeps reclaimable preparation outside +/// writer gates while the latter owns ordered acquisition, revalidation, +/// recovery arming, and effects. pub(crate) struct StagedMutation { /// One entry per table that had pending write batches or delete - /// predicates successfully staged (Phase A complete, no HEAD advance). + /// predicates prepared (and, for an existing ref, successfully staged). /// Deletes flow through this same vector as inserts/updates/overwrites — /// there is no separate inline-commit path. staged: Vec, /// Pre-write manifest version per table — the publisher's CAS fence. expected_versions: HashMap, - /// Strictest op_kind per touched table, propagated from - /// `MutationStaging::op_kinds` so `commit_all`'s drift check - /// fires only on read-modify-write tables. - op_kinds: HashMap, } /// Per-table state captured during `stage_all` and consumed by -/// `commit_all`. Holds the opened snapshot (so `commit_staged` doesn't -/// re-open) plus the staged Lance transaction that `commit_staged` -/// will execute. Both held as opaque `TableStorage` handles per MR-793 -/// §III.9 — the inner `lance::Dataset` / `StagedWrite` are not visible -/// to engine code outside the storage layer. +/// `commit_all`. Holds the opened snapshot plus either a staged Lance +/// transaction or a deferred first-touch plan. Storage handles remain opaque +/// per MR-793 §III.9 — the inner `lance::Dataset` / `StagedWrite` are not +/// visible to engine code outside the storage layer. struct StagedTableEntry { table_key: String, path: StagedTablePath, expected_version: u64, dataset: SnapshotHandle, - staged_write: StagedHandle, + staged_write: Option, + deferred_stage: Option, + planned_transaction: crate::table_store::StagedTransactionIdentity, } -/// Output of [`StagedMutation::commit_all`] (Phase B): the publisher's input plus -/// the queue guards the caller must hold across the manifest publish. +fn pre_minted_transaction_identity( + read_version: u64, +) -> crate::table_store::StagedTransactionIdentity { + crate::table_store::StagedTransactionIdentity { + read_version, + // Lance treats the UUID as an opaque transaction identity/path + // component. A ULID is equally unique and filesystem-safe while + // avoiding a second UUID generator in the engine surface. + uuid: format!("omnigraph-{}", ulid::Ulid::new()), + } +} + +async fn stage_deferred_plan( + db: &crate::db::Omnigraph, + target: SnapshotHandle, + plan: DeferredStagePlan, + planned: &crate::table_store::StagedTransactionIdentity, +) -> Result { + let mut staged = match plan { + DeferredStagePlan::Pending { mode, batch } => match mode { + PendingMode::Append => db.storage().stage_append(&target, batch, &[]).await?, + PendingMode::Merge => { + db.storage() + .stage_merge_insert( + target, + batch, + vec!["id".to_string()], + lance::dataset::WhenMatched::UpdateAll, + lance::dataset::WhenNotMatched::InsertAll, + ) + .await? + } + PendingMode::Overwrite => db.storage().stage_overwrite(&target, batch).await?, + }, + DeferredStagePlan::Delete { predicate } => db + .storage() + .stage_delete(&target, &predicate) + .await? + .ok_or_else(|| { + OmniError::manifest_read_set_changed( + "deferred_delete_match", + Some("matching row at inherited pin".to_string()), + Some("no matching row on exact target fork".to_string()), + ) + })?, + }; + staged.bind_transaction_identity(planned)?; + Ok(staged) +} + +/// Output of [`StagedMutation::commit_all`] after Stage F: the publisher's input +/// plus the queue guards the caller must hold through Stage G manifest publish. pub(crate) struct CommittedMutation { /// Per-table updates to publish to the manifest. pub(crate) updates: Vec, - /// Per-table manifest pins refreshed under the write queue — the publisher's CAS fence. + /// Per-table physical pins captured in the immutable `WriteTxn`; the + /// publisher checks them together with native branch identity, exact graph + /// head, and schema identity as one authority precondition. pub(crate) expected_versions: HashMap, - /// Recovery sidecar to delete after Phase C succeeds (`None` when nothing staged). + /// Recovery sidecar to delete during Stage H after manifest CAS succeeds + /// (`None` when nothing staged). pub(crate) sidecar_handle: Option, - /// Per-`(table, branch)` write-queue guards — the caller MUST hold these across - /// the manifest publish (see `commit_all`) so no writer interleaves between - /// `commit_staged` and the publish. + /// Root schema, coarse branch, and sorted `(table, branch)` guards. The + /// caller MUST hold the complete set across manifest publish (see + /// `commit_all`) so no same-process writer interleaves after revalidation. pub(crate) guards: Vec>, - /// Post-`commit_staged` handle per STAGED table (table_key → handle at the - /// just-committed version). Carried out (RFC-013 step 3b, collapse #4) so the - /// publish-prepare index build reuses it instead of a fresh `reopen_for_mutation` - /// at the same version. Deletes are staged too, so their committed handle is - /// present here like any other write. - pub(crate) committed_handles: HashMap, } impl StagedMutation { - /// **Phase B** of the two-phase commit: acquire per-`(table_key, - /// branch)` queues, revalidate manifest pins, write the recovery - /// sidecar, run `commit_staged` per table to advance Lance HEAD, and - /// return the publisher's input plus the queue guards. + /// RFC-022 Stages C–F: acquire ordered schema/branch/table gates, revalidate + /// the complete read set, arm the recovery sidecar, run `commit_staged` per + /// table to advance Lance HEAD, and return the publisher input plus guards. /// /// **Caller must hold the returned `_guards` Vec across the /// subsequent manifest publish.** Releasing guards before publish - /// would let another writer interleave their commit_staged between - /// ours and our publish — which would correctly fail our CAS but - /// leave Lance HEAD advanced (the residual class MR-870 recovers - /// from). Holding the guards across publish keeps the residual - /// unreachable for op-execution failures on the happy path. + /// would let another writer interleave its physical effects between ours + /// and publish. The exact publisher precondition would still reject stale + /// authority, but our already-advanced Lance HEAD would then require the + /// durable v3 sidecar to recover. Holding the guards closes that avoidable + /// same-process window; the sidecar remains the crash/foreign-process + /// correctness authority. /// - /// Revalidation: between `stage_all` and `commit_all`, another - /// writer (in the same process or another process sharing the - /// graph) may have committed to one of our touched tables, advancing - /// the manifest pin past our `expected_version`. We revalidate - /// under the queue and fail-fast with `manifest_conflict` before - /// any `commit_staged` so the orphaned uncommitted fragments stay - /// unreferenced (cleaned by `cleanup_old_versions`'s age sweep) - /// rather than being committed and creating a Lance-HEAD-ahead - /// residual. - /// `held_guards`: when the caller already holds the per-`(table_key, - /// branch)` write queues for every touched table (the fork path acquires - /// them up front, before the fork, and holds them through the manifest - /// publish), it passes `(acquired_keys, guards)` here so `commit_all` - /// reuses them instead of re-acquiring — the queue is a non-re-entrant - /// `tokio::Mutex`, so re-acquiring a held key would self-deadlock. - /// `None` (the steady-state path) means `commit_all` acquires them - /// itself. `acquired_keys` must cover every key `commit_all` would - /// acquire (debug-asserted below) — the guards from `acquire_many` don't - /// carry their keys, so the caller hands the key set alongside them. The - /// fork path guarantees coverage by keying every touched table uniformly - /// by the resolved target branch. + /// Revalidation: between `stage_all` and `commit_all`, another writer may + /// change any member of the attempt's branch-wide read authority, including + /// a table this attempt does not write. Under the ordered gates we first + /// reject any newly armed overlapping sidecar, then compare the complete + /// immutable token. A safe insert/Append/Merge caller discards and fully + /// reprepares; strict Update/Delete/Overwrite returns `ReadSetChanged`. + /// Both outcomes occur before `commit_staged`, so staged transaction files + /// remain unreferenced and reclaimable. + /// First-touch named-branch forks are also deferred to this phase. The + /// method acquires schema → branch → table gates, revalidates the complete + /// read token, arms v3 recovery, and only then creates any Lance refs. pub(crate) async fn commit_all( self, db: &crate::db::Omnigraph, branch: Option<&str>, sidecar_kind: SidecarKind, actor_id: Option<&str>, - held_guards: Option<( - Vec<(String, Option)>, - Vec>, - )>, - txn: Option<&crate::db::WriteTxn>, + txn: &crate::db::WriteTxn, + lineage_intent: &crate::db::manifest::LineageIntent, ) -> Result { let StagedMutation { mut staged, - mut expected_versions, - op_kinds, + expected_versions, } = self; // Per-(table_key, branch) queues for every touched table. Sorted by @@ -661,81 +766,63 @@ impl StagedMutation { for entry in &staged { queue_keys.push((entry.table_key.clone(), entry.path.table_branch.clone())); } - // Reuse the caller's guards (fork path) when handed in, else acquire - // our own. When reusing, every key we would acquire MUST already be - // covered — re-acquiring a held non-re-entrant key would deadlock, and - // a key we'd need but DON'T hold would commit unserialized. This is a - // load-bearing safety invariant, so it is checked in ALL builds (not a - // debug_assert) and fails the write loudly+safely rather than silently - // proceeding unguarded if a future execution path ever touches a table - // outside the caller's pre-computed set. - let guards = match held_guards { - Some((acquired_keys, guards)) => { - let held: std::collections::HashSet<&(String, Option)> = - acquired_keys.iter().collect(); - if let Some(missing) = queue_keys.iter().find(|k| !held.contains(k)) { - return Err(OmniError::manifest_internal(format!( - "commit_all: pre-held write-queue guards do not cover touched table \ - '{}' on branch {:?} — the caller's up-front acquisition set diverged \ - from the staged/inline set (a touched-table-set bug)", - missing.0, missing.1 - ))); - } - guards - } - None => db.write_queue().acquire_many(&queue_keys).await, - }; + // Total order shared with schema apply: schema gate, branch gate, then + // sorted per-table gates. Hold the full set through manifest publish. + let schema_guard = db + .write_queue() + .acquire(&crate::db::manifest::schema_apply_serial_queue_key()) + .await; + let branch_guard = db.write_queue().acquire_branch(branch).await; + let mut guards = vec![schema_guard, branch_guard]; + guards.extend(db.write_queue().acquire_many(&queue_keys).await); + + // Stage A ran the roll-forward-only healer before preparation, while + // reclaimable transaction files were staged outside these gates. A + // different writer can therefore arm a recovery intent in that gap. + // Re-list after acquiring schema -> branch -> table gates and reject + // that intent before revalidation or any new sidecar/ref/table effect. + // Do NOT invoke the healer here: recovery acquires this same ordered + // gate set and would deadlock/re-enter the write protocol. + // + // OCC is branch-wide, so every unresolved sidecar for this normalized + // graph branch is relevant even when it names a disjoint table. Schema + // apply changes the catalog contract and conservatively blocks every + // enrolled data branch. `list_sidecars` is URI-sorted; choosing the + // first relevant operation makes conflict attribution deterministic. + let enrolled_branch = branch.filter(|name| *name != "main"); + let pending_sidecars = + crate::db::manifest::list_sidecars(db.root_uri(), db.storage_adapter()).await?; + if let Some(owner) = pending_sidecars.iter().find(|sidecar| { + let sidecar_branch = sidecar.branch.as_deref().filter(|name| *name != "main"); + sidecar.writer_kind == SidecarKind::SchemaApply || sidecar_branch == enrolled_branch + }) { + return Err(OmniError::recovery_required( + owner.operation_id.clone(), + format!( + "pending {:?} recovery operation blocks writes on branch '{}'", + owner.writer_kind, + enrolled_branch.unwrap_or("main"), + ), + )); + } // Re-capture manifest pins under the queue (PR 2 / MR-686). // // expected_versions was captured when the mutation first opened // each table for mutation (the query's read-time pin). For - // non-strict inserts / merge-style appends, a writer may advance - // the table before we acquire the queue and Lance can still - // safely rebase the write, so we refresh expected_versions to - // the queued manifest pin. + // Any writer may advance the branch while we stage reclaimable files + // outside the gate. RFC-022 does not patch those prepared table pins or + // transparently rebase a validated transaction: every mismatch rejects + // the complete attempt before effects. Mutation inserts and load + // append/merge may then reprepare at their outer bounded retry loop; + // strict read-modify-write operations return the typed conflict. // - // Strict read-modify-write ops (update / delete / - // schema-rewrite) are different: the staged batch was computed - // against the read-time pin, even if stage_all later re-opened - // the dataset at HEAD. For those ops, compare read-time - // expected_version to the queued manifest pin and fail before - // any Lance HEAD movement if the target drifted. This can - // over-reject a single mutation that inserts, then upgrades to - // update, while another writer advances the table between the - // two touches; that is safe-by-default and keeps one invariant - // until `ensure_path` learns how to bump expected_version on - // op-kind upgrade. - // - // Why a fresh per-branch snapshot (and not the bound-branch - // `db.snapshot()` / `snapshot_for_branch()` fast path): a stale - // engine handle may be bound to the same branch it is writing. For - // non-strict Insert/Merge, that stale local view is allowed to rebase - // to the live manifest pin under the queue; only uncovered Lance - // HEAD>manifest drift is refused. For writes targeting a branch other - // than the engine's bound branch (e.g., feature-branch ingest from a - // server handle bound to main), the same helper also resolves the - // correct branch pin. The cost is one fresh manifest read per mutation - // plus one Lance HEAD open per staged table for the drift guard below. - // - // Multi-coordinator deployments (§VI.27 aspirational) get - // genuine cross-process drift detection from this read for - // free. - // - // This MUST be as fresh as a FRESH per-branch manifest read for the OCC - // re-capture below. With a `WriteTxn` the schema contract was already - // validated at capture, so the txn arm uses `occ_snapshot_for_branch` - // (RFC-013 PR2 #1a): a cheap incarnation probe reuses the warm - // coordinator when it is already current (probe-match ⟺ warm == fresh) - // and falls through to the cold `_unchecked` read on any mismatch — so - // freshness AND cross-process drift detection are preserved while the - // common same-branch sequential write skips the O(fragments) scan. - // Without a txn, keep the checked cold read (legacy path). - let snapshot = match txn { - Some(_) => db.occ_snapshot_for_branch(branch).await?, - None => db.fresh_snapshot_for_branch(branch).await?, - }; - for entry in staged.iter_mut() { + // Revalidate the complete coarse authority under the branch/table + // gates. The helper probe-reuses a current warm coordinator and opens + // the target branch fresh on any mismatch, returning the snapshot from + // that same authority view. No prepared table pin is patched forward. + let snapshot = db.revalidate_write_txn(txn).await?; + for entry in &staged { let current = snapshot .entry(&entry.table_key) .map(|e| e.table_version) @@ -746,28 +833,40 @@ impl StagedMutation { )) })?; - // Insert / Merge tables skip this check: concurrent inserts on - // disjoint keys legitimately coexist via Lance's auto-rebase, so - // the check would over-reject the existing Phase 2 same-key - // insert path (`change_concurrent_inserts_same_key_serialize_without_409`). - let strict = op_kinds - .get(&entry.table_key) - .map(|k| k.strict_pre_stage_version_check()) - .unwrap_or(false); - if strict && entry.expected_version != current { - return Err(OmniError::manifest_expected_version_mismatch( - entry.table_key.clone(), - entry.expected_version, - current, + // RFC-022-enrolled mutation/load attempts never patch a prepared + // table expectation to a newer manifest pin. The complete branch + // token was just revalidated; any remaining table mismatch is a + // stale/internally-inconsistent read set and must restart the + // whole validation attempt before effects. + if entry.expected_version != current { + return Err(OmniError::manifest_read_set_changed( + format!("table_head:{}", entry.table_key), + Some(entry.expected_version.to_string()), + Some(current.to_string()), )); } + // A deferred fork is intentionally staged from the exact inherited + // source entry. The source ref (often main) may advance after the + // graph branch was cut; that is unrelated to this branch's pinned + // snapshot. The target ref does not exist yet, so there is no live + // target HEAD to compare until after the recovery intent is armed. + if entry.path.deferred_fork.is_some() { + if entry.dataset.version() != current { + return Err(OmniError::manifest_read_set_changed( + format!("table_head:{}", entry.table_key), + Some(current.to_string()), + Some(entry.dataset.version().to_string()), + )); + } + continue; + } + // Separate manifest-visible concurrency from uncovered Lance drift. - // Non-strict inserts/merges are allowed to rebase from their staged - // read version to the fresh manifest pin above, but only if the - // live Lance HEAD still equals that manifest pin. If an external - // raw Lance write or a pre-fix maintenance path moved HEAD without - // publishing `__manifest`, this write must not silently fold it. + // The exact table pin already matched above; now verify that the + // live Lance HEAD also equals that pin. If an external raw Lance + // write or a pre-fix maintenance path moved HEAD without publishing + // `__manifest`, this write must not silently fold it. // // `latest_version_id` reads the latest manifest pointer off the // already-open staged handle (the #2 staging open) WITHOUT a fresh @@ -789,59 +888,78 @@ impl StagedMutation { ))); } if head > current { - // Error path only: tell the operator which drift class - // this is. Uncovered drift (external raw Lance write, - // pre-fix maintenance) goes through `omnigraph repair`. - // Sidecar-covered drift reaching this guard means the - // write-entry heal deferred it (rollback-eligible), and - // `repair` refuses while a sidecar is pending — the - // recovery path is a read-write reopen. A list failure - // must not mask the conflict — and must not pick a - // class confidently either: "could not classify" names - // both paths and the cause, never routing the operator - // to a command that will refuse. - let action = match crate::db::manifest::list_sidecars( + // Error path only: attribute covered drift to the exact + // recovery operation that owns it. The entry-point heal ran + // before this attempt was prepared, so another writer can arm + // and confirm a sidecar while this attempt stages reclaimable + // files outside the gates. Calling the healer here would be a + // lock-order violation: recovery acquires the same + // schema -> branch -> table gates we currently hold. + // + // `list_sidecars` returns URI-sorted sidecars. Use the first + // matching table/ref claim so attribution stays deterministic + // even if malformed/concurrent state contains more than one + // claimant. Uncovered drift (external raw Lance write or a + // pre-fix maintenance path) remains a generic conflict routed + // through `omnigraph repair`. + let sidecars = crate::db::manifest::list_sidecars( db.root_uri(), db.storage_adapter(), ) - .await - { + .await; + match sidecars { Ok(sidecars) => { - let covered = sidecars.iter().any(|sidecar| { + if let Some(owner) = sidecars.iter().find(|sidecar| { sidecar.tables.iter().any(|pin| { - // Branch-aware: a sidecar pinning the - // same table on ANOTHER branch does not - // cover this branch's drift — a reopen - // would recover that sidecar but leave - // this drift for `repair`. + // Branch-aware: a sidecar pinning the same + // table on another branch does not own this + // branch's drift. pin.table_key == entry.table_key && pin.table_branch == entry.path.table_branch }) - }); - if covered { - "a pending recovery sidecar requires rollback — reopen the \ - graph read-write (e.g. restart the server) to recover" - .to_string() - } else { - "run `omnigraph repair` before writing".to_string() + }) { + return Err(OmniError::recovery_required( + owner.operation_id.clone(), + format!( + "table '{}' has Lance HEAD version {} ahead of manifest \ + version {}; the pending recovery operation owns this drift", + entry.table_key, head, current, + ), + )); } + return Err(OmniError::manifest_conflict(format!( + "table '{}' has Lance HEAD version {} ahead of manifest version {}; \ + run `omnigraph repair` before writing", + entry.table_key, head, current, + ))); } - Err(list_err) => format!( - "could not classify the drift (sidecar listing failed: {}); \ - run `omnigraph repair`, or reopen the graph read-write if \ - repair reports a pending recovery sidecar", - list_err - ), - }; - return Err(OmniError::manifest_conflict(format!( - "table '{}' has Lance HEAD version {} ahead of manifest version {}; {}", - entry.table_key, head, current, action - ))); + Err(list_err) => { + let action = format!( + "could not classify the drift (sidecar listing failed: {}); \ + run `omnigraph repair`, or reopen the graph read-write if \ + repair reports a pending recovery sidecar", + list_err + ); + return Err(OmniError::manifest_conflict(format!( + "table '{}' has Lance HEAD version {} ahead of manifest version {}; {}", + entry.table_key, head, current, action + ))); + } + } } - - entry.expected_version = current; - expected_versions.insert(entry.table_key.clone(), current); } + // An empty load/mutation has no independently durable table effect. + // It may still publish its fixed lineage intent, but arming recovery + // would manufacture an effect plan where none exists. + if staged.is_empty() { + return Ok(CommittedMutation { + updates: Vec::new(), + expected_versions, + sidecar_handle: None, + guards, + }); + } + // Sidecar protocol: build the per-table pin list and write the // sidecar BEFORE any `commit_staged` advances Lance HEAD, so any // commit→publish residual is recoverable on the next open. Deletes @@ -849,38 +967,137 @@ impl StagedMutation { // `staged` entry here — one pin at `expected + 1` (a single staged // commit advances exactly one version), no inline special-casing. let mut pins: Vec = Vec::with_capacity(staged.len()); + let mut planned_transactions = HashMap::with_capacity(staged.len()); for entry in &staged { + planned_transactions.insert( + entry.table_key.clone(), + entry.planned_transaction.clone(), + ); pins.push(SidecarTablePin { table_key: entry.table_key.clone(), table_path: entry.path.full_path.clone(), expected_version: entry.expected_version, post_commit_pin: entry.expected_version + 1, - // Mutation/Load use strict single-commit classification, not - // BranchMerge's Phase-B confirmation — left None. confirmed_version: None, table_branch: entry.path.table_branch.clone(), }); } - let sidecar_handle = if pins.is_empty() { - None - } else { - let sidecar = new_sidecar( - sidecar_kind, - branch.map(|s| s.to_string()), - actor_id.map(str::to_string), - pins, - ); - Some(write_sidecar(db.root_uri(), db.storage_adapter(), &sidecar).await?) + let authority = RecoveryAuthorityToken { + branch_identifier: txn.authority.branch_identifier.clone(), + graph_head: txn.authority.graph_head.clone(), + schema_ir_hash: txn.authority.schema_ir_hash.clone(), + schema_identity_version: txn.authority.schema_identity_version, }; + let recovery_lineage = RecoveryLineageIntent { + graph_commit_id: lineage_intent.graph_commit_id.clone(), + branch: lineage_intent.branch.clone(), + actor_id: lineage_intent.actor_id.clone(), + merged_parent_commit_id: lineage_intent.merged_parent_commit_id.clone(), + created_at: lineage_intent.created_at, + }; + let mut sidecar = new_occ_sidecar( + sidecar_kind, + branch.map(str::to_string), + actor_id.map(str::to_string), + pins, + authority, + recovery_lineage, + planned_transactions, + )?; + // Deterministic pre-effect race point: authority has been validated and + // the intent is fully prepared in memory, but neither the durable + // sidecar nor a target fork exists yet. A concurrent winner may publish; + // this attempt then discards/reprepares on the collision below. + crate::failpoints::maybe_fail(crate::failpoints::names::FORK_BEFORE_CLASSIFY)?; + let sidecar_handle = Some( + write_sidecar(db.root_uri(), db.storage_adapter(), &sidecar).await?, + ); + let operation_id = sidecar.operation_id.clone(); + if staged.iter().any(|entry| entry.path.deferred_fork.is_some()) { + crate::failpoints::maybe_fail( + crate::failpoints::names::MUTATION_POST_SIDECAR_PRE_FORK, + ) + .map_err(|error| { + OmniError::recovery_required(operation_id.clone(), error.to_string()) + })?; + } + + // The v3 intent is now durable. Only now may a first-touch named-table + // fork become visible in Lance. Its transaction is then staged on the + // fresh target handle (so Lance writes branch-local file paths) and + // bound to the pre-minted UUID already recorded in the sidecar. + let mut created_any_fork = false; + for entry in &mut staged { + let Some(fork) = entry.path.deferred_fork.clone() else { + continue; + }; + match db + .fork_dataset_from_entry_state_under_intent( + &entry.table_key, + &entry.path.full_path, + fork.source_entry.table_branch.as_deref(), + fork.source_entry.table_version, + &fork.target_branch, + Some(&operation_id), + ) + .await + { + Ok(target) => { + entry.dataset = target; + created_any_fork = true; + let plan = entry.deferred_stage.take().ok_or_else(|| { + OmniError::manifest_internal(format!( + "deferred fork for '{}' has no deferred stage plan", + entry.table_key + )) + })?; + let staged_write = stage_deferred_plan( + db, + entry.dataset.clone(), + plan, + &entry.planned_transaction, + ) + .await + .map_err(|error| { + OmniError::recovery_required( + operation_id.clone(), + format!( + "staging on deferred table fork '{}' failed after recovery \ + intent was armed: {error}", + entry.table_key + ), + ) + })?; + entry.staged_write = Some(staged_write); + } + Err(error) => { + // Once the intent is durable, a fork error is not proof that + // no physical effect occurred. Lance may have created the ref + // successfully and then failed while reopening/verifying it; + // reclaim-and-refork can likewise fail after deleting or + // recreating a ref. Keep the sidecar for exact Full recovery + // even when this is the first table. A future typed storage + // outcome may recover the pre-effect retry optimization only + // when it can *prove* the ref was never changed. + return Err(OmniError::recovery_required( + operation_id, + format!("deferred table fork failed after recovery intent was armed: {error}"), + )); + } + } + } + if created_any_fork { + crate::failpoints::maybe_fail( + crate::failpoints::names::MUTATION_POST_FORK_PRE_COMMIT, + ) + .map_err(|error| { + OmniError::recovery_required(operation_id.clone(), error.to_string()) + })?; + } let mut updates: Vec = Vec::with_capacity(staged.len()); - - // Carry each staged table's post-`commit_staged` handle out so the - // publish-prepare index build reuses it (collapse #4) instead of - // re-opening the dataset at the same just-committed version. - let mut committed_handles: HashMap = - HashMap::with_capacity(staged.len()); + let mut committed_transactions = HashMap::with_capacity(staged.len()); for entry in staged { let StagedTableEntry { @@ -889,10 +1106,48 @@ impl StagedMutation { expected_version: _, dataset, staged_write, + deferred_stage: _, + planned_transaction: _, } = entry; - let new_ds = db.storage().commit_staged(dataset, staged_write).await?; - let state = db.storage().table_state(&path.full_path, &new_ds).await?; + let staged_write = staged_write.ok_or_else(|| { + OmniError::manifest_internal(format!( + "table '{}' reached commit without a staged transaction", + table_key + )) + })?; + + let outcome = db + .storage() + .commit_staged_exact(dataset, staged_write) + .await + .map_err(|error| { + OmniError::recovery_required(operation_id.clone(), error.to_string()) + })?; + if !outcome.is_exact() { + return Err(OmniError::recovery_required( + operation_id, + format!( + "table '{}' committed a rebased Lance transaction at version {} (planned {:?}, committed {:?})", + table_key, + outcome.committed_version(), + outcome.planned_transaction(), + outcome.committed_transaction(), + ), + )); + } + committed_transactions.insert( + table_key.clone(), + outcome.committed_transaction().clone(), + ); + let new_ds = outcome.into_snapshot(); + let state = db + .storage() + .table_state(&path.full_path, &new_ds) + .await + .map_err(|error| { + OmniError::recovery_required(operation_id.clone(), error.to_string()) + })?; updates.push(SubTableUpdate { table_key: table_key.clone(), table_version: state.version, @@ -900,7 +1155,21 @@ impl StagedMutation { row_count: state.row_count, version_metadata: state.version_metadata, }); - committed_handles.insert(table_key, new_ds); + } + + if let Err(error) = confirm_occ_sidecar_phase_b( + db.root_uri(), + db.storage_adapter(), + &mut sidecar, + &updates, + &committed_transactions, + ) + .await + { + return Err(OmniError::recovery_required( + operation_id, + error.to_string(), + )); } Ok(CommittedMutation { @@ -908,7 +1177,6 @@ impl StagedMutation { expected_versions, sidecar_handle, guards, - committed_handles, }) } } diff --git a/crates/omnigraph/src/failpoints.rs b/crates/omnigraph/src/failpoints.rs index ee1281a6..e8db9129 100644 --- a/crates/omnigraph/src/failpoints.rs +++ b/crates/omnigraph/src/failpoints.rs @@ -39,31 +39,87 @@ pub(crate) fn maybe_fail_retryable_contention(name: &str) -> Result<()> { /// compile error rather than a silently-never-firing failpoint. pub mod names { pub const BRANCH_DELETE_BEFORE_TABLE_CLEANUP: &str = "branch_delete.before_table_cleanup"; - pub const BRANCH_MERGE_ADOPT_AFTER_APPEND_PRE_UPSERT: &str = "branch_merge.adopt_after_append_pre_upsert"; - pub const BRANCH_MERGE_ADOPT_AFTER_UPSERT_PRE_DELETE: &str = "branch_merge.adopt_after_upsert_pre_delete"; - pub const BRANCH_MERGE_POST_PHASE_B_PRE_MANIFEST_COMMIT: &str = "branch_merge.post_phase_b_pre_manifest_commit"; - pub const BRANCH_MERGE_REWRITE_AFTER_DELETE_PRE_INDEX: &str = "branch_merge.rewrite_after_delete_pre_index"; - pub const BRANCH_MERGE_REWRITE_AFTER_MERGE_PRE_DELETE: &str = "branch_merge.rewrite_after_merge_pre_delete"; + /// Branch delete holds the schema, target-branch, and fresh-catalog table + /// envelope and has completed its final recovery check, before the native + /// manifest-ref mutation. + pub const BRANCH_DELETE_POST_TABLE_GATES: &str = "branch_delete.post_table_gates"; + /// After native branch control completed its first recovery barrier, before + /// it acquires schema -> branch -> table gates and performs the final check. + pub const BRANCH_CONTROL_POST_RECOVERY_BARRIER: &str = + "branch_control.post_recovery_barrier"; + pub const BRANCH_MERGE_ADOPT_AFTER_APPEND_PRE_UPSERT: &str = + "branch_merge.adopt_after_append_pre_upsert"; + pub const BRANCH_MERGE_ADOPT_AFTER_UPSERT_PRE_DELETE: &str = + "branch_merge.adopt_after_upsert_pre_delete"; + /// Source/target heads and snapshots have been captured while the schema + /// and both branch-incarnation gates are held, before merge planning or + /// any durable table effect. + pub const BRANCH_MERGE_POST_AUTHORITY_CAPTURE: &str = + "branch_merge.post_authority_capture"; + pub const BRANCH_MERGE_POST_PHASE_B_PRE_MANIFEST_COMMIT: &str = + "branch_merge.post_phase_b_pre_manifest_commit"; + /// Every merge table effect is complete, but the sidecar is still in its + /// pre-confirmation shape. + pub const BRANCH_MERGE_POST_EFFECTS_PRE_CONFIRM: &str = + "branch_merge.post_effects_pre_confirm"; + pub const BRANCH_MERGE_REWRITE_AFTER_DELETE_PRE_INDEX: &str = + "branch_merge.rewrite_after_delete_pre_index"; + pub const BRANCH_MERGE_REWRITE_AFTER_MERGE_PRE_DELETE: &str = + "branch_merge.rewrite_after_merge_pre_delete"; pub const CLASSIFY_FRESH_READ: &str = "classify.fresh_read"; pub const CLEANUP_RECONCILE_FORK: &str = "cleanup.reconcile_fork"; + /// After cleanup's fast empty-sidecar probe, before it acquires the closed + /// schema/branch/table GC gate set and performs the authoritative recheck. + pub const CLEANUP_POST_RECOVERY_CHECK_PRE_GATES: &str = + "cleanup.post_recovery_check_pre_gates"; pub const CLEANUP_RESOLVE_BRANCH_SNAPSHOT: &str = "cleanup.resolve_branch_snapshot"; pub const CLEANUP_TABLE_GC: &str = "cleanup.table_gc"; - pub const ENSURE_INDICES_POST_PHASE_B_PRE_MANIFEST_COMMIT: &str = "ensure_indices.post_phase_b_pre_manifest_commit"; - pub const ENSURE_INDICES_POST_STAGE_PRE_COMMIT_BTREE: &str = "ensure_indices.post_stage_pre_commit_btree"; + pub const ENSURE_INDICES_POST_PHASE_B_PRE_MANIFEST_COMMIT: &str = + "ensure_indices.post_phase_b_pre_manifest_commit"; + pub const ENSURE_INDICES_POST_STAGE_PRE_COMMIT_BTREE: &str = + "ensure_indices.post_stage_pre_commit_btree"; pub const FORK_BEFORE_CLASSIFY: &str = "fork.before_classify"; pub const FORK_BEFORE_RECLAIM: &str = "fork.before_reclaim"; + /// After Lance durably creates a target table ref, before the caller can + /// reopen and verify it. An error here is post-effect and must retain the + /// recovery sidecar. + pub const FORK_POST_CREATE_PRE_OPEN: &str = "fork.post_create_pre_open"; pub const GRAPH_PUBLISH_AFTER_MANIFEST_COMMIT: &str = "graph_publish.after_manifest_commit"; pub const GRAPH_PUBLISH_BEFORE_COMMIT_APPEND: &str = "graph_publish.before_commit_append"; pub const INIT_AFTER_COORDINATOR_INIT: &str = "init.after_coordinator_init"; pub const INIT_AFTER_SCHEMA_CONTRACT_WRITTEN: &str = "init.after_schema_contract_written"; pub const INIT_AFTER_SCHEMA_PG_WRITTEN: &str = "init.after_schema_pg_written"; - pub const MUTATION_DELETE_NODE_PRE_PRIMARY_DELETE: &str = "mutation.delete_node_pre_primary_delete"; + pub const MUTATION_DELETE_NODE_PRE_PRIMARY_DELETE: &str = + "mutation.delete_node_pre_primary_delete"; + /// After every deferred first-touch table ref is created under a durable + /// v3 sidecar, before any staged data transaction advances target HEAD. + pub const MUTATION_POST_FORK_PRE_COMMIT: &str = "mutation.post_fork_pre_commit"; + /// After the v3 ownership sidecar is durable but before the first deferred + /// named-table ref is created. Recovery must accept the absent target ref. + pub const MUTATION_POST_SIDECAR_PRE_FORK: &str = "mutation.post_sidecar_pre_fork"; + /// Deterministic OCC rendezvous after a mutation has validated and staged + /// its complete attempt, but before the RFC-022 branch effect gate is + /// acquired and the write authority token is revalidated. Tests park the + /// first writer here, commit a conflicting second writer, then prove the + /// first attempt is discarded and validation is rerun from a fresh token. + pub const MUTATION_POST_STAGE_PRE_EFFECT_GATE: &str = "mutation.post_stage_pre_effect_gate"; pub const MUTATION_POST_FINALIZE_PRE_PUBLISHER: &str = "mutation.post_finalize_pre_publisher"; + /// Open owns the schema gate and is about to read source/IR/state as one + /// catalog view. + pub const OPEN_BEFORE_SCHEMA_CONTRACT_READ: &str = "open.before_schema_contract_read"; pub const OPTIMIZE_BEFORE_COMPACT: &str = "optimize.before_compact"; pub const OPTIMIZE_INJECT_REINDEX_CONFLICT: &str = "optimize.inject_reindex_conflict"; - pub const OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT: &str = "optimize.post_phase_b_pre_manifest_commit"; + pub const OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT: &str = + "optimize.post_phase_b_pre_manifest_commit"; pub const RECOVERY_BEFORE_ROLL_FORWARD_PUBLISH: &str = "recovery.before_roll_forward_publish"; + /// Recovery has listed/parsed its discovery snapshot but has not yet taken + /// per-sidecar gates. Tests rewrite confirmation state in this window. + pub const RECOVERY_POST_LIST_PRE_GATES: &str = "recovery.post_list_pre_gates"; pub const RECOVERY_ORPHAN_DISCARD_AUDIT_APPEND: &str = "recovery.orphan_discard_audit_append"; + /// After the fixed rollback lineage/table-pin publish is durable, before + /// its operator-facing audit row is appended. + pub const RECOVERY_POST_ROLLBACK_PUBLISH_PRE_AUDIT: &str = + "recovery.post_rollback_publish_pre_audit"; pub const RECOVERY_RECORD_AUDIT: &str = "recovery.record_audit"; pub const RECOVERY_SIDECAR_CONFIRM: &str = "recovery.sidecar_confirm"; pub const RECOVERY_SIDECAR_DELETE: &str = "recovery.sidecar_delete"; @@ -72,6 +128,9 @@ pub mod names { pub const SCHEMA_APPLY_AFTER_MANIFEST_COMMIT: &str = "schema_apply.after_manifest_commit"; pub const SCHEMA_APPLY_AFTER_STAGING_WRITE: &str = "schema_apply.after_staging_write"; pub const SCHEMA_APPLY_BEFORE_STAGING_WRITE: &str = "schema_apply.before_staging_write"; + /// Reload owns the schema gate and is about to read/publish one contract view. + pub const SCHEMA_RELOAD_BEFORE_CONTRACT_READ: &str = + "schema_reload.before_contract_read"; /// Injects a retryable `RowLevelCasContention` from `load_publish_state` so a /// test can prove the publisher's outer retry re-runs the load. pub const PUBLISH_LOAD_STATE_RETRYABLE_CONTENTION: &str = diff --git a/crates/omnigraph/src/instrumentation.rs b/crates/omnigraph/src/instrumentation.rs index f4ef9ef9..d235038e 100644 --- a/crates/omnigraph/src/instrumentation.rs +++ b/crates/omnigraph/src/instrumentation.rs @@ -313,6 +313,8 @@ pub struct StorageReadCounts { pub exists: AtomicU64, pub read_text_versioned: AtomicU64, pub list_dir: AtomicU64, + pub write_text: AtomicU64, + pub delete: AtomicU64, } impl StorageReadCounts { @@ -328,6 +330,12 @@ impl StorageReadCounts { pub fn list_dir(&self) -> u64 { self.list_dir.load(Ordering::Relaxed) } + pub fn write_text(&self) -> u64 { + self.write_text.load(Ordering::Relaxed) + } + pub fn delete(&self) -> u64 { + self.delete.load(Ordering::Relaxed) + } } /// Boundary decorator over a [`StorageAdapter`] that counts read-facing calls. @@ -360,6 +368,7 @@ impl StorageAdapter for CountingStorageAdapter { } async fn write_text(&self, uri: &str, contents: &str) -> Result<()> { + self.counts.write_text.fetch_add(1, Ordering::Relaxed); self.inner.write_text(uri, contents).await } @@ -377,6 +386,7 @@ impl StorageAdapter for CountingStorageAdapter { } async fn delete(&self, uri: &str) -> Result<()> { + self.counts.delete.fetch_add(1, Ordering::Relaxed); self.inner.delete(uri).await } diff --git a/crates/omnigraph/src/loader/mod.rs b/crates/omnigraph/src/loader/mod.rs index d0ab9ecb..e6639d29 100644 --- a/crates/omnigraph/src/loader/mod.rs +++ b/crates/omnigraph/src/loader/mod.rs @@ -187,17 +187,6 @@ impl Omnigraph { &omnigraph_policy::ResourceScope::Branch(branch.to_string()), actor_id, )?; - // Schema-contract validation is captured ONCE per write via the - // `WriteTxn` opened in `load_jsonl_reader` (after branch resolution). - // The redundant `ensure_schema_state_valid` that used to run here is - // subsumed by `open_write_txn`'s `resolved_branch_target` call. - // Converge any pending recovery sidecar (a previously failed - // writer's Phase B → Phase C residual) before staging anything: - // without this, sidecar-covered drift wedges every load on the - // commit-time drift guard until a process restart — `repair` - // refuses while a sidecar is pending. One `list_dir` when no - // sidecars exist (the steady state). - self.heal_pending_recovery_sidecars().await?; // Reject internal `__run__*` / system-prefixed branches at the // public write boundary. Direct-publish paths assert this // explicitly so a caller can't write to legacy or system @@ -215,6 +204,23 @@ impl Omnigraph { } None => None, }; + // Schema/catalog authority is captured once via the `WriteTxn` (plus its + // cheap trailing identity-marker fence); the only second full validation + // is the required pre-effect recheck under gates. Per-table resolution + // performs no additional contract reads. + // + // Stage A precedes both an implicit target-branch fork and data staging. + // The target branch and an explicit base are read/write authority for the + // operation, so an unresolved intent on either closes the barrier. The + // helper folds `Some("main")` to main's canonical `None` identity. + let mut recovery_branches = vec![requested.as_deref()]; + if base.is_some() { + // `base_branch` retains `Some("main")` for the result DTO; the + // barrier helper canonicalizes it to main's `None` identity. + recovery_branches.push(base_branch.as_deref()); + } + self.heal_pending_recovery_sidecars_for_write(&recovery_branches) + .await?; // Fork-if-missing only when a base branch was explicitly given. // `requested == None` is `main`, which always exists. let mut branch_created = false; @@ -238,7 +244,7 @@ impl Omnigraph { } // Direct-to-target writes: no Run state machine, no `__run__` staging // branch. Cross-table OCC is enforced by the publisher's - // `expected_table_versions` CAS inside `load_jsonl_reader`. + // `expected_table_versions` CAS inside the load attempt. let mut result = self .load_direct_on_branch(requested.as_deref(), data, mode, actor_id) .await?; @@ -274,8 +280,7 @@ impl Omnigraph { mode: LoadMode, actor_id: Option<&str>, ) -> Result { - let reader = BufReader::new(Cursor::new(data.as_bytes())); - load_jsonl_reader(self, branch, reader, mode, actor_id).await + load_jsonl_data(self, branch, data, mode, actor_id).await } } @@ -312,14 +317,55 @@ impl LoadResult { } } -async fn load_jsonl_reader( +async fn load_jsonl_data( + db: &Omnigraph, + branch: Option<&str>, + data: &str, + mode: LoadMode, + actor_id: Option<&str>, +) -> Result { + const MAX_PRE_EFFECT_REPREPARES: usize = 32; + + // Every public load entry point already owns a stable `&str` payload + // (`load_file_as` reads its file once). Replay that slice directly on a + // pre-effect retry; copying it into a second raw byte buffer would double + // peak input memory before the parser's per-type materialization. + let retryable = matches!(mode, LoadMode::Append | LoadMode::Merge); + for attempt in 0..=MAX_PRE_EFFECT_REPREPARES { + let replay = BufReader::new(Cursor::new(data.as_bytes())); + match load_jsonl_reader_once(db, branch, replay, mode, actor_id).await { + Err(err) + if retryable + && err.is_read_set_changed() + && attempt < MAX_PRE_EFFECT_REPREPARES => + { + tracing::debug!( + attempt = attempt + 1, + branch = branch.unwrap_or("main"), + "prepared load authority changed before effects; repreparing" + ); + db.refresh().await?; + } + result => return result, + } + } + unreachable!("bounded load retry loop always returns") +} + +async fn load_jsonl_reader_once( db: &Omnigraph, branch: Option<&str>, reader: R, mode: LoadMode, actor_id: Option<&str>, ) -> Result { - let catalog = db.catalog().clone(); + // Capture the manifest/schema authority before interpreting any input. The + // catalog rides the WriteTxn and was built from the exact accepted IR named + // by its schema token; a long-lived handle's global catalog may legitimately + // lag a schema apply completed through another handle. + let txn = db.open_write_txn(branch).await?; + let catalog = Arc::clone(&txn.catalog); + let snapshot = txn.base.clone(); // Phase 1: Parse all lines, spool into per-type collections let mut node_rows: HashMap> = HashMap::new(); @@ -400,16 +446,10 @@ async fn load_jsonl_reader( // inline path. let mut result = LoadResult::default(); - // Capture-once write transaction (RFC-013 step 3b). `open_write_txn` - // validates the schema contract ONCE and pins the base snapshot. Threaded - // as `Some(&txn)` through the per-table opens and the manifest publish so - // each resolve point reuses the pinned base instead of re-validating the - // contract. The branch already exists here (fork-if-missing ran in - // `load_as` before this), so this captures the post-fork snapshot. The - // load's own base read (`db.snapshot_for_branch` previously) is the same - // per-branch snapshot, so reuse `txn.base` for it — dropping a validation. - let txn = db.open_write_txn(branch).await?; - let snapshot = txn.base.clone(); + // The branch-wide WriteTxn captured above is threaded through every table + // open and the manifest publish, so the parsed batches, validation catalog, + // base snapshot, native branch identity, exact graph head, and schema + // identity form one immutable authority unit. let mut staging = MutationStaging::default(); let pending_mode = match mode { LoadMode::Merge => PendingMode::Merge, @@ -419,56 +459,17 @@ async fn load_jsonl_reader( LoadMode::Append => PendingMode::Append, LoadMode::Overwrite => PendingMode::Overwrite, }; - // Map LoadMode to MutationOpKind for the version-check policy. - // Append/Merge skip the strict pre-stage check (concurrency-safe - // under the per-(table, branch) queue + publisher CAS); Overwrite - // uses the strict check because it truncates and replaces the - // dataset — concurrent advances change what "replace" means. + // Map LoadMode to the early table-version policy. Append/Merge may stage + // reclaimable files before the effect gates, then revalidate the complete + // branch token and fully reprepare on a bounded pre-effect conflict. + // Overwrite keeps the strict early check because it replaces the image; its + // later branch-wide mismatch surfaces `ReadSetChanged` without replay. let load_op_kind = match mode { LoadMode::Append => crate::db::MutationOpKind::Insert, LoadMode::Merge => crate::db::MutationOpKind::Merge, LoadMode::Overwrite => crate::db::MutationOpKind::SchemaRewrite, }; - // Up-front fork-queue acquisition. The first write to a table on a - // non-main branch forks it (create_branch), which advances Lance state - // before the manifest publish; the reclaim of any manifest-unreferenced - // leftover (`reclaim_orphaned_fork_and_refork`) must not race a concurrent - // in-process fork. So when this load will fork at least one touched table, - // acquire the per-(table, branch) write queues for ALL touched tables up - // front (one sorted `acquire_many`, keyed uniformly by the target branch - // so it covers what `commit_all` recomputes) and hold them through the - // publish. Main-branch loads never fork; branch loads where every touched - // table is already forked skip this and let `commit_all` acquire at commit. - let fork_queue_guards: Option<( - Vec<(String, Option)>, - Vec>, - )> = if let Some(active) = branch { - let touched: Vec<(String, Option)> = node_rows - .keys() - .map(|t| (format!("node:{t}"), Some(active.to_string()))) - .chain( - edge_rows - .keys() - .map(|e| (format!("edge:{e}"), Some(active.to_string()))), - ) - .collect(); - let needs_fork = touched.iter().any(|(table_key, _)| { - snapshot - .entry(table_key) - .map(|e| e.table_branch.as_deref() != Some(active)) - .unwrap_or(false) - }); - if needs_fork { - let guards = db.write_queue().acquire_many(&touched).await; - Some((touched, guards)) - } else { - None - } - } else { - None - }; - // Phase 2a: build and validate every node batch up front. Cheap and // synchronous — surfaces validation errors before any S3 traffic. let mut prepared_nodes: Vec<(String, String, RecordBatch, usize)> = @@ -499,6 +500,7 @@ async fn load_jsonl_reader( &table_key, opened.full_path, opened.table_branch, + opened.deferred_fork, opened.expected_version, load_op_kind, ); @@ -534,6 +536,7 @@ async fn load_jsonl_reader( &table_key, opened.full_path, opened.table_branch, + opened.deferred_fork, opened.expected_version, load_op_kind, ); @@ -581,43 +584,60 @@ async fn load_jsonl_reader( let staged = staging .stage_all_with_concurrency(db, branch, load_write_concurrency()) .await?; - // `_queue_guards` holds per-(table_key, branch) write queues - // across the manifest publish below — see exec/mutation.rs for - // the rationale (interleaving prevention). + crate::failpoints::maybe_fail(crate::failpoints::names::MUTATION_POST_STAGE_PRE_EFFECT_GATE)?; + let lineage_intent = db.new_lineage_intent_for_branch(branch, actor_id).await?; + // `_queue_guards` holds the root-shared schema → branch → sorted-table + // gates across manifest publication. This closes same-process + // interleaving across the v3 sidecar/effect lifetime. The exact publisher + // token and durable sidecar remain persistent correctness authorities, but + // these local gates do not expand the documented single-writer-process + // recovery boundary. let crate::exec::staging::CommittedMutation { updates, expected_versions, sidecar_handle, guards: _queue_guards, - committed_handles, } = staged .commit_all( db, branch, crate::db::manifest::SidecarKind::Load, actor_id, - fork_queue_guards, - Some(&txn), + &txn, + &lineage_intent, ) .await?; - // Same finalize → publisher residual as mutations: per-table - // staged commits have advanced Lance HEAD, but the manifest - // publish has not run yet. Reuse the mutation failpoint name so - // one failpoint pins the shared `MutationStaging` boundary. + // Same confirmed-effects → publisher boundary as mutations: table HEADs + // have advanced and the v3 sidecar contains their exact transaction + // identities, but the graph manifest has not published the result. Reuse + // the mutation failpoint name so one failpoint pins the shared boundary. crate::failpoints::maybe_fail(crate::failpoints::names::MUTATION_POST_FINALIZE_PRE_PUBLISHER)?; - db.commit_updates_on_branch_with_expected( - branch, - &updates, - &expected_versions, - actor_id, - Some(&txn), - committed_handles, - ) - .await?; - // The recovery sidecar protects the per-table commit_staged → - // manifest publish window. Phase C succeeded — clean up - // best-effort: failing the user here would error out a write - // that already landed durably. + let publish_result = db + .commit_updates_on_branch_with_expected( + branch, + &updates, + &expected_versions, + actor_id, + &txn, + lineage_intent, + ) + .await; + if let Err(err) = publish_result { + // Empty loads can still publish lineage but have no table effect and + // therefore no recovery sidecar. Preserve that publish error instead + // of manufacturing an "unknown" recovery operation. + return match sidecar_handle.as_ref() { + Some(handle) => Err(OmniError::recovery_required( + handle.operation_id.clone(), + err.to_string(), + )), + None => Err(err), + }; + } + // The v3 recovery sidecar protects every independently durable table effect + // through the one manifest visibility point. Phase C succeeded — clean up + // best-effort: failing the user here would error out a write that already + // landed durably; a leftover fixed outcome is idempotently finalized later. if let Some(handle) = sidecar_handle { if let Err(err) = crate::db::manifest::delete_sidecar(&handle, db.storage_adapter()).await { tracing::warn!( @@ -1843,7 +1863,10 @@ edge WorksAt: Person -> Company let result = db .load_as("nonexistent", None, TEST_DATA, LoadMode::Merge, None) .await; - assert!(result.is_err(), "load without base must not create branches"); + assert!( + result.is_err(), + "load without base must not create branches" + ); assert!( !db.branch_list() .await @@ -1853,7 +1876,10 @@ edge WorksAt: Person -> Company ); // Loads to main carry the default branch metadata. - let main_load = db.load("main", TEST_DATA, LoadMode::Overwrite).await.unwrap(); + let main_load = db + .load("main", TEST_DATA, LoadMode::Overwrite) + .await + .unwrap(); assert_eq!(main_load.branch, "main"); assert_eq!(main_load.base_branch, None); assert!(!main_load.branch_created); diff --git a/crates/omnigraph/src/storage.rs b/crates/omnigraph/src/storage.rs index 357f9906..90ec2ebf 100644 --- a/crates/omnigraph/src/storage.rs +++ b/crates/omnigraph/src/storage.rs @@ -502,6 +502,54 @@ pub fn normalize_root_uri(uri: &str) -> Result { } } +/// Process-local identity used to share writer queues across handles for the +/// same graph root. +/// +/// The storage URI remains unchanged: this identity is used only as the key in +/// the in-process queue registry. Local paths are first made absolute +/// lexically, then the deepest canonicalizable ancestor is resolved so aliases +/// through symlinks converge. Any suffix that does not exist yet is appended +/// unchanged, which makes the identity safe to compute before `init` creates +/// the graph directory. Object-store and caller-defined URI schemes are opaque +/// and retain their normalized spelling. +pub(crate) fn write_queue_root_identity(normalized_root: &str) -> Result { + let local_path = if normalized_root.starts_with(FILE_SCHEME_PREFIX) { + local_path_from_file_uri(normalized_root)? + } else if Path::new(normalized_root).is_absolute() { + PathBuf::from(normalized_root) + } else if has_uri_scheme(normalized_root) { + return Ok(normalized_root.to_string()); + } else { + PathBuf::from(normalized_root) + }; + + let absolute = absolutize_lexically(local_path)?; + let mut ancestor = absolute.as_path(); + let mut suffix = Vec::new(); + + loop { + if let Ok(canonical) = std::fs::canonicalize(ancestor) { + let mut identity = canonical; + for component in suffix.iter().rev() { + identity.push(component); + } + return Ok(normalize_local_path(&identity)); + } + + let Some(name) = ancestor.file_name() else { + // Filesystem roots should always canonicalize. Falling back to the + // lexical absolute path preserves queue availability on unusual + // platforms/filesystems without changing the storage root. + return Ok(normalize_local_path(&absolute)); + }; + suffix.push(name.to_os_string()); + let Some(parent) = ancestor.parent() else { + return Ok(normalize_local_path(&absolute)); + }; + ancestor = parent; + } +} + pub fn join_uri(root_uri: &str, relative_path: &str) -> String { let relative_path = relative_path.trim_start_matches('/'); match storage_kind_for_uri(root_uri) { @@ -534,6 +582,18 @@ fn local_path_from_uri(uri: &str) -> Result { Ok(PathBuf::from(uri)) } +fn has_uri_scheme(value: &str) -> bool { + let Some(colon) = value.find(':') else { + return false; + }; + let scheme = &value[..colon]; + !scheme.is_empty() + && scheme.as_bytes()[0].is_ascii_alphabetic() + && scheme + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'+' | b'-' | b'.')) +} + /// Lexically absolutize a local path: join relative paths onto the current /// working directory and fold `.` / `..` components, without touching the /// filesystem. Required because `object_store::path::Path` rejects @@ -908,6 +968,17 @@ mod tests { ); } + #[test] + fn write_queue_identity_keeps_remote_and_custom_schemes_opaque() { + for root in [ + "s3://bucket/prefix", + "memory://write-queue/custom", + "custom+transport:opaque-root", + ] { + assert_eq!(write_queue_root_identity(root).unwrap(), root); + } + } + #[test] fn join_uri_handles_local_file_and_s3_roots() { assert_eq!( diff --git a/crates/omnigraph/src/storage_layer.rs b/crates/omnigraph/src/storage_layer.rs index 250937ac..665500bf 100644 --- a/crates/omnigraph/src/storage_layer.rs +++ b/crates/omnigraph/src/storage_layer.rs @@ -66,7 +66,7 @@ use lance::dataset::{WhenMatched, WhenNotMatched}; use crate::db::{Snapshot, SubTableEntry}; use crate::error::Result; -use crate::table_store::{StagedWrite, TableState, TableStore}; +use crate::table_store::{StagedTransactionIdentity, StagedWrite, TableState, TableStore}; // ─── sealed module ────────────────────────────────────────────────────────── @@ -174,6 +174,59 @@ impl StagedHandle { pub(crate) fn into_staged(self) -> StagedWrite { self.inner } + + /// Lance transaction identity captured when this effect was staged. + pub fn transaction_identity(&self) -> StagedTransactionIdentity { + self.inner.transaction_identity() + } + + /// Replace Lance's random transaction UUID with the identity durably armed + /// before a deferred first-touch fork. The read version must still match. + pub(crate) fn bind_transaction_identity( + &mut self, + planned: &StagedTransactionIdentity, + ) -> Result<()> { + self.inner.bind_transaction_identity(planned) + } +} + +/// Result of the no-conflict-retry commit path used by RFC-022-enrolled +/// writers. `is_exact` checks both transaction identity and achieved version: +/// Lance's initial conflict-resolution pass can preserve `(read_version, uuid)` +/// while committing at a later version. The table effect is durable when that +/// happens, so the caller must leave its recovery sidecar armed. +#[derive(Debug)] +pub struct ExactCommitOutcome { + snapshot: SnapshotHandle, + planned_transaction: StagedTransactionIdentity, + committed_transaction: StagedTransactionIdentity, +} + +impl ExactCommitOutcome { + pub fn is_exact(&self) -> bool { + self.planned_transaction == self.committed_transaction + && self.snapshot.version() == self.planned_transaction.read_version + 1 + } + + pub fn planned_transaction(&self) -> &StagedTransactionIdentity { + &self.planned_transaction + } + + pub fn committed_transaction(&self) -> &StagedTransactionIdentity { + &self.committed_transaction + } + + pub fn committed_version(&self) -> u64 { + self.snapshot.version() + } + + pub fn snapshot(&self) -> &SnapshotHandle { + &self.snapshot + } + + pub fn into_snapshot(self) -> SnapshotHandle { + self.snapshot + } } /// Helper: clone the inner `StagedWrite` out of each `StagedHandle` and @@ -324,6 +377,20 @@ pub trait TableStorage: sealed::Sealed + Send + Sync + Debug { key_column: Option<&str>, ) -> Result>; + /// Full-schema blob-aware sibling of `scan_with_pending` for mutation + /// updates. The committed predicate scan retains row ids without projecting + /// blobs; only matched rows are then taken and rebuilt as Lance's logical + /// blob input arrays before unioning the in-memory pending view. This keeps + /// the eventual merge source schema independent of scalar-index state. + async fn scan_with_pending_materialized_blobs( + &self, + snapshot: &SnapshotHandle, + pending: &[RecordBatch], + pending_schema: Option, + filter: Option<&str>, + key_column: Option<&str>, + ) -> Result>; + async fn first_row_id_for_filter( &self, snapshot: &SnapshotHandle, @@ -366,6 +433,15 @@ pub trait TableStorage: sealed::Sealed + Send + Sync + Debug { staged: StagedHandle, ) -> Result; + /// Commit one staged effect with Lance conflict retries disabled and expose + /// the transaction identity that actually landed. Legacy callers retain + /// `commit_staged`; RFC-022 adapters opt into this method explicitly. + async fn commit_staged_exact( + &self, + snapshot: SnapshotHandle, + staged: StagedHandle, + ) -> Result; + /// Stage an overwrite (Operation::Overwrite). MR-793 Phase 2. async fn stage_overwrite( &self, @@ -636,6 +712,25 @@ impl TableStorage for TableStore { .await } + async fn scan_with_pending_materialized_blobs( + &self, + snapshot: &SnapshotHandle, + pending: &[RecordBatch], + pending_schema: Option, + filter: Option<&str>, + key_column: Option<&str>, + ) -> Result> { + TableStore::scan_with_pending_materialized_blobs( + self, + snapshot.dataset(), + pending, + pending_schema, + filter, + key_column, + ) + .await + } + async fn first_row_id_for_filter( &self, snapshot: &SnapshotHandle, @@ -701,6 +796,22 @@ impl TableStorage for TableStore { .map(SnapshotHandle::new) } + async fn commit_staged_exact( + &self, + snapshot: SnapshotHandle, + staged: StagedHandle, + ) -> Result { + let planned_transaction = staged.transaction_identity(); + let ds_arc = snapshot.into_arc(); + let (dataset, committed_transaction) = + TableStore::commit_staged_exact(self, ds_arc, staged.into_staged()).await?; + Ok(ExactCommitOutcome { + snapshot: SnapshotHandle::new(dataset), + planned_transaction, + committed_transaction, + }) + } + async fn stage_overwrite( &self, snapshot: &SnapshotHandle, diff --git a/crates/omnigraph/src/table_store.rs b/crates/omnigraph/src/table_store.rs index d34cb899..513daf62 100644 --- a/crates/omnigraph/src/table_store.rs +++ b/crates/omnigraph/src/table_store.rs @@ -16,13 +16,14 @@ use lance::dataset::{ use lance::datatypes::{BlobKind, Schema as LanceSchema}; use lance::index::DatasetIndexExt; use lance::index::scalar::IndexDetails; -use lance_select::mask::RowAddrTreeMap; use lance_file::version::LanceFileVersion; use lance_index::scalar::{InvertedIndexParams, ScalarIndexParams}; use lance_index::{IndexType, is_system_index}; use lance_linalg::distance::MetricType; +use lance_select::mask::RowAddrTreeMap; use lance_table::format::{Fragment, IndexMetadata, RowIdMeta}; use lance_table::rowids::{RowIdSequence, write_row_ids}; +use serde::{Deserialize, Serialize}; use std::sync::Arc; use crate::db::manifest::TableVersionMetadata; @@ -50,6 +51,30 @@ pub enum IndexCoverage { Degraded { reason: String }, } +/// Stable identity of one Lance transaction. +/// +/// Lance persists both fields in the transaction file referenced by the +/// committed manifest. Recovery uses the pair, rather than a numeric table +/// version alone, to prove that an observed HEAD was produced by the staged +/// effect named in a recovery sidecar. The UUID distinguishes two writers that +/// started from the same version. Lance may preserve both fields while +/// rebasing, so enrolled callers must also require the achieved table version +/// to be exactly `read_version + 1`. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct StagedTransactionIdentity { + pub read_version: u64, + pub uuid: String, +} + +impl From<&Transaction> for StagedTransactionIdentity { + fn from(transaction: &Transaction) -> Self { + Self { + read_version: transaction.read_version, + uuid: transaction.uuid.clone(), + } + } +} + /// A Lance write that has produced fragment files on object storage but is /// not yet committed to the dataset's manifest. The staged-write primitives /// are consumed by `MutationStaging` (`exec/staging.rs`, @@ -137,6 +162,30 @@ impl StagedWrite { pub fn removed_fragment_ids(&self) -> &[u64] { &self.removed_fragment_ids } + + /// Identity Lance assigned when this effect was staged. + pub fn transaction_identity(&self) -> StagedTransactionIdentity { + StagedTransactionIdentity::from(&self.transaction) + } + + /// Bind a pre-minted recovery identity to a transaction staged after a + /// deferred branch fork. The operation and read version still come from + /// Lance; only its otherwise-random UUID is replaced so the sidecar can be + /// durable before the target ref (and its branch-local fragment paths) + /// exist. + pub(crate) fn bind_transaction_identity( + &mut self, + planned: &StagedTransactionIdentity, + ) -> Result<()> { + if self.transaction.read_version != planned.read_version { + return Err(OmniError::manifest_internal(format!( + "staged transaction read version {} does not match pre-minted recovery pin {}", + self.transaction.read_version, planned.read_version + ))); + } + self.transaction.uuid.clone_from(&planned.uuid); + Ok(()) + } } #[derive(Debug, Clone)] @@ -376,6 +425,11 @@ impl TableStore { return Err(OmniError::Lance(create_err.to_string())); } + // The ref is now independently durable. Any error from this point is an + // ambiguous/post-effect outcome to the caller and must retain an armed + // recovery intent rather than being treated as a safe pre-effect retry. + crate::failpoints::maybe_fail(crate::failpoints::names::FORK_POST_CREATE_PRE_OPEN)?; + let ds = self .open_dataset_head(dataset_uri, Some(target_branch)) .await?; @@ -446,6 +500,30 @@ impl TableStore { .copied() .collect::>(); + Self::materialize_blob_batch_with_row_ids(ds, batch, &row_ids).await + } + + /// Rebuild the blob columns in `batch` using explicit stable row ids. + /// + /// Most rewrite callers scan with `_rowid` and use + /// [`Self::materialize_blob_batch`]. A predicate-filtered blob mutation + /// cannot include blob descriptors in that scan on the pinned Lance + /// revision (the filter projection panics), so it first scans only + /// non-blob columns + `_rowid`, takes the full descriptor rows by id, and + /// calls this sibling with the ids captured by the safe scan. + async fn materialize_blob_batch_with_row_ids( + ds: &Dataset, + batch: RecordBatch, + row_ids: &[u64], + ) -> Result { + if batch.num_rows() != row_ids.len() { + return Err(OmniError::Lance(format!( + "blob materialization row count {} does not match {} row ids", + batch.num_rows(), + row_ids.len() + ))); + } + let schema: SchemaRef = Arc::new(ds.schema().into()); let mut columns = Vec::with_capacity(schema.fields().len()); for field in schema.fields() { @@ -1263,6 +1341,44 @@ impl TableStore { /// the publisher at end-of-query to materialize all staged writes before /// the meta-manifest commit. pub async fn commit_staged(&self, ds: Arc, staged: StagedWrite) -> Result { + self.commit_staged_with_retry_budget(ds, staged, None) + .await + .map(|(dataset, _)| dataset) + } + + /// Commit an RFC-022-enrolled staged effect with no commit-conflict retry. + /// + /// `CommitBuilder::with_max_retries(0)` gives Lance one commit attempt. It + /// can still perform its initial conflict-resolution pass before that + /// attempt, so this method also reads back and returns the identity of the + /// transaction that actually landed. Callers must compare it with + /// [`StagedWrite::transaction_identity`] AND require the returned dataset + /// version to equal `read_version + 1`: Lance's preflight rebase can + /// preserve the transaction fields while committing at a later version. + /// Either mismatch is a post-effect recovery case, not permission to widen + /// the prepared plan. + pub async fn commit_staged_exact( + &self, + ds: Arc, + staged: StagedWrite, + ) -> Result<(Dataset, StagedTransactionIdentity)> { + let (dataset, committed_identity) = self + .commit_staged_with_retry_budget(ds, staged, Some(0)) + .await?; + let committed_identity = committed_identity.ok_or_else(|| { + OmniError::manifest_internal( + "Lance committed a staged effect without a readable transaction identity", + ) + })?; + Ok((dataset, committed_identity)) + } + + async fn commit_staged_with_retry_budget( + &self, + ds: Arc, + staged: StagedWrite, + max_retries: Option, + ) -> Result<(Dataset, Option)> { // Skip Lance's auto-cleanup hook on every commit. OmniGraph owns version // GC explicitly (optimize.rs::cleanup_all_tables); Lance's hook fires off // the *dataset's stored* `lance.auto_cleanup.*` config, which graphs @@ -1272,13 +1388,27 @@ impl TableStore { // data path) for new and legacy datasets alike, preventing Lance from // GC'ing versions the __manifest still pins for snapshots/time-travel. let mut builder = CommitBuilder::new(ds).with_skip_auto_cleanup(true); + if let Some(max_retries) = max_retries { + builder = builder.with_max_retries(max_retries); + } if let Some(affected_rows) = staged.commit_metadata.affected_rows { builder = builder.with_affected_rows(affected_rows); } - builder + let dataset = builder .execute(staged.transaction) .await - .map_err(|e| OmniError::Lance(e.to_string())) + .map_err(|e| OmniError::Lance(e.to_string()))?; + let committed_identity = if max_retries.is_some() { + dataset + .read_transaction() + .await + .map_err(|e| OmniError::Lance(e.to_string()))? + .as_ref() + .map(StagedTransactionIdentity::from) + } else { + None + }; + Ok((dataset, committed_identity)) } /// Stage an overwrite (write_fragments + Operation::Overwrite { schema, fragments }). @@ -1581,33 +1711,119 @@ impl TableStore { } let committed = self.scan(committed_ds, projection, filter, None).await?; - if pending_batches.is_empty() { - return Ok(committed); + combine_committed_with_pending( + committed, + pending_batches, + pending_schema, + projection, + filter, + key_column, + ) + .await + } + + /// Read a blob-bearing table as a full logical merge source while retaining + /// [`Self::scan_with_pending`]'s merge-shadow semantics. + /// + /// Lance normally scans blob-v2 columns as physical descriptor structs. + /// Those descriptors are a read representation, not valid writer input: a + /// full-row merge sends them through the blob writer, which requires the + /// logical `Struct` shape and fails with `Blob struct missing + /// data field`. This remained hidden while an `id` BTREE forced Lance's + /// partial-column merge plan; once index creation became reconciler-owned, + /// an index-absent table correctly selected the full-scan plan and exposed + /// the representation mismatch. + /// + /// A first scan evaluates `filter` with blob columns excluded and retains + /// stable row ids. Full descriptor rows are then taken by those ids without + /// a filter, and only their payloads are rebuilt as logical + /// [`BlobArrayBuilder`] columns. Pending rows already carry logical blob + /// arrays; both sides have the dataset's full logical schema before the + /// existing shadow union. The resulting batch is therefore valid for either + /// of Lance's merge plans, making physical index presence a performance + /// detail rather than a correctness precondition. + pub async fn scan_with_pending_materialized_blobs( + &self, + committed_ds: &Dataset, + pending_batches: &[RecordBatch], + pending_schema: Option, + filter: Option<&str>, + key_column: Option<&str>, + ) -> Result> { + let blob_columns = committed_ds + .schema() + .fields + .iter() + .filter(|field| field.is_blob()) + .map(|field| field.name.clone()) + .collect::>(); + if blob_columns.is_empty() { + return self + .scan_with_pending( + committed_ds, + pending_batches, + pending_schema, + None, + filter, + key_column, + ) + .await; } - // Shadow committed rows whose key value also appears in pending. - // This makes scan_with_pending implement merge semantics rather - // than naive union: any row that has a pending update is - // represented ONLY by its pending value, never by both its - // (stale) committed value and its (current) pending value. - let committed = match key_column { - Some(key_col) => { - let pending_keys = collect_string_column_values(pending_batches, key_col)?; - if pending_keys.is_empty() { - committed - } else { - filter_out_rows_where_string_in(committed, key_col, &pending_keys)? - } + // The pinned Lance revision cannot combine a predicate filter with a + // full blob-v2 projection: `FilteredReadExec` applies the descriptor + // child projection to the logical blob field and panics. Select the + // matched rows without blob columns, retaining stable `_rowid`, then + // take exactly those full descriptor rows without a filter and rebuild + // their logical blobs. Thus payload I/O remains proportional to matched + // rows while avoiding any dependence on which merge/index plan Lance + // selects later. + let non_blob_columns = committed_ds + .schema() + .fields + .iter() + .filter(|field| !field.is_blob()) + .map(|field| field.name.as_str()) + .collect::>(); + let matched = Self::scan_stream(committed_ds, Some(&non_blob_columns), filter, None, true) + .await? + .try_collect::>() + .await + .map_err(|error| OmniError::Lance(error.to_string()))?; + + let mut committed = Vec::with_capacity(matched.len()); + for batch in matched { + let row_ids = batch + .column_by_name("_rowid") + .and_then(|column| column.as_any().downcast_ref::()) + .ok_or_else(|| { + OmniError::Lance("expected _rowid in predicate-matched blob scan".to_string()) + })? + .values() + .to_vec(); + if row_ids.is_empty() { + continue; } - None => committed, - }; + let descriptors = committed_ds + .take_rows(&row_ids, committed_ds.schema().clone()) + .await + .map_err(|error| OmniError::Lance(error.to_string()))?; + committed.push( + Self::materialize_blob_batch_with_row_ids(committed_ds, descriptors, &row_ids) + .await?, + ); + } - let pending = - scan_pending_batches(pending_batches, pending_schema, projection, filter).await?; - - let mut out = committed; - out.extend(pending); - Ok(out) + let combined = combine_committed_with_pending( + committed, + pending_batches, + pending_schema, + None, + filter, + key_column, + ) + .await?; + Ok(combined) } /// `count_rows` variant that respects staged writes. Used for @@ -1844,6 +2060,42 @@ fn assign_row_id_meta(fragments: &mut [Fragment], start_row_id: u64) -> Result<( Ok(()) } +/// Apply the in-memory half of `scan_with_pending` to an already-scanned +/// committed view. Kept as one helper so the ordinary descriptor scan and the +/// blob-materializing scan cannot drift in their key-shadow semantics. +async fn combine_committed_with_pending( + committed: Vec, + pending_batches: &[RecordBatch], + pending_schema: Option, + projection: Option<&[&str]>, + filter: Option<&str>, + key_column: Option<&str>, +) -> Result> { + if pending_batches.is_empty() { + return Ok(committed); + } + + // Shadow committed rows whose key value also appears in pending. This is + // deliberately applied before filtering pending: a pending row that no + // longer matches must still suppress its stale committed predecessor. + let committed = match key_column { + Some(key_col) => { + let pending_keys = collect_string_column_values(pending_batches, key_col)?; + if pending_keys.is_empty() { + committed + } else { + filter_out_rows_where_string_in(committed, key_col, &pending_keys)? + } + } + None => committed, + }; + + let pending = scan_pending_batches(pending_batches, pending_schema, projection, filter).await?; + let mut out = committed; + out.extend(pending); + Ok(out) +} + /// Collect the set of values in a Utf8 column across multiple batches. /// Used by `scan_with_pending`'s merge-semantic path to identify /// committed rows that are shadowed by pending writes. NULL values are diff --git a/crates/omnigraph/src/validate.rs b/crates/omnigraph/src/validate.rs index 5d750c8d..1d0dc0d7 100644 --- a/crates/omnigraph/src/validate.rs +++ b/crates/omnigraph/src/validate.rs @@ -238,9 +238,10 @@ pub(crate) struct CommittedState<'a> { /// tables, not a global flag — an edges-only overwrite still sees committed /// nodes for RI. Empty on the merge / mutation / append / merge-load paths. overwritten: HashSet, - /// Write path only: open edge tables at LIVE HEAD for `@card` (the #298 - /// stale-handle fix). `None` on the merge and load paths — there the snapshot - /// (or empty) is the committed view for every check. + /// Write path only: open edge tables from a fresh graph-branch manifest + /// snapshot for `@card` (the #298 stale-handle fix). This is the live + /// committed graph view, not a raw Lance HEAD that may be unpublished or + /// belong to an inherited source ref. `None` on merge/load. live: Option<(&'a Omnigraph, Option<&'a str>)>, } @@ -255,7 +256,8 @@ impl<'a> CommittedState<'a> { } /// Write path: existence/uniqueness read `committed` (the write's pinned - /// base); cardinality reads LIVE HEAD per edge table via `db` (#298). + /// base); cardinality reads the live committed branch snapshot via `db` + /// (#298). pub(crate) fn write(committed: &'a Snapshot, db: &'a Omnigraph, branch: Option<&'a str>) -> Self { Self { committed: Some(committed), @@ -300,10 +302,12 @@ impl<'a> CommittedState<'a> { } } - /// Open an edge table for cardinality counting: LIVE HEAD on the write path - /// (so an edge a concurrent writer committed since the base was pinned is - /// counted — #298), the committed snapshot otherwise. `None` if there is no - /// committed view (Overwrite load) or the table isn't in it. + /// Open an edge table for cardinality counting: the current manifest-visible + /// graph-branch snapshot on the write path (so a concurrent published edge + /// is counted — #298), the pinned committed snapshot otherwise. Resolving + /// through the fresh graph snapshot is load-bearing for first-touch named + /// branches: their table still inherits another Lance ref until this write's + /// sidecar is armed, so opening the target ref directly would be invalid. async fn open_cardinality(&self, table_key: &str) -> Result> { if self.overwritten.contains(table_key) { return Ok(None); @@ -311,14 +315,21 @@ impl<'a> CommittedState<'a> { let Some(committed) = self.committed else { return Ok(None); }; - let Some(entry) = committed.entry(table_key) else { + let Some(_entry) = committed.entry(table_key) else { return Ok(None); }; match self.live { Some((db, branch)) => { - let full_path = db.storage().dataset_uri(&entry.table_path); - let handle = db.storage().open_dataset_head(&full_path, branch).await?; - Ok(Some(handle.dataset().clone())) + // `CommittedState::write` is constructed only after WriteTxn + // schema validation, so use the unchecked manifest refresh to + // avoid another full contract read while retaining live branch + // authority. Snapshot::open follows the entry's actual + // `table_branch` and pinned version (including inheritance). + let live = db.fresh_snapshot_for_branch_unchecked(branch).await?; + match live.entry(table_key) { + Some(_) => Ok(Some(live.open(table_key).await?)), + None => Ok(None), + } } None => Ok(Some(committed.open(table_key).await?)), } diff --git a/crates/omnigraph/tests/branching.rs b/crates/omnigraph/tests/branching.rs index 7f4fb4ba..8c2ed65e 100644 --- a/crates/omnigraph/tests/branching.rs +++ b/crates/omnigraph/tests/branching.rs @@ -351,6 +351,22 @@ async fn branch_merge_with_blob_columns_preserves_blob_data() { ) .await .unwrap(); + + // This regression must not rely on an incidental physical index selecting + // Lance's legacy partial-column merge plan. The materialized-blob update + // path is correct even when the table has no user index at all. + let ds = snapshot_main(&main) + .await + .unwrap() + .open("node:Document") + .await + .unwrap(); + let indices = ds.load_indices().await.unwrap(); + assert!( + indices.iter().all(is_system_index), + "blob correctness regression requires an index-absent table" + ); + main.branch_create("feature").await.unwrap(); let mut feature = Omnigraph::open(uri).await.unwrap(); @@ -629,7 +645,10 @@ async fn same_branch_insert_after_external_commit_is_linear() { .iter() .filter(|c| c.parent_commit_id.as_deref() == Some(c0.graph_commit_id.as_str())) .count(); - assert_eq!(c0_children, 1, "C0 must have exactly one child; two is the fork"); + assert_eq!( + c0_children, 1, + "C0 must have exactly one child; two is the fork" + ); } /// Strict update after a read: Fix 1's `refresh_manifest_only` makes the read @@ -676,7 +695,10 @@ async fn same_branch_update_after_external_commit_and_read_is_linear() { .await .unwrap() .unwrap(); - assert_eq!(cb.parent_commit_id.as_deref(), Some(ca.graph_commit_id.as_str())); + assert_eq!( + cb.parent_commit_id.as_deref(), + Some(ca.graph_commit_id.as_str()) + ); // A reads main: the stale-probe path refreshes A's MANIFEST (via // refresh_manifest_only) but not its commit-graph head, freshening the @@ -713,7 +735,10 @@ async fn same_branch_update_after_external_commit_and_read_is_linear() { .iter() .filter(|c| c.parent_commit_id.as_deref() == Some(ca.graph_commit_id.as_str())) .count(); - assert_eq!(ca_children, 1, "Ca must have exactly one child; two is the fork"); + assert_eq!( + ca_children, 1, + "Ca must have exactly one child; two is the fork" + ); } #[tokio::test] diff --git a/crates/omnigraph/tests/consistency.rs b/crates/omnigraph/tests/consistency.rs index aab0114e..09525c2a 100644 --- a/crates/omnigraph/tests/consistency.rs +++ b/crates/omnigraph/tests/consistency.rs @@ -815,16 +815,15 @@ query high_value() { assert_eq!(values.value(8), 499); } -// ─── Stale handle must refresh-and-retry (no silent rebase) ────────────── +// ─── Long-lived strict handle refreshes before preparation ─────────────── #[tokio::test] -async fn stale_handle_public_mutation_must_refresh_then_retry() { - // With the Run state machine removed, the engine no longer - // auto-rebases stale-handle mutations onto the latest target head. - // The publisher's `expected_table_versions` CAS makes the contract - // explicit — a stale writer fails loudly with - // `ExpectedVersionMismatch` and the client decides whether to - // refresh-and-retry. +async fn long_lived_handle_prepares_strict_mutation_from_current_head() { + // Merely opening a handle before another completed commit does not make the + // next attempt stale: open_write_txn probes the manifest incarnation and + // captures the current branch authority before it plans the strict update. + // ReadSetChanged is reserved for movement during an already-prepared + // attempt (covered deterministically in the failpoint suite). let dir = tempfile::tempdir().unwrap(); let _db = init_and_load(&dir).await; drop(_db); @@ -843,26 +842,8 @@ async fn stale_handle_public_mutation_must_refresh_then_retry() { .await .unwrap(); - // Writer 2 is now stale. Its first attempt must fail with - // ExpectedVersionMismatch — no silent rebase. - let stale_err = mutate_main( - &mut db2, - MUTATION_QUERIES, - "set_age", - &mixed_params(&[("$name", "Alice")], &[("$age", 99)]), - ) - .await - .expect_err("stale writer must hit ExpectedVersionMismatch"); - let omnigraph::error::OmniError::Manifest(manifest_err) = stale_err else { - panic!("expected Manifest error"); - }; - assert!(matches!( - manifest_err.details, - Some(omnigraph::error::ManifestConflictDetails::ExpectedVersionMismatch { .. }) - )); - - // Refresh and retry — the canonical client recovery path. - db2.sync_branch("main").await.unwrap(); + // Writer 2's handle predates Eve, but its strict attempt prepares from the + // current head and succeeds in one call. mutate_main( &mut db2, MUTATION_QUERIES, diff --git a/crates/omnigraph/tests/failpoints.rs b/crates/omnigraph/tests/failpoints.rs index 91ec9085..7982f12e 100644 --- a/crates/omnigraph/tests/failpoints.rs +++ b/crates/omnigraph/tests/failpoints.rs @@ -4,7 +4,7 @@ mod helpers; use fail::FailScenario; use omnigraph::db::Omnigraph; -use omnigraph::error::{ManifestErrorKind, OmniError}; +use omnigraph::error::OmniError; use omnigraph::failpoints::ScopedFailPoint; use omnigraph::failpoints::names; use omnigraph::loader::LoadMode; @@ -15,14 +15,38 @@ use helpers::recovery::{ branch_head_commit_id, single_sidecar_operation_id, }; use helpers::{ - MUTATION_QUERIES, collect_column_strings, mixed_params, mutate_main, read_table, test_session, - version_main, + MUTATION_QUERIES, collect_column_strings, count_rows, mixed_params, mutate_main, params, + read_table, test_session, version_main, }; const SCHEMA_V1: &str = "node Person { name: String @key }\n"; const SCHEMA_V2_ADDED_TYPE: &str = "node Person { name: String @key }\nnode Company { name: String @key }\n"; +const OCC_UNIQUE_SCHEMA: &str = r#" +node User { + name: String @key + email: String? + @unique(email) +} +"#; + +const OCC_UNIQUE_MUTATIONS: &str = r#" +query insert_user($name: String, $email: String) { + insert User { name: $name, email: $email } +} +"#; + +const OCC_DISJOINT_MUTATIONS: &str = r#" +query set_age($name: String, $age: I32) { + update Person set { age: $age } where name = $name +} + +query insert_company($name: String) { + insert Company { name: $name } +} +"#; + fn node_table_uri(root: &str, type_name: &str) -> String { let mut hash: u64 = 0xcbf2_9ce4_8422_2325; for &b in type_name.as_bytes() { @@ -178,10 +202,11 @@ async fn recreate_over_orphaned_fork_self_heals_without_cleanup() { } // The write-path orphan reclaim shares the same fresh-authority classifier as -// cleanup. If that classifier is Indeterminate (transient read on a live -// branch), the write must return a clear retryable authority-read conflict and -// leave the ref in place. It must not squeeze the ambiguity through -// ExpectedVersionMismatch with expected == actual, which lies about the cause. +// cleanup. If that classifier is Indeterminate after v3 recovery has been +// armed, the write must retain recovery ownership and return RecoveryRequired; +// deleting the sidecar would lose authority over a target ref that may already +// have been created. A full read-write reopen resolves the armed no-HEAD-effect +// attempt before the write can be retried. #[tokio::test] #[serial] async fn recreate_over_orphaned_fork_reports_indeterminate_authority_read() { @@ -206,16 +231,10 @@ async fn recreate_over_orphaned_fork_reports_indeterminate_authority_read() { .await .expect_err("indeterminate authority read must fail retryably"); - match &err { - OmniError::Manifest(manifest) => { - assert_eq!(manifest.kind, ManifestErrorKind::Conflict); - assert!( - manifest.details.is_none(), - "indeterminate authority read is not an expected-version mismatch: {manifest:?}" - ); - } - other => panic!("expected manifest conflict, got {other:?}"), - } + assert!( + matches!(err, OmniError::RecoveryRequired { .. }), + "an ambiguous post-arm fork error must retain recovery ownership: {err:?}" + ); let message = err.to_string(); assert!( message.contains("could not verify") @@ -235,9 +254,13 @@ async fn recreate_over_orphaned_fork_reports_indeterminate_authority_read() { ); } + drop(db); + let db = Omnigraph::open(&uri) + .await + .expect("read-write reopen must resolve the retained armed sidecar"); db.load_as("feature", None, row, LoadMode::Merge, None) .await - .expect("when fresh authority is available, the orphan is reclaimed and write converges"); + .expect("after recovery, fresh authority reclaims the orphan and the write converges"); } // cleanup is the guaranteed convergence backstop, so one table's transient @@ -409,12 +432,11 @@ async fn reconcile_skips_fork_when_fresh_recheck_is_unavailable_then_converges() } } - // A fork collision must be classified by the manifest authority, not by Lance // branch versions. When a concurrent first-write legitimately wins the fork -// race, the loser sees a version mismatch — but that is a stale snapshot, not -// an orphan, so it must be a retryable "refresh and retry", never a misleading -// "run cleanup". +// race, the loser sees a changed read set — but that is a safe pre-effect +// retry for Insert. RFC-022 discards and reprepares it automatically, never +// misclassifying the live fork as an orphan that needs cleanup. // // Ordering is made deterministic (no fixed sleeps) via the shared rendezvous: // it parks the first arrival (writer A) at the fork point until released; later @@ -422,7 +444,7 @@ async fn reconcile_skips_fork_when_fresh_recheck_is_unavailable_then_converges() // lets B win and commit the fork, then releases A. #[tokio::test(flavor = "multi_thread")] #[serial] -async fn fork_collision_with_live_concurrent_fork_is_retryable() { +async fn fork_collision_with_live_concurrent_fork_reprepares() { let _scenario = FailScenario::setup(); let dir = tempfile::tempdir().unwrap(); @@ -460,24 +482,673 @@ async fn fork_collision_with_live_concurrent_fork_is_retryable() { .await .unwrap(); - // Release A; it resumes, re-reads the manifest, and sees the fork is live. + // Release A; it resumes, sees that B changed branch authority, discards its + // stale attempt, and reprepares Eve against the now-live feature fork. rv.release(); - let err = writer_a + writer_a .await .unwrap() - .expect_err("A's stale-snapshot fork should be a retryable conflict"); + .expect("A's retryable insert must reprepare after B wins the fork"); - let msg = err.to_string(); + let db = Omnigraph::open(&uri).await.unwrap(); + assert_eq!( + helpers::count_rows_branch(&db, "feature", "node:Person").await, + 6, + "feature must preserve four inherited rows plus both concurrent inserts" + ); + assert_eq!( + helpers::count_rows(&db, "node:Person").await, + 4, + "feature fork retries must not change main" + ); +} + +/// RFC-022 first-touch forks are physical effects and must be protected before +/// `create_branch` runs. Writer A parks after its target ref exists but before +/// any data commit. A separately-opened writer B shares the root-scoped gates: +/// it waits rather than reclaiming/deleting A's live ref merely because the +/// graph manifest has not published it yet. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn cross_handle_reclaim_never_deletes_live_intent_owned_fork() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + let main = helpers::init_and_load(&dir).await; + main.branch_create("feature").await.unwrap(); + drop(main); + + // Open both handles before A creates a pending sidecar. Opening B afterward + // would deliberately invoke the quiesced full recovery sweep instead of the + // live-writer roll-forward-only path this race exercises. + let db_a = std::sync::Arc::new(Omnigraph::open(&uri).await.unwrap()); + let db_b = std::sync::Arc::new(Omnigraph::open(&uri).await.unwrap()); + let rendezvous = + helpers::failpoint::Rendezvous::park_first(names::MUTATION_POST_FORK_PRE_COMMIT); + + let writer_a_db = std::sync::Arc::clone(&db_a); + let writer_a = tokio::spawn(async move { + writer_a_db + .mutate( + "feature", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "Eve")], &[("$age", 22)]), + ) + .await + }); + rendezvous.wait_until_reached().await; + + let person_uri = node_table_uri(&uri, "Person"); + let fork_before = lance::Dataset::open(&person_uri) + .await + .unwrap() + .checkout_branch("feature") + .await + .unwrap() + .branch_identifier() + .await + .unwrap(); + + let writer_b_db = std::sync::Arc::clone(&db_b); + let mut writer_b = tokio::spawn(async move { + writer_b_db + .mutate( + "feature", + MUTATION_QUERIES, + "set_age", + &mixed_params(&[("$name", "Alice")], &[("$age", 99)]), + ) + .await + }); assert!( - !msg.contains("cleanup"), - "a live concurrent fork must not be misclassified as an orphan, got: {msg}" + tokio::time::timeout(std::time::Duration::from_millis(200), &mut writer_b) + .await + .is_err(), + "the second handle must wait on A's root-scoped effect gates" + ); + + let fork_after = lance::Dataset::open(&person_uri) + .await + .unwrap() + .checkout_branch("feature") + .await + .unwrap() + .branch_identifier() + .await + .unwrap(); + assert_eq!( + fork_after, fork_before, + "B must not delete and recreate A's intent-owned Lance ref" + ); + + rendezvous.release(); + writer_a + .await + .unwrap() + .expect("A must commit normally after the competing reclaim is rejected"); + writer_b + .await + .unwrap() + .expect("B must reprepare after A publishes and update without recreating the ref"); + drop(db_a); + + let mut db = Omnigraph::open(&uri).await.unwrap(); + assert_eq!( + helpers::count_rows_branch(&db, "feature", "node:Person").await, + 5, + "feature must contain the four inherited rows plus A's insert" + ); + let alice = helpers::query_branch( + &mut db, + "feature", + helpers::TEST_QUERIES, + "get_person", + ¶ms(&[("$name", "Alice")]), + ) + .await + .unwrap(); + assert_eq!(alice.to_rust_json()[0]["p.age"], serde_json::json!(99)); +} + +#[tokio::test] +#[serial] +async fn armed_first_touch_recovery_accepts_missing_target_ref() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + let db = helpers::init_and_load(&dir).await; + let main_rows = helpers::count_rows(&db, "node:Person").await; + db.branch_create("feature").await.unwrap(); + + { + let _failpoint = ScopedFailPoint::new(names::MUTATION_POST_SIDECAR_PRE_FORK, "return"); + let err = db + .mutate( + "feature", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "Eve")], &[("$age", 22)]), + ) + .await + .expect_err("failure after arming must leave recovery intent"); + assert!(matches!(err, OmniError::RecoveryRequired { .. })); + } + let operation_id = single_sidecar_operation_id(dir.path()); + let person_uri = node_table_uri(&uri, "Person"); + assert!( + !lance::Dataset::open(&person_uri) + .await + .unwrap() + .list_branches() + .await + .unwrap() + .contains_key("feature"), + "precondition: crash happened before the target ref was created" + ); + + // Stage A is a synchronous, branch-aware barrier. The Armed sidecar has no + // table effect for a HEAD-drift check to discover, but it is still ownership: + // another mutation on the same branch must name it and stop before preparing + // a competing first-touch effect. + let mutation_err = db + .mutate( + "feature", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "Follower")], &[("$age", 23)]), + ) + .await + .expect_err("same-branch mutation must stop at the unresolved Armed intent"); + match mutation_err { + OmniError::RecoveryRequired { + operation_id: actual, + .. + } => assert_eq!(actual, operation_id), + other => panic!("expected RecoveryRequired for the Armed intent, got {other}"), + } + + // An explicit load base is read authority too. Reject before the implicit + // child branch is created, even though the load target itself has no sidecar. + let load_err = db + .load_as( + "child", + Some("feature"), + r#"{"type":"Person","data":{"name":"LoadFollower","age":24}}"#, + LoadMode::Merge, + None, + ) + .await + .expect_err("load must stop on an unresolved intent for its explicit base"); + match load_err { + OmniError::RecoveryRequired { + operation_id: actual, + .. + } => assert_eq!(actual, operation_id), + other => panic!("expected RecoveryRequired for the load base, got {other}"), + } + assert!( + !db.branch_list() + .await + .unwrap() + .iter() + .any(|branch| branch == "child"), + "barrier rejection must precede the load's implicit branch creation" + ); + + // The feature intent does not overlap main. Branch filtering must avoid + // turning one branch's rollback-eligible residue into a graph-wide outage. + db.mutate( + "main", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "MainFollower")], &[("$age", 25)]), + ) + .await + .expect("an unresolved feature intent must not block a main mutation"); + assert_eq!(helpers::count_rows(&db, "node:Person").await, main_rows + 1); + assert_eq!( + helpers::recovery::sidecar_operation_ids(dir.path()), + vec![operation_id.clone()], + "blocked attempts must neither replace nor delete the owning sidecar" + ); + drop(db); + + let recovered = Omnigraph::open(&uri).await.unwrap(); + assert_eq!( + helpers::count_rows_branch(&recovered, "feature", "node:Person").await, + main_rows, + "abandoned intent must leave the feature table inherited" ); assert!( - msg.contains("refresh and retry") || msg.contains("expected manifest table version"), - "expected a retryable stale-view error, got: {msg}" + !dir.path() + .join("__recovery") + .join(format!("{operation_id}.json")) + .exists(), + "full recovery must remove the empty armed intent" ); } +#[tokio::test] +#[serial] +async fn load_without_explicit_base_does_not_add_main_to_recovery_scope() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + let db = helpers::init_and_load(&dir).await; + let main_rows = helpers::count_rows(&db, "node:Person").await; + db.branch_create("feature").await.unwrap(); + + // Leave an Armed main intent with an exact physical effect. RollForwardOnly + // must defer it, giving the load barrier a real unresolved main operation to + // filter rather than a synthetic test-only record. + { + let _failpoint = ScopedFailPoint::new(names::RECOVERY_SIDECAR_CONFIRM, "return"); + let err = db + .mutate( + "main", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "InterruptedMain")], &[("$age", 26)]), + ) + .await + .expect_err("confirmation failure must leave an Armed main intent"); + assert!(matches!(err, OmniError::RecoveryRequired { .. })); + } + let operation_id = single_sidecar_operation_id(dir.path()); + + // `base=None` means no explicit base dependency; it must not be represented + // as a phantom main branch in the recovery scope. The feature snapshot was + // cut before the interrupted main effect and can proceed independently. + db.load( + "feature", + r#"{"type":"Person","data":{"name":"FeatureLoad","age":27}}"#, + LoadMode::Merge, + ) + .await + .expect("a feature load without --from must ignore unresolved main recovery"); + assert_eq!( + helpers::count_rows_branch(&db, "feature", "node:Person").await, + main_rows + 1 + ); + assert_eq!( + helpers::recovery::sidecar_operation_ids(dir.path()), + vec![operation_id], + "the unrelated main intent must remain for Full recovery" + ); + + drop(db); + let recovered = Omnigraph::open(&uri) + .await + .expect("Full recovery must still compensate the interrupted main effect"); + assert_eq!( + helpers::count_rows(&recovered, "node:Person").await, + main_rows + ); + assert_eq!( + helpers::count_rows_branch(&recovered, "feature", "node:Person").await, + main_rows + 1 + ); +} + +#[tokio::test] +#[serial] +async fn armed_first_touch_recovery_reclaims_exact_no_effect_fork() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + let db = helpers::init_and_load(&dir).await; + let main_rows = helpers::count_rows(&db, "node:Person").await; + db.branch_create("feature").await.unwrap(); + + { + let _failpoint = ScopedFailPoint::new(names::MUTATION_POST_FORK_PRE_COMMIT, "return"); + let err = db + .mutate( + "feature", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "Eve")], &[("$age", 22)]), + ) + .await + .expect_err("failure after the fork must require recovery"); + assert!(matches!(err, OmniError::RecoveryRequired { .. })); + } + let operation_id = single_sidecar_operation_id(dir.path()); + let person_uri = node_table_uri(&uri, "Person"); + assert!( + lance::Dataset::open(&person_uri) + .await + .unwrap() + .list_branches() + .await + .unwrap() + .contains_key("feature"), + "precondition: intent-owned target ref exists without a committed effect" + ); + drop(db); + + let recovered = Omnigraph::open(&uri).await.unwrap(); + assert_eq!( + helpers::count_rows_branch(&recovered, "feature", "node:Person").await, + main_rows, + "recovery must leave the feature table inherited" + ); + assert!( + !lance::Dataset::open(&person_uri) + .await + .unwrap() + .list_branches() + .await + .unwrap() + .contains_key("feature"), + "full recovery must reclaim the exact unpublished no-effect ref" + ); + assert!( + !dir.path() + .join("__recovery") + .join(format!("{operation_id}.json")) + .exists(), + "full recovery must delete the abandoned sidecar" + ); +} + +/// A live writer keeps its schema/branch/table guards through Phase D. A +/// refresh from a separately-opened handle must share those root-scoped gates, +/// wait, and then skip the sidecar after A publishes/deletes it. Recovering the +/// confirmed effect early would make A's own publisher lose authority. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn cross_handle_refresh_waits_for_live_confirmed_writer() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + drop(helpers::init_and_load(&dir).await); + let db_a = std::sync::Arc::new(Omnigraph::open(&uri).await.unwrap()); + let db_b = std::sync::Arc::new(Omnigraph::open(&uri).await.unwrap()); + + let rendezvous = + helpers::failpoint::Rendezvous::park_first(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER); + let writer_db = std::sync::Arc::clone(&db_a); + let writer = tokio::spawn(async move { + writer_db + .mutate( + "main", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "Eve")], &[("$age", 22)]), + ) + .await + }); + rendezvous.wait_until_reached().await; + + let refresher_db = std::sync::Arc::clone(&db_b); + let mut refresher = tokio::spawn(async move { refresher_db.refresh().await }); + assert!( + tokio::time::timeout(std::time::Duration::from_millis(200), &mut refresher) + .await + .is_err(), + "refresh must wait for the live writer's shared table gate" + ); + + rendezvous.release(); + writer + .await + .unwrap() + .expect("A must publish normally; refresh must not steal its sidecar"); + refresher + .await + .unwrap() + .expect("refresh must skip A's now-deleted sidecar"); + assert_eq!(helpers::count_rows(&db_b, "node:Person").await, 5); + let recovery_dir = dir.path().join("__recovery"); + assert!( + !recovery_dir.exists() || std::fs::read_dir(recovery_dir).unwrap().next().is_none(), + "the successful writer must leave no recovery sidecar" + ); +} + +/// A ReadWrite open is itself a recovery actor. It must join the same +/// root-scoped gates as an already-open handle before running Full recovery; +/// otherwise it can classify an Armed pre-fork sidecar as dead and delete it +/// while the live writer is still about to create the target ref. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn read_write_open_waits_for_live_armed_prefork_writer() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + let db_a = std::sync::Arc::new(helpers::init_and_load(&dir).await); + db_a.branch_create("feature").await.unwrap(); + + let rendezvous = + helpers::failpoint::Rendezvous::park_first(names::MUTATION_POST_SIDECAR_PRE_FORK); + let writer_db = std::sync::Arc::clone(&db_a); + let writer = tokio::spawn(async move { + writer_db + .mutate( + "feature", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "Eve")], &[("$age", 22)]), + ) + .await + }); + rendezvous.wait_until_reached().await; + + let opener_uri = uri.clone(); + let mut opener = tokio::spawn(async move { Omnigraph::open(&opener_uri).await }); + assert!( + tokio::time::timeout(std::time::Duration::from_millis(200), &mut opener) + .await + .is_err(), + "ReadWrite open must wait on the live writer's root-scoped schema gate" + ); + + rendezvous.release(); + writer + .await + .unwrap() + .expect("A must finish; Full recovery must not delete its live intent"); + let opened = opener + .await + .unwrap() + .expect("the waiting open must succeed after A releases its gates"); + assert_eq!( + helpers::count_rows_branch(&opened, "feature", "node:Person").await, + 5 + ); + let recovery_dir = dir.path().join("__recovery"); + assert!(!recovery_dir.exists() || std::fs::read_dir(recovery_dir).unwrap().next().is_none()); +} + +/// Two dead Armed attempts may name the same first-touch ref: A created the +/// exact no-effect fork and failed, then B armed and failed before its fork. +/// Full recovery is quiesced by the shared gates: while another claim exists a +/// no-effect sidecar discards only itself, and the last survivor performs the +/// exact-ref cleanup. Mutual foreign-claim rejection would wedge every future +/// ReadWrite open forever. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn full_recovery_converges_multiple_no_effect_claims_for_one_fork() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + let db = helpers::init_and_load(&dir).await; + let main_rows = helpers::count_rows(&db, "node:Person").await; + db.branch_create("feature").await.unwrap(); + + // B passes the synchronous recovery barrier before A's sidecar exists, + // then parks before gates/effects. This models a foreign process or an + // already-prepared attempt; a newly-started in-process write is blocked by + // Stage A once A leaves its unresolved intent. + let before_effect = + helpers::failpoint::Rendezvous::park_first(names::MUTATION_POST_STAGE_PRE_EFFECT_GATE); + let writer_b_db = std::sync::Arc::new(db); + let writer_b_handle = std::sync::Arc::clone(&writer_b_db); + let writer_b = tokio::spawn(async move { + writer_b_handle + .mutate( + "feature", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "B")], &[("$age", 21)]), + ) + .await + }); + before_effect.wait_until_reached().await; + + { + let _failpoint = ScopedFailPoint::new(names::MUTATION_POST_FORK_PRE_COMMIT, "return"); + let err = writer_b_db + .mutate( + "feature", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "A")], &[("$age", 20)]), + ) + .await + .unwrap_err(); + assert!(matches!(err, OmniError::RecoveryRequired { .. })); + } + let first_operation = single_sidecar_operation_id(dir.path()); + let first_sidecar_path = dir + .path() + .join("__recovery") + .join(format!("{first_operation}.json")); + let first_sidecar_body = std::fs::read_to_string(&first_sidecar_path).unwrap(); + // Model a foreign/already-prepared B that did not observe A at the current + // binary's second Stage-A check; restore A afterward to exercise recovery's + // compatibility path for the preexisting two-claim state. + std::fs::remove_file(&first_sidecar_path).unwrap(); + { + let _failpoint = ScopedFailPoint::new(names::MUTATION_POST_SIDECAR_PRE_FORK, "return"); + before_effect.release(); + let err = writer_b.await.unwrap().unwrap_err(); + assert!(matches!(err, OmniError::RecoveryRequired { .. })); + } + std::fs::write(&first_sidecar_path, first_sidecar_body).unwrap(); + assert_eq!( + helpers::recovery::sidecar_operation_ids(dir.path()).len(), + 2, + "precondition: both no-effect claims are durable" + ); + let person_uri = node_table_uri(&uri, "Person"); + assert!( + lance::Dataset::open(&person_uri) + .await + .unwrap() + .list_branches() + .await + .unwrap() + .contains_key("feature"), + "precondition: A's exact no-effect target ref exists" + ); + drop(writer_b_db); + + let recovered = Omnigraph::open(&uri) + .await + .expect("ordered no-effect claim recovery must converge"); + assert_eq!( + helpers::count_rows_branch(&recovered, "feature", "node:Person").await, + main_rows + ); + assert!(helpers::recovery::sidecar_operation_ids(dir.path()).is_empty()); + assert!( + !lance::Dataset::open(&person_uri) + .await + .unwrap() + .list_branches() + .await + .unwrap() + .contains_key("feature"), + "the last no-effect claim must reclaim the exact unpublished ref" + ); +} + +/// A no-effect Armed intent must not delete a ref while a competing confirmed +/// intent claims it. A starts first and fails before fork; a synthesized +/// foreign/older writer B does not observe A, creates/commits/confirms that same +/// ref, and fails before publish. Full recovery discards A without touching B's +/// ref, then rolls B's exact effect forward. +#[tokio::test] +#[serial] +async fn full_recovery_discards_no_effect_claim_before_confirmed_competitor() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + let db = std::sync::Arc::new(helpers::init_and_load(&dir).await); + let main_rows = helpers::count_rows(&db, "node:Person").await; + db.branch_create("feature").await.unwrap(); + + // Produce a real, valid Armed pre-fork sidecar for A. + let armed_sidecar_path; + let armed_sidecar_body; + { + let _failpoint = ScopedFailPoint::new(names::MUTATION_POST_SIDECAR_PRE_FORK, "return"); + let err = db + .mutate( + "feature", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "Abandoned")], &[("$age", 40)]), + ) + .await + .unwrap_err(); + assert!(matches!(err, OmniError::RecoveryRequired { .. })); + let armed_operation = single_sidecar_operation_id(dir.path()); + armed_sidecar_path = dir + .path() + .join("__recovery") + .join(format!("{armed_operation}.json")); + armed_sidecar_body = std::fs::read_to_string(&armed_sidecar_path).unwrap(); + } + + // Current binaries close the Stage-A TOCTOU in commit_all and would reject + // B while A is visible. Temporarily hide A to synthesize the compatible + // preexisting/foreign-process state an older writer can leave, then restore + // the exact sidecar bytes after B's confirmed effect is durable. + std::fs::remove_file(&armed_sidecar_path).unwrap(); + { + let _failpoint = + ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); + let err = db + .mutate( + "feature", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "Winner")], &[("$age", 41)]), + ) + .await + .unwrap_err(); + assert!( + err.to_string() + .contains("mutation.post_finalize_pre_publisher"), + "unexpected confirmed-writer failure: {err}" + ); + } + std::fs::write(&armed_sidecar_path, armed_sidecar_body).unwrap(); + assert_eq!( + helpers::recovery::sidecar_operation_ids(dir.path()).len(), + 2, + "precondition: Armed A and EffectsConfirmed B are both pending" + ); + drop(db); + + let recovered = Omnigraph::open(&uri) + .await + .expect("A must discard without deleting B's confirmed ref"); + assert_eq!( + helpers::count_rows_branch(&recovered, "feature", "node:Person").await, + main_rows + 1 + ); + let people = helpers::read_table_branch(&recovered, "feature", "node:Person").await; + let names = collect_column_strings(&people, "name"); + assert!(names.iter().any(|name| name == "Winner")); + assert!(!names.iter().any(|name| name == "Abandoned")); + assert!(helpers::recovery::sidecar_operation_ids(dir.path()).is_empty()); +} + #[tokio::test(flavor = "multi_thread")] #[serial] async fn graph_publish_failpoint_triggers_before_commit_append() { @@ -502,6 +1173,585 @@ async fn graph_publish_failpoint_triggers_before_commit_append() { ); } +/// RFC-022 coarse OCC must protect the *validated plan*, not only the table +/// version handed to Lance. Writer A validates that an email is free and parks +/// after staging but before the branch effect gate. Writer B then commits the +/// same `@unique` email under a different key. Releasing A must discard the +/// stale attempt and rerun validation against B's commit; merely refreshing +/// A's expected table version would publish an invalid duplicate. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn mutation_revalidates_unique_after_pre_effect_authority_change() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap(); + let db = std::sync::Arc::new(Omnigraph::init(uri, OCC_UNIQUE_SCHEMA).await.unwrap()); + + let rendezvous = + helpers::failpoint::Rendezvous::park_first(names::MUTATION_POST_STAGE_PRE_EFFECT_GATE); + let writer_a_db = std::sync::Arc::clone(&db); + let writer_a = tokio::spawn(async move { + writer_a_db + .mutate( + "main", + OCC_UNIQUE_MUTATIONS, + "insert_user", + ¶ms(&[("$name", "stale-plan"), ("$email", "winner@example.com")]), + ) + .await + }); + + rendezvous.wait_until_reached().await; + db.mutate( + "main", + OCC_UNIQUE_MUTATIONS, + "insert_user", + ¶ms(&[("$name", "winner"), ("$email", "winner@example.com")]), + ) + .await + .expect("the second writer commits while the first attempt is parked pre-effect"); + let winner_manifest_pin = helpers::snapshot_main(&db) + .await + .unwrap() + .entry("node:User") + .unwrap() + .table_version; + let user_uri = node_table_uri(uri, "User"); + let winner_lance_head = lance::Dataset::open(&user_uri) + .await + .unwrap() + .latest_version_id() + .await + .unwrap(); + rendezvous.release(); + + let err = writer_a + .await + .unwrap() + .expect_err("the stale attempt must be replanned and fail @unique validation"); + assert!( + err.to_string().contains("@unique violation on User.email"), + "expected fresh @unique validation, got: {err}" + ); + + let final_manifest_pin = helpers::snapshot_main(&db) + .await + .unwrap() + .entry("node:User") + .unwrap() + .table_version; + let final_lance_head = lance::Dataset::open(&user_uri) + .await + .unwrap() + .latest_version_id() + .await + .unwrap(); + assert_eq!( + (final_manifest_pin, final_lance_head), + (winner_manifest_pin, winner_lance_head), + "the rejected stale attempt must not move either manifest pin or Lance HEAD" + ); + + assert_eq!(count_rows(&db, "node:User").await, 1); + let users = read_table(&db, "node:User").await; + assert_eq!(collect_column_strings(&users, "name"), vec!["winner"]); +} + +/// The coarse token is branch-wide: a commit to a table that the prepared +/// mutation does not write can still invalidate schema/cardinality/RI inputs. +/// A strict update therefore reports `ReadSetChanged` before moving its Person +/// table HEAD when a disjoint Company commit wins during preparation. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn strict_mutation_rejects_disjoint_head_change_before_effects() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap(); + let db = std::sync::Arc::new(helpers::init_and_load(&dir).await); + + let rendezvous = + helpers::failpoint::Rendezvous::park_first(names::MUTATION_POST_STAGE_PRE_EFFECT_GATE); + let writer_a_db = std::sync::Arc::clone(&db); + let writer_a = tokio::spawn(async move { + writer_a_db + .mutate( + "main", + OCC_DISJOINT_MUTATIONS, + "set_age", + &helpers::mixed_params(&[("$name", "Alice")], &[("$age", 99)]), + ) + .await + }); + + rendezvous.wait_until_reached().await; + db.mutate( + "main", + OCC_DISJOINT_MUTATIONS, + "insert_company", + ¶ms(&[("$name", "ConcurrentCo")]), + ) + .await + .expect("the disjoint Company insert commits while Person update is parked"); + + let winner_person_pin = helpers::snapshot_main(&db) + .await + .unwrap() + .entry("node:Person") + .unwrap() + .table_version; + let person_uri = node_table_uri(uri, "Person"); + let winner_person_head = lance::Dataset::open(&person_uri) + .await + .unwrap() + .latest_version_id() + .await + .unwrap(); + rendezvous.release(); + + let err = writer_a + .await + .unwrap() + .expect_err("strict stale read set must fail rather than auto-reprepare"); + let OmniError::Manifest(manifest_err) = err else { + panic!("expected a typed manifest conflict"); + }; + assert!(matches!( + manifest_err.details, + Some(omnigraph::error::ManifestConflictDetails::ReadSetChanged { + ref member, + .. + }) if member == "graph_head:main" + )); + + let final_person_pin = helpers::snapshot_main(&db) + .await + .unwrap() + .entry("node:Person") + .unwrap() + .table_version; + let final_person_head = lance::Dataset::open(&person_uri) + .await + .unwrap() + .latest_version_id() + .await + .unwrap(); + assert_eq!( + (final_person_pin, final_person_head), + (winner_person_pin, winner_person_head), + "strict rejection must happen before any Person table effect" + ); +} + +/// The load adapter shares the same prepared-write boundary as mutations. +/// Append is retryable, but a retry means rebuilding and revalidating the +/// whole attempt; it must never mean rebasing an already-validated batch. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn append_load_revalidates_unique_after_pre_effect_authority_change() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap(); + let db = std::sync::Arc::new(Omnigraph::init(uri, OCC_UNIQUE_SCHEMA).await.unwrap()); + + let rendezvous = + helpers::failpoint::Rendezvous::park_first(names::MUTATION_POST_STAGE_PRE_EFFECT_GATE); + let writer_a_db = std::sync::Arc::clone(&db); + let writer_a = tokio::spawn(async move { + writer_a_db + .load( + "main", + r#"{"type":"User","data":{"name":"stale-plan","email":"winner@example.com"}}"#, + LoadMode::Append, + ) + .await + }); + + rendezvous.wait_until_reached().await; + db.load( + "main", + r#"{"type":"User","data":{"name":"winner","email":"winner@example.com"}}"#, + LoadMode::Append, + ) + .await + .expect("the second load commits while the first attempt is parked pre-effect"); + let winner_manifest_pin = helpers::snapshot_main(&db) + .await + .unwrap() + .entry("node:User") + .unwrap() + .table_version; + let user_uri = node_table_uri(uri, "User"); + let winner_lance_head = lance::Dataset::open(&user_uri) + .await + .unwrap() + .latest_version_id() + .await + .unwrap(); + rendezvous.release(); + + let err = writer_a + .await + .unwrap() + .expect_err("the stale append must be rebuilt and fail @unique validation"); + assert!( + err.to_string().contains("@unique violation on User.email"), + "expected fresh @unique validation, got: {err}" + ); + + let final_manifest_pin = helpers::snapshot_main(&db) + .await + .unwrap() + .entry("node:User") + .unwrap() + .table_version; + let final_lance_head = lance::Dataset::open(&user_uri) + .await + .unwrap() + .latest_version_id() + .await + .unwrap(); + assert_eq!( + (final_manifest_pin, final_lance_head), + (winner_manifest_pin, winner_lance_head), + "the rejected stale load must not move either manifest pin or Lance HEAD" + ); + + assert_eq!(count_rows(&db, "node:User").await, 1); + let users = read_table(&db, "node:User").await; + assert_eq!(collect_column_strings(&users, "name"), vec!["winner"]); +} + +/// Overwrite is strict because its replacement image was computed from the +/// captured branch state. Even a disjoint graph commit invalidates that coarse +/// read token, and rejection must happen before the overwritten table moves. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn overwrite_load_rejects_disjoint_head_change_before_effects() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap(); + let db = std::sync::Arc::new(helpers::init_and_load(&dir).await); + + let rendezvous = + helpers::failpoint::Rendezvous::park_first(names::MUTATION_POST_STAGE_PRE_EFFECT_GATE); + let writer_a_db = std::sync::Arc::clone(&db); + let writer_a = tokio::spawn(async move { + writer_a_db + .load( + "main", + r#"{"type":"Person","data":{"name":"Alice","age":31}} +{"type":"Person","data":{"name":"Bob","age":25}} +{"type":"Person","data":{"name":"Charlie","age":35}} +{"type":"Person","data":{"name":"Diana","age":28}}"#, + LoadMode::Overwrite, + ) + .await + }); + + rendezvous.wait_until_reached().await; + db.mutate( + "main", + OCC_DISJOINT_MUTATIONS, + "insert_company", + ¶ms(&[("$name", "ConcurrentCo")]), + ) + .await + .expect("the disjoint Company insert commits while overwrite is parked"); + + let winner_person_pin = helpers::snapshot_main(&db) + .await + .unwrap() + .entry("node:Person") + .unwrap() + .table_version; + let person_uri = node_table_uri(uri, "Person"); + let winner_person_head = lance::Dataset::open(&person_uri) + .await + .unwrap() + .latest_version_id() + .await + .unwrap(); + rendezvous.release(); + + let err = writer_a + .await + .unwrap() + .expect_err("strict overwrite must reject the changed read set"); + let OmniError::Manifest(manifest_err) = err else { + panic!("expected a typed manifest conflict"); + }; + assert!(matches!( + manifest_err.details, + Some(omnigraph::error::ManifestConflictDetails::ReadSetChanged { + ref member, + .. + }) if member == "graph_head:main" + )); + + let final_person_pin = helpers::snapshot_main(&db) + .await + .unwrap() + .entry("node:Person") + .unwrap() + .table_version; + let final_person_head = lance::Dataset::open(&person_uri) + .await + .unwrap() + .latest_version_id() + .await + .unwrap(); + assert_eq!( + (final_person_pin, final_person_head), + (winner_person_pin, winner_person_head), + "overwrite rejection must happen before any Person table effect" + ); +} + +/// Stage A's entry-point heal and Stage C's effect gate are separated by +/// reclaimable preparation. B can therefore observe a clean graph, stage, and +/// pause while A leaves an EffectsConfirmed sidecar behind. Once B acquires +/// the shared gates it must attribute the barrier to A's exact operation — not +/// commit a second effect and not collapse the condition into generic manifest +/// drift. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn follower_after_initial_heal_reports_exact_pending_recovery_operation() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + drop(helpers::init_and_load(&dir).await); + let db_a = std::sync::Arc::new(Omnigraph::open(&uri).await.unwrap()); + let db_b = std::sync::Arc::new(Omnigraph::open(&uri).await.unwrap()); + let person_uri = node_table_uri(&uri, "Person"); + let initial_manifest_pin = helpers::snapshot_main(&db_a) + .await + .unwrap() + .entry("node:Person") + .unwrap() + .table_version; + + // B passes its initial (empty) recovery heal, stages from the old pin, and + // pauses before acquiring the effect gates. + let before_effect = + helpers::failpoint::Rendezvous::park_first(names::MUTATION_POST_STAGE_PRE_EFFECT_GATE); + let follower_db = std::sync::Arc::clone(&db_b); + let follower = tokio::spawn(async move { + follower_db + .mutate( + "main", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "Follower")], &[("$age", 44)]), + ) + .await + }); + before_effect.wait_until_reached().await; + + // A passes through the same pre-effect point (only the rendezvous's first + // arrival parks), commits its exact Lance transaction, confirms the v3 + // sidecar, then fails before manifest visibility. + { + let _post_effect_failure = + ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); + let err = db_a + .mutate( + "main", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "Interrupted")], &[("$age", 55)]), + ) + .await + .expect_err("A must stop after confirming its physical effect"); + assert!( + err.to_string() + .contains("mutation.post_finalize_pre_publisher"), + "unexpected A failure: {err}" + ); + + let operation_id = single_sidecar_operation_id(dir.path()); + let sidecar: serde_json::Value = serde_json::from_str( + &std::fs::read_to_string( + dir.path() + .join("__recovery") + .join(format!("{operation_id}.json")), + ) + .unwrap(), + ) + .unwrap(); + assert_eq!( + sidecar["protocol_v3"]["effect_phase"], "EffectsConfirmed", + "A must leave the exact confirmed-effect state this race targets" + ); + + let a_effect_head = lance::Dataset::open(&person_uri) + .await + .unwrap() + .latest_version_id() + .await + .unwrap(); + assert_eq!( + a_effect_head, + initial_manifest_pin + 1, + "A must advance exactly one physical table version" + ); + assert_eq!( + helpers::snapshot_main(&db_a) + .await + .unwrap() + .entry("node:Person") + .unwrap() + .table_version, + initial_manifest_pin, + "A's failed publisher must leave the manifest pin unchanged" + ); + + before_effect.release(); + let follower_err = follower + .await + .unwrap() + .expect_err("B must stop at A's recovery barrier before effects"); + match follower_err { + OmniError::RecoveryRequired { + operation_id: actual, + .. + } => assert_eq!( + actual, operation_id, + "B must name the deterministic sidecar that owns the barrier" + ), + other => panic!("expected exact RecoveryRequired attribution, got: {other}"), + } + + let after_b_head = lance::Dataset::open(&person_uri) + .await + .unwrap() + .latest_version_id() + .await + .unwrap(); + assert_eq!( + after_b_head, a_effect_head, + "B must not advance a second physical effect" + ); + assert_eq!( + helpers::snapshot_main(&db_b) + .await + .unwrap() + .entry("node:Person") + .unwrap() + .table_version, + initial_manifest_pin, + "B must not publish around A's pending recovery" + ); + assert_eq!( + helpers::recovery::sidecar_operation_ids(dir.path()), + vec![operation_id.clone()], + "the attributed sidecar must remain available for recovery" + ); + } + + drop(before_effect); + drop(db_a); + drop(db_b); + let recovered = Omnigraph::open(&uri) + .await + .expect("A's confirmed sidecar must remain recoverable"); + assert!(helpers::recovery::sidecar_operation_ids(dir.path()).is_empty()); + let people = read_table(&recovered, "node:Person").await; + let names = collect_column_strings(&people, "name"); + assert!(names.iter().any(|name| name == "Interrupted")); + assert!( + !names.iter().any(|name| name == "Follower"), + "B's staged-but-uncommitted row must never become visible" + ); +} + +/// Separately-opened handles share the root-scoped branch gate. Once A's exact +/// Lance effect is durable, a disjoint B write on the same graph branch must +/// wait through A's manifest publish instead of moving graph_head and forcing +/// A into post-effect recovery. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn cross_handle_branch_gate_serializes_post_effect_publish() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + let db = helpers::init_and_load(&dir).await; + drop(db); + + let db_a = std::sync::Arc::new(Omnigraph::open(&uri).await.unwrap()); + let db_b = std::sync::Arc::new(Omnigraph::open(&uri).await.unwrap()); + + // B prepares first but pauses before effects. A then commits its Person + // table effect and pauses before visibility. Releasing B makes it contend + // for the shared branch gate, which A still holds through Phase D. + let before_effect = + helpers::failpoint::Rendezvous::park_first(names::MUTATION_POST_STAGE_PRE_EFFECT_GATE); + let after_effect = + helpers::failpoint::Rendezvous::park_first(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER); + + let writer_b_db = std::sync::Arc::clone(&db_b); + let mut writer_b = tokio::spawn(async move { + writer_b_db + .mutate( + "main", + OCC_DISJOINT_MUTATIONS, + "insert_company", + ¶ms(&[("$name", "VisibleCo")]), + ) + .await + }); + before_effect.wait_until_reached().await; + + let writer_a_db = std::sync::Arc::clone(&db_a); + let writer_a = tokio::spawn(async move { + writer_a_db + .mutate( + "main", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "Doomed")], &[("$age", 55)]), + ) + .await + }); + after_effect.wait_until_reached().await; + + before_effect.release(); + assert!( + tokio::time::timeout(std::time::Duration::from_millis(200), &mut writer_b) + .await + .is_err(), + "B must wait while A holds the shared branch gate after its table effect" + ); + after_effect.release(); + + writer_a + .await + .unwrap() + .expect("A must publish normally while B waits"); + writer_b + .await + .unwrap() + .expect("B must reprepare against A's published authority and then commit"); + + assert_eq!( + count_rows(&db_a, "node:Person").await, + 5, + "A's Person insert must remain visible" + ); + assert_eq!( + count_rows(&db_b, "node:Company").await, + 3, + "B's disjoint Company insert must publish after A" + ); + assert!( + !std::path::Path::new(&uri).join("__recovery").exists() + || std::fs::read_dir(std::path::Path::new(&uri).join("__recovery")) + .unwrap() + .next() + .is_none(), + "both successful writers must delete their recovery intents" + ); +} + // Atomic schema apply: schema apply writes staging files first, then commits // the manifest, then renames staging → final. Tests below inject crashes at // the two boundaries and assert that reopening the graph yields a consistent @@ -653,7 +1903,8 @@ async fn recovery_rolls_forward_after_finalize_publisher_failure() { // Setup: trigger the residual. { let mut db = Omnigraph::init(&uri, helpers::TEST_SCHEMA).await.unwrap(); - let _failpoint = ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); + let _failpoint = + ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); // The mutation's finalize completes (commit_staged advances Lance // HEAD on node:Person AND writes a `__recovery/{ulid}.json` @@ -708,7 +1959,7 @@ async fn recovery_rolls_forward_after_finalize_publisher_failure() { assert_post_recovery_invariants( dir.path(), &operation_id, - RecoveryExpectation::RolledForward { + RecoveryExpectation::RolledForwardOriginalLineage { tables: vec![TableExpectation::main("node:Person").follow_up_mutation( FollowUpMutation::new( "main", @@ -775,9 +2026,8 @@ async fn open_sweep_roll_forward_converges_when_manifest_advances_concurrently() // Park the FIRST sweep to reach the publish window; later arrivals fall // through. wait_until_reached gates the second open so it is guaranteed // to be the one that converges the sidecar. - let rv = helpers::failpoint::Rendezvous::park_first( - names::RECOVERY_BEFORE_ROLL_FORWARD_PUBLISH, - ); + let rv = + helpers::failpoint::Rendezvous::park_first(names::RECOVERY_BEFORE_ROLL_FORWARD_PUBLISH); let uri_parked = uri.clone(); let parked_open = tokio::spawn(async move { Omnigraph::open(&uri_parked).await }); @@ -940,7 +2190,8 @@ async fn recovery_rolls_forward_load_on_feature_branch() { .table_version; feature_parent_commit_id = branch_head_commit_id(dir.path(), "feature").await.unwrap(); - let _failpoint = ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); + let _failpoint = + ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); let err = db .load( "feature", @@ -974,7 +2225,7 @@ async fn recovery_rolls_forward_load_on_feature_branch() { assert_post_recovery_invariants( dir.path(), &operation_id, - RecoveryExpectation::RolledForward { + RecoveryExpectation::RolledForwardOriginalLineage { tables: vec![ TableExpectation::branch("node:Person", "feature") .expected_main_manifest_pin(main_person_pin) @@ -1033,7 +2284,8 @@ async fn recovery_rolls_forward_load_overwrite() { .unwrap(); parent_commit_id = branch_head_commit_id(dir.path(), "main").await.unwrap(); - let _failpoint = ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); + let _failpoint = + ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); let err = db .load( "main", @@ -1062,7 +2314,7 @@ async fn recovery_rolls_forward_load_overwrite() { assert_post_recovery_invariants( dir.path(), &operation_id, - RecoveryExpectation::RolledForward { + RecoveryExpectation::RolledForwardOriginalLineage { tables: vec![ TableExpectation::main("node:Person") .expected_recovery_parent_commit_id(parent_commit_id) @@ -1119,6 +2371,11 @@ async fn recovery_rolls_forward_ensure_indices_on_feature_branch() { .await .unwrap(); + // RFC-022 writes no longer build declared indexes inline. Materialize the + // feature index once so the setup below can deliberately drop it and test + // ensure_indices recovery rather than first materialization. + db.ensure_indices_on("feature").await.unwrap(); + main_person_pin = db .snapshot_of(omnigraph::db::ReadTarget::branch("main")) .await @@ -1161,8 +2418,10 @@ async fn recovery_rolls_forward_ensure_indices_on_feature_branch() { feature_parent_commit_id = branch_head_commit_id(dir.path(), "feature").await.unwrap(); { - let _failpoint = - ScopedFailPoint::new(names::ENSURE_INDICES_POST_PHASE_B_PRE_MANIFEST_COMMIT, "return"); + let _failpoint = ScopedFailPoint::new( + names::ENSURE_INDICES_POST_PHASE_B_PRE_MANIFEST_COMMIT, + "return", + ); let err = db.ensure_indices_on("feature").await.unwrap_err(); assert!( err.to_string().contains( @@ -1240,7 +2499,8 @@ async fn refresh_runs_roll_forward_recovery_in_process() { // Setup: trigger the residual (sidecar persists; manifest unchanged). { - let _failpoint = ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); + let _failpoint = + ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); let err = mutate_main( &mut db, MUTATION_QUERIES, @@ -1301,6 +2561,73 @@ async fn refresh_runs_roll_forward_recovery_in_process() { assert_eq!(helpers::count_rows(&db, "node:Person").await, 2); } +/// Destructive version GC must not erase the transaction/version history that +/// schema-v3 recovery uses to prove exact effect ownership. A pending sidecar +/// is therefore a graph-wide cleanup barrier, checked before orphan reclaim or +/// any per-table fault-isolated GC begins. +#[tokio::test] +#[serial] +async fn cleanup_refuses_pending_v3_sidecar_before_version_gc() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + let mut db = Omnigraph::init(&uri, helpers::TEST_SCHEMA).await.unwrap(); + + { + let _failpoint = + ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); + mutate_main( + &mut db, + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "pending-gc")], &[("$age", 22)]), + ) + .await + .expect_err("failpoint must leave a confirmed v3 recovery intent"); + } + + let person_uri = node_table_uri(&uri, "Person"); + let before = lance::Dataset::open(&person_uri).await.unwrap(); + let before_head = before.version().version; + let before_versions = before.versions().await.unwrap().len(); + assert_eq!( + std::fs::read_dir(dir.path().join("__recovery")) + .unwrap() + .count(), + 1, + "test precondition: one v3 sidecar is pending" + ); + + let err = db + .cleanup(omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }) + .await + .expect_err("cleanup must not outrun recovery"); + assert!( + err.to_string() + .contains("cleanup requires a clean recovery state"), + "unexpected cleanup error: {err}" + ); + + let after = lance::Dataset::open(&person_uri).await.unwrap(); + assert_eq!(after.version().version, before_head); + assert_eq!(after.versions().await.unwrap().len(), before_versions); + assert_eq!( + std::fs::read_dir(dir.path().join("__recovery")) + .unwrap() + .count(), + 1, + "refused cleanup must leave recovery ownership intact" + ); + + drop(db); + Omnigraph::open(&uri) + .await + .expect("read-write reopen resolves the retained sidecar"); +} + /// The long-lived-process contract for `load`: a Phase B → Phase C /// failure (per-table `commit_staged` advanced Lance HEAD, manifest /// publish did not land, sidecar persists) must not wedge subsequent @@ -1324,7 +2651,8 @@ async fn load_after_finalize_publisher_failure_heals_without_reopen() { // commit_staged (Lance HEAD advances on three tables), then the // publisher is wedged before the manifest commit. { - let _failpoint = ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); + let _failpoint = + ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); let err = load_jsonl( &mut db, r#"{"type":"Person","data":{"name":"Alice","age":30}} @@ -1491,7 +2819,8 @@ async fn s3_load_recovers_after_publisher_failure_without_reopen() { // Failed load: commit_staged lands on S3, manifest publish does not; // the sidecar PUT went through the S3 adapter. { - let _failpoint = ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); + let _failpoint = + ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); let err = load_jsonl( &mut db, r#"{"type":"Person","data":{"name":"Alice","age":30}} @@ -1550,7 +2879,8 @@ async fn record_audit_failure_after_roll_forward_converges_on_next_write() { // Pending sidecar with real drift. { - let _failpoint = ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); + let _failpoint = + ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); load_jsonl( &mut db, r#"{"type":"Person","data":{"name":"Alice","age":30}} @@ -1642,7 +2972,8 @@ async fn sidecar_list_failure_fails_write_and_open_loudly_then_clears() { // Pending sidecar via the usual finalize → publisher failure. { - let _failpoint = ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); + let _failpoint = + ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); let err = load_jsonl( &mut db, r#"{"type":"Person","data":{"name":"Alice","age":30}} @@ -1870,7 +3201,8 @@ async fn mutation_after_finalize_publisher_failure_heals_without_reopen() { let mut db = Omnigraph::init(&uri, helpers::TEST_SCHEMA).await.unwrap(); { - let _failpoint = ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); + let _failpoint = + ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); let err = mutate_main( &mut db, MUTATION_QUERIES, @@ -1936,7 +3268,8 @@ async fn schema_apply_after_finalize_publisher_failure_heals_without_reopen() { let mut db = Omnigraph::init(&uri, helpers::TEST_SCHEMA).await.unwrap(); { - let _failpoint = ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); + let _failpoint = + ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); let err = load_jsonl( &mut db, r#"{"type":"Person","data":{"name":"Alice","age":30}} @@ -1957,7 +3290,10 @@ async fn schema_apply_after_finalize_publisher_failure_heals_without_reopen() { // Additive migration on the SAME handle. Must heal the load's // sidecar first, then apply normally. - let desired = format!("{}\nnode Tag {{ name: String @key }}\n", helpers::TEST_SCHEMA); + let desired = format!( + "{}\nnode Tag {{ name: String @key }}\n", + helpers::TEST_SCHEMA + ); db.apply_schema(&desired).await.expect( "schema apply on the same handle must heal sidecar-covered \ drift in-process instead of failing until restart", @@ -2018,7 +3354,8 @@ async fn branch_merge_after_finalize_publisher_failure_heals_without_reopen() { // Failed load on MAIN: Person drifts ahead of the manifest with a // sidecar covering it. { - let _failpoint = ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); + let _failpoint = + ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); let err = load_jsonl( &mut db, r#"{"type":"Person","data":{"name":"Bob","age":25}} @@ -2127,8 +3464,13 @@ async fn orphaned_branch_discard_is_idempotent_across_delete_failure() { ) .unwrap(); - // Orphan the sidecar. - db.branch_delete("feature").await.unwrap(); + // Orphan the synthetic sidecar with a raw substrate ref delete. The public + // branch-control protocol now runs the synchronous recovery barrier and + // correctly refuses to delete a branch named by pending recovery intent; + // this test needs the crash/external-state shape that recovery must handle. + let manifest_uri = format!("{uri}/__manifest"); + let mut manifest = lance::Dataset::open(&manifest_uri).await.unwrap(); + manifest.delete_branch("feature").await.unwrap(); // First write: the discard path writes its audit row, then the // sidecar delete fails (injected). The write fails loudly. @@ -2170,15 +3512,13 @@ async fn orphaned_branch_discard_is_idempotent_across_delete_failure() { ); } -/// When the commit-time drift guard cannot LIST sidecars to classify -/// the drift (transient storage fault on the guard's list, after the -/// entry heal's list succeeded), it must say so and name BOTH recovery -/// paths — not confidently route to `omnigraph repair`, which refuses -/// while a sidecar is pending. Sequenced failpoint: first list (entry -/// heal) passes, second list (the guard) fails. +/// The closed mutation/load Stage-A barrier owns pending covered drift before +/// the later HEAD guard has to guess between recovery and repair. It reports the +/// exact operation id and recovery action instead of routing an enrolled write +/// toward `repair`. #[tokio::test] #[serial] -async fn drift_guard_names_both_paths_when_sidecar_list_fails() { +async fn stage_a_barrier_reports_exact_operation_before_drift_guard() { use omnigraph::loader::{LoadMode, load_jsonl}; let _scenario = FailScenario::setup(); @@ -2235,9 +3575,6 @@ async fn drift_guard_names_both_paths_when_sidecar_list_fails() { ) .unwrap(); - // First list (entry heal) passes and defers the sidecar; second - // list (the guard's classification) fails. - let _failpoint = ScopedFailPoint::new(names::RECOVERY_SIDECAR_LIST, "1*off->1*return"); let err = load_jsonl( &mut db, "{\"type\":\"Person\",\"data\":{\"name\":\"bob\",\"age\":25}}\n", @@ -2248,22 +3585,23 @@ async fn drift_guard_names_both_paths_when_sidecar_list_fails() { .expect("drift must still fail the write"); let msg = err.to_string(); assert!( - msg.contains("could not classify the drift") - && msg.contains("omnigraph repair") - && msg.contains("reopen the graph read-write"), - "an unclassifiable drift must name BOTH recovery paths, not \ - confidently route to repair; got: {msg}" + matches!( + err, + OmniError::RecoveryRequired { ref operation_id, .. } + if operation_id == "01H0000000000000000000LSTF" + ) && msg.contains("reopen the graph read-write") + && !msg.contains("omnigraph repair"), + "the Stage-A barrier must attribute the pending recovery exactly; got: {msg}" ); } /// The other half of the orphan-discard fault matrix: the audit append /// fails AFTER the recovery commit landed. The retry (keyed on the /// audit row, the operator-facing record) must converge to exactly one -/// audit row and a consumed sidecar. The second recovery commit the -/// retry appends is the documented not-atomic-pair-write tolerance -/// (same class as `record_audit` and the manifest→commit-graph Known -/// Gap): bounded commit-graph noise, never a lost or duplicated audit -/// record under clean failures. +/// audit row and a consumed sidecar. Lineage and the recovery audit remain two +/// durable writes; a legacy sidecar has no fixed recovery-commit id, so a retry +/// after the lineage CAS but before the audit append may add bounded lineage +/// noise. The operator-facing audit record still lands exactly once. #[tokio::test] #[serial] async fn orphaned_branch_discard_converges_across_audit_append_failure() { @@ -2320,13 +3658,19 @@ async fn orphaned_branch_discard_converges_across_audit_append_failure() { &sidecar_json, ) .unwrap(); - db.branch_delete("feature").await.unwrap(); + // Manufacture the dead-branch recovery shape below the public branch + // control protocol, which now refuses a ref delete while this sidecar is + // pending. + let manifest_uri = format!("{uri}/__manifest"); + let mut manifest = lance::Dataset::open(&manifest_uri).await.unwrap(); + manifest.delete_branch("feature").await.unwrap(); // First write: the recovery commit lands, then the audit append // fails (injected). The write fails loudly; the sidecar survives so // the discard is retried with the audit still owed. { - let _failpoint = ScopedFailPoint::new(names::RECOVERY_ORPHAN_DISCARD_AUDIT_APPEND, "return"); + let _failpoint = + ScopedFailPoint::new(names::RECOVERY_ORPHAN_DISCARD_AUDIT_APPEND, "return"); let err = load_jsonl( &mut db, "{\"type\":\"Person\",\"data\":{\"name\":\"Bob\",\"age\":25}}\n", @@ -2478,13 +3822,19 @@ async fn heal_does_not_promote_live_schema_apply_staging() { let rv = helpers::failpoint::Rendezvous::park_first(names::SCHEMA_APPLY_AFTER_STAGING_WRITE); let apply_db = Arc::clone(&db); - let desired = format!("{}\nnode Tag {{ name: String @key }}\n", helpers::TEST_SCHEMA); + let desired = format!( + "{}\nnode Tag {{ name: String @key }}\n", + helpers::TEST_SCHEMA + ); let apply = tokio::spawn(async move { apply_db.apply_schema(&desired).await }); // Wait until the apply is parked in the window (staging files written). rv.wait_until_reached().await; let staging_pg = dir.path().join("_schema.pg.staging"); - assert!(staging_pg.exists(), "schema apply never reached the paused window"); + assert!( + staging_pg.exists(), + "schema apply never reached the paused window" + ); // Concurrent load on the same handle: its entry heal runs while the // apply is paused. The load itself may fail (schema apply in @@ -2644,11 +3994,9 @@ async fn refresh_defers_rollback_eligible_sidecar_to_next_open() { pre_head={pre_head}, post_head={post_head}", ); - // A write attempt while the rollback-eligible sidecar is deferred: - // the write-entry heal defers it again (roll-forward-only), and the - // commit-time drift guard must name the actual recovery path (a - // read-write reopen) — NOT `omnigraph repair`, which refuses while - // a sidecar is pending. + // A write attempt while the rollback-eligible sidecar is deferred stops at + // the synchronous Stage-A barrier, before base capture/effects, and names + // the exact operation that requires a read-write reopen. let err = mutate_main( &mut db, MUTATION_QUERIES, @@ -2657,12 +4005,19 @@ async fn refresh_defers_rollback_eligible_sidecar_to_next_open() { ) .await .unwrap_err(); - assert!( - err.to_string() - .contains("a pending recovery sidecar requires rollback"), - "drift guard must point at a read-write reopen for sidecar-covered \ - rollback-eligible drift; got: {err}" - ); + match err { + OmniError::RecoveryRequired { + operation_id, + reason, + } => { + assert_eq!(operation_id, "01H0000000000000000000RBCK"); + assert!( + reason.contains("reopen the graph read-write"), + "barrier must point at the safe recovery path; got: {reason}" + ); + } + other => panic!("expected typed RecoveryRequired barrier, got: {other}"), + } // Cross-check: drop the engine and reopen — full sweep handles // the rollback (will use Dataset::restore safely; no concurrent @@ -2719,7 +4074,8 @@ async fn finalize_publisher_residual_does_not_drift_untouched_tables() { .unwrap(); { - let _failpoint = ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); + let _failpoint = + ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); let _ = mutate_main( &mut db, MUTATION_QUERIES, @@ -2745,17 +4101,16 @@ async fn finalize_publisher_residual_does_not_drift_untouched_tables() { /// (`stage_create_btree_index` succeeded; `commit_staged` not yet called) /// leaves NO Lance-HEAD drift, so other tables stay writable. /// -/// Under iss-848 schema apply no longer builds indexes inline — the build -/// happens in the reconciler (`ensure_indices`/`optimize`) and at load. So this +/// Under RFC-022 schema apply and enrolled data writes no longer build indexes +/// inline; the build happens in the reconciler (`ensure_indices`/`optimize`). This /// fires the failpoint where it lives now: an `ensure_indices` build of a BTREE /// that a prior apply declared (`@index`) but deferred. The failpoint fires /// between `stage_create_btree_index` and `commit_staged`, so the staged /// segment is written under `_indices//` but `node:Person`'s Lance HEAD is -/// unchanged. `ensure_indices` fails and its EnsureIndices sidecar pins only -/// Person at NoMovement (a clean no-op on the next open). A write to a -/// different, unpinned table (`node:Company`) is unaffected: mutations/loads run -/// a roll-forward-only heal and proceed — they do not refuse on a pending -/// sidecar the way `optimize`/`repair` do — so the write succeeds with no drift. +/// unchanged. `ensure_indices` fails and its EnsureIndices sidecar pins Person +/// at NoMovement. The coarse branch-wide Stage-A barrier nevertheless blocks a +/// Company write until Full ReadWrite-open recovery consumes that unresolved +/// intent; after reopen the unrelated table is writable with no drift. #[tokio::test] #[serial] async fn ensure_indices_stage_btree_failure_leaves_existing_tables_writable() { @@ -2764,7 +4119,8 @@ async fn ensure_indices_stage_btree_failure_leaves_existing_tables_writable() { let uri = dir.path().to_str().unwrap().to_string(); let mut db = Omnigraph::init(&uri, helpers::TEST_SCHEMA).await.unwrap(); - // Seed a Person row — the load builds Person's id BTREE + name FTS. + // Seed a Person row. The enrolled mutation publishes only its logical data + // effect; physical index construction remains reconciler-owned. mutate_main( &mut db, helpers::MUTATION_QUERIES, @@ -2794,15 +4150,30 @@ async fn ensure_indices_stage_btree_failure_leaves_existing_tables_writable() { ); } - // A different, unpinned table is untouched by the failed index build. + // A different table is physically untouched, but the branch-wide recovery + // barrier does not let a new graph commit advance around any unresolved + // main-branch intent. use omnigraph::loader::{LoadMode, load_jsonl}; - load_jsonl( + let err = load_jsonl( &mut db, r#"{"type": "Company", "data": {"name": "Acme"}}"#, LoadMode::Append, ) .await - .expect("Company write on a table untouched by the failed ensure_indices should succeed"); + .expect_err("Stage A must block even a disjoint table on the same graph branch"); + assert!(matches!(err, OmniError::RecoveryRequired { .. })); + + drop(db); + let mut recovered = Omnigraph::open(&uri) + .await + .expect("Full recovery must consume the no-effect EnsureIndices sidecar"); + load_jsonl( + &mut recovered, + r#"{"type": "Company", "data": {"name": "Acme"}}"#, + LoadMode::Append, + ) + .await + .expect("Company write must succeed after Full recovery closes the barrier"); } fn assert_no_staging_files(graph: &std::path::Path) { @@ -3175,17 +4546,13 @@ async fn optimize_phase_b_failure_recovered_on_next_open() { .expect("schema apply after optimize recovery must succeed"); } -/// Cross-process race (the prod bug): a served write advances the manifest on the -/// same table while a SEPARATE `optimize` process is paused between its compaction -/// and its manifest publish. The in-process write queue does NOT serialize across -/// processes, so optimize's equality-CAS publish (expected = its pre-compaction -/// version) finds the manifest already advanced. optimize must CONVERGE — the -/// concurrent write built on top of the compacted HEAD, so the compaction is -/// already reflected — not fail with "expected X but current Y". RED before the -/// monotonic-publish fix. +/// Separately-opened handles for one root share the table/recovery gates. A +/// served insert that starts while optimize is paused before compaction must +/// wait, then commit after optimize releases its sidecar lifetime; it must not +/// deadlock in the synchronous recovery barrier or lose either result. #[tokio::test(flavor = "multi_thread", worker_threads = 4)] #[serial(optimize)] -async fn optimize_survives_concurrent_insert_advancing_manifest() { +async fn optimize_serializes_concurrent_insert_across_handles() { let _scenario = FailScenario::setup(); let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap().to_string(); @@ -3204,10 +4571,10 @@ async fn optimize_survives_concurrent_insert_advancing_manifest() { } } - // Pause optimize BEFORE it compacts, so the concurrent insert lands while - // HEAD == manifest (no in-flight optimize drift for the writer to trip on); the - // insert advances the manifest, then optimize compacts on top and must converge - // its publish over the advanced manifest rather than fail the equality CAS. + let db_b = std::sync::Arc::new(Omnigraph::open(&uri).await.unwrap()); + + // Pause optimize before compaction while it owns the Person table's + // sidecar/queue lifetime. let failpoint = ScopedFailPoint::new(names::OPTIMIZE_BEFORE_COMPACT, "pause"); let uri_opt = uri.clone(); @@ -3222,26 +4589,33 @@ async fn optimize_survives_concurrent_insert_advancing_manifest() { "optimize never reached the pre-compact pause", ); - // Concurrent insert on the SAME table via a SEPARATE handle (= separate - // in-process write queue = a different process) advances the manifest. - { - let db_b = Omnigraph::open(&uri).await.unwrap(); - db_b.mutate( - "main", - MUTATION_QUERIES, - "insert_person", - &mixed_params(&[("$name", "eve")], &[("$age", 34)]), - ) - .await - .unwrap(); - } + let writer_db = std::sync::Arc::clone(&db_b); + let writer = tokio::spawn(async move { + writer_db + .mutate( + "main", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "eve")], &[("$age", 34)]), + ) + .await + }); + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + assert!( + !writer.is_finished(), + "same-root writer must wait for optimize's guarded sidecar lifetime" + ); drop(failpoint); // release optimize let result = tokio::time::timeout(std::time::Duration::from_secs(20), optimize) .await .expect("optimize task hung") .unwrap(); - result.expect("optimize must survive a concurrent same-table write (cross-process)"); + result.expect("optimize must finish before the queued same-table writer"); + writer + .await + .expect("writer task panicked") + .expect("queued insert must resume after optimize"); // No lost write: 4 seed + eve all present; graph remains re-optimizable. let db = Omnigraph::open(&uri).await.unwrap(); @@ -3255,17 +4629,12 @@ async fn optimize_survives_concurrent_insert_advancing_manifest() { .expect("graph must remain healthy / re-optimizable"); } -/// Cross-process race: a served DELETE commits on the same table while a SEPARATE -/// `optimize` process is parked just before its compaction. Lance rebases the -/// compaction past the delete cleanly (so this surfaces as a manifest-CAS mismatch -/// at publish, not a Lance `Rewrite` conflict — the genuine `Rewrite`-vs-`Rewrite` -/// overlap is the rarer many-fragment/concurrent-compaction case, covered by the -/// shared `is_retryable_lance_conflict` retry the internal-table path already -/// exercises). optimize must converge its publish over the advanced manifest and -/// preserve the delete. RED before the fix. +/// Same as the insert cell, for a strict delete: the second handle waits for +/// optimize, then replans from the manifest-visible post-optimize state and +/// preserves the deletion. #[tokio::test(flavor = "multi_thread", worker_threads = 4)] #[serial(optimize)] -async fn optimize_survives_concurrent_delete_before_compaction() { +async fn optimize_serializes_concurrent_delete_across_handles() { let _scenario = FailScenario::setup(); let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap().to_string(); @@ -3284,6 +4653,8 @@ async fn optimize_survives_concurrent_delete_before_compaction() { } } + let db_b = std::sync::Arc::new(Omnigraph::open(&uri).await.unwrap()); + // Pause optimize BEFORE its compaction commits. let failpoint = ScopedFailPoint::new(names::OPTIMIZE_BEFORE_COMPACT, "pause"); @@ -3298,27 +4669,33 @@ async fn optimize_survives_concurrent_delete_before_compaction() { "optimize never reached the pre-compact pause", ); - // Concurrent DELETE of an existing row writes a deletion vector onto the - // fragment optimize is about to compact → optimize's Rewrite overlap-conflicts - // at the Lance level ("Rewrite … preempted by concurrent Delete/Update"). - { - let db_b = Omnigraph::open(&uri).await.unwrap(); - db_b.mutate( - "main", - MUTATION_QUERIES, - "remove_person", - &mixed_params(&[("$name", "alice")], &[]), - ) - .await - .unwrap(); - } + let writer_db = std::sync::Arc::clone(&db_b); + let writer = tokio::spawn(async move { + writer_db + .mutate( + "main", + MUTATION_QUERIES, + "remove_person", + &mixed_params(&[("$name", "alice")], &[]), + ) + .await + }); + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + assert!( + !writer.is_finished(), + "same-root delete must wait for optimize's guarded sidecar lifetime" + ); drop(failpoint); // release optimize let result = tokio::time::timeout(std::time::Duration::from_secs(20), optimize) .await .expect("optimize task hung") .unwrap(); - result.expect("optimize must reopen+replan past a concurrent overlapping delete"); + result.expect("optimize must finish before the queued delete"); + writer + .await + .expect("writer task panicked") + .expect("queued delete must resume after optimize"); // No lost write: alice's delete persisted (3 rows); graph remains re-optimizable. let db = Omnigraph::open(&uri).await.unwrap(); @@ -3468,8 +4845,10 @@ async fn branch_merge_phase_b_failure_recovered_on_next_open() { // but before commit_manifest_updates. Sidecar persists. { let db = Omnigraph::open(&uri).await.unwrap(); - let _failpoint = - ScopedFailPoint::new(names::BRANCH_MERGE_POST_PHASE_B_PRE_MANIFEST_COMMIT, "return"); + let _failpoint = ScopedFailPoint::new( + names::BRANCH_MERGE_POST_PHASE_B_PRE_MANIFEST_COMMIT, + "return", + ); let err = db.branch_merge("feature", "main").await.unwrap_err(); assert!( err.to_string().contains( @@ -3533,16 +4912,13 @@ async fn branch_merge_phase_b_failure_recovered_on_next_open() { // `merged_parent_commit_id`. Only a recovered branch_merge can produce one // here (we never completed a normal merge in this test). { - let commits = - omnigraph::db::commit_graph::CommitGraph::open(dir.path().to_str().unwrap()) - .await - .unwrap() - .load_commits() - .await - .unwrap(); - let found_recovery_merge = commits - .iter() - .any(|c| c.merged_parent_commit_id.is_some()); + let commits = omnigraph::db::commit_graph::CommitGraph::open(dir.path().to_str().unwrap()) + .await + .unwrap() + .load_commits() + .await + .unwrap(); + let found_recovery_merge = commits.iter().any(|c| c.merged_parent_commit_id.is_some()); assert!( found_recovery_merge, "recovered branch_merge must record `merged_parent_commit_id` so future \ @@ -3601,8 +4977,10 @@ async fn branch_merge_adopt_with_delta_phase_b_failure_recovered_on_next_open() // Fail after the per-table publish loop, before commit_manifest_updates. { let db = Omnigraph::open(&uri).await.unwrap(); - let _failpoint = - ScopedFailPoint::new(names::BRANCH_MERGE_POST_PHASE_B_PRE_MANIFEST_COMMIT, "return"); + let _failpoint = ScopedFailPoint::new( + names::BRANCH_MERGE_POST_PHASE_B_PRE_MANIFEST_COMMIT, + "return", + ); let err = db.branch_merge("feature", "main").await.unwrap_err(); assert!( err.to_string().contains( @@ -3869,7 +5247,10 @@ async fn pre_upgrade_v1_branch_merge_sidecar_rolls_forward_not_back() { // sidecar lands on disk. { let db = Omnigraph::open(&uri).await.unwrap(); - let _fp = ScopedFailPoint::new(names::BRANCH_MERGE_POST_PHASE_B_PRE_MANIFEST_COMMIT, "return"); + let _fp = ScopedFailPoint::new( + names::BRANCH_MERGE_POST_PHASE_B_PRE_MANIFEST_COMMIT, + "return", + ); db.branch_merge("feature", "main").await.unwrap_err(); } @@ -3975,8 +5356,10 @@ async fn branch_merge_phase_b_failure_recovered_on_non_main_target() { // branch=Some("target_branch"). { let db = Omnigraph::open(&uri).await.unwrap(); - let _failpoint = - ScopedFailPoint::new(names::BRANCH_MERGE_POST_PHASE_B_PRE_MANIFEST_COMMIT, "return"); + let _failpoint = ScopedFailPoint::new( + names::BRANCH_MERGE_POST_PHASE_B_PRE_MANIFEST_COMMIT, + "return", + ); let err = db .branch_merge("source_branch", "target_branch") .await @@ -4078,8 +5461,10 @@ async fn branch_merge_sidecar_pins_table_branch_to_active_branch() { { let db = Omnigraph::open(&uri).await.unwrap(); - let _failpoint = - ScopedFailPoint::new(names::BRANCH_MERGE_POST_PHASE_B_PRE_MANIFEST_COMMIT, "return"); + let _failpoint = ScopedFailPoint::new( + names::BRANCH_MERGE_POST_PHASE_B_PRE_MANIFEST_COMMIT, + "return", + ); let _ = db .branch_merge("source_branch", "target_branch") .await @@ -4127,18 +5512,15 @@ async fn branch_merge_sidecar_pins_table_branch_to_active_branch() { /// (every declared index already built, or empty tables that the loop /// skips), the sidecar is omitted entirely. /// -/// Test setup: `load_jsonl` auto-builds indices via -/// `prepare_updates_for_commit`. So after the load, every Person/Knows -/// index is built and Company is empty. `ensure_indices` correctly -/// produces zero pins → no sidecar. The failpoint still fires (it sits -/// after the loops), so the call returns Err — but no recovery state -/// persists. Reopen is a clean no-op. +/// Test setup: RFC-022 leaves index materialization to the reconciler, so the +/// first `ensure_indices` after load builds Person's declared index. A second +/// call is then the steady-state no-work case: zero pins → no sidecar. The +/// failpoint still fires (it sits after the loops), so the call returns Err — +/// but no recovery state persists. Reopen is a clean no-op. /// -/// Triggering an actual sidecar persistence requires bypassing -/// `load_jsonl`'s auto-build via raw `TableStore::append_batch` — the -/// helper-direct path. That's covered structurally by the -/// `needs_index_work_*` code path and the -/// `recovery_ensure_indices_handles_empty_tables` integration test. +/// Actual EnsureIndices sidecar persistence and staged-index failure are covered +/// by `ensure_indices_stage_btree_failure_leaves_existing_tables_writable`; this +/// test deliberately reaches the second, no-work reconciliation pass. #[tokio::test] #[serial] async fn ensure_indices_phase_b_failure_does_not_leak_sidecar_when_no_work_needed() { @@ -4148,9 +5530,8 @@ async fn ensure_indices_phase_b_failure_does_not_leak_sidecar_when_no_work_neede let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap().to_string(); - // Seed: load_jsonl auto-builds Person's indices via - // prepare_updates_for_commit. After this, ensure_indices has no - // work to do (steady state). + // Seed, then reconcile the index declaration once. RFC-022 writes publish + // only their exact data effect; index construction is derived work. { let mut db = Omnigraph::init(&uri, helpers::TEST_SCHEMA).await.unwrap(); load_jsonl( @@ -4162,6 +5543,7 @@ async fn ensure_indices_phase_b_failure_does_not_leak_sidecar_when_no_work_neede ) .await .unwrap(); + db.ensure_indices().await.unwrap(); } // Setup: trigger the failpoint. Steady-state ensure_indices @@ -4170,8 +5552,10 @@ async fn ensure_indices_phase_b_failure_does_not_leak_sidecar_when_no_work_neede // still fires, surfacing the Err. { let db = Omnigraph::open(&uri).await.unwrap(); - let _failpoint = - ScopedFailPoint::new(names::ENSURE_INDICES_POST_PHASE_B_PRE_MANIFEST_COMMIT, "return"); + let _failpoint = ScopedFailPoint::new( + names::ENSURE_INDICES_POST_PHASE_B_PRE_MANIFEST_COMMIT, + "return", + ); let err = db.ensure_indices().await.unwrap_err(); assert!( err.to_string().contains( @@ -4390,3 +5774,620 @@ async fn publisher_retries_retryable_load_publish_state_error() { .await .expect("publisher must retry the one-shot retryable load_publish_state error and commit"); } + +/// `create_branch` can succeed before reopening the new ref fails. That error is +/// post-effect even on the first deferred table: the v3 sidecar must remain so +/// Full recovery can reclaim the exact untouched ref. +#[tokio::test] +#[serial] +async fn first_touch_post_create_open_error_keeps_recovery_ownership() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + let db = helpers::init_and_load(&dir).await; + db.branch_create("feature").await.unwrap(); + + let error = { + let _fp = ScopedFailPoint::new(names::FORK_POST_CREATE_PRE_OPEN, "return"); + db.mutate( + "feature", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "post-create")], &[("$age", 22)]), + ) + .await + .expect_err("post-create reopen failure must fail into recovery") + }; + assert!(matches!(error, OmniError::RecoveryRequired { .. })); + assert_eq!( + std::fs::read_dir(dir.path().join("__recovery")) + .unwrap() + .count(), + 1, + "ambiguous post-create failure must retain its ownership sidecar" + ); + let person_uri = node_table_uri(&uri, "Person"); + assert!( + lance::Dataset::open(&person_uri) + .await + .unwrap() + .list_branches() + .await + .unwrap() + .contains_key("feature"), + "test seam fires only after the target ref is durable" + ); + + drop(db); + let recovered = Omnigraph::open(&uri).await.unwrap(); + assert_eq!( + helpers::count_rows_branch(&recovered, "feature", "node:Person").await, + 4, + "failed first touch must not publish its row" + ); + assert!( + !lance::Dataset::open(&person_uri) + .await + .unwrap() + .list_branches() + .await + .unwrap() + .contains_key("feature"), + "Full recovery must reclaim the sidecar-owned untouched ref" + ); +} + +/// A branch delete's first recovery probe is not its authority boundary. A data +/// writer can arm an intent in the prepare-to-gate gap; after delete acquires +/// schema/branch/all-table gates the owner is no longer live, so removing the +/// native ref safely makes that branch-local intent unreachable. The next heal +/// must audit/discard it instead of wedging the graph. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn branch_delete_orphans_sidecar_armed_after_initial_barrier() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + let initial = helpers::init_and_load(&dir).await; + initial.branch_create("feature").await.unwrap(); + drop(initial); + let delete_db = std::sync::Arc::new(Omnigraph::open(&uri).await.unwrap()); + let writer_db = Omnigraph::open(&uri).await.unwrap(); + + let branch_rv = + helpers::failpoint::Rendezvous::park_first(names::BRANCH_CONTROL_POST_RECOVERY_BARRIER); + let delete_handle = std::sync::Arc::clone(&delete_db); + let delete_task = tokio::spawn(async move { delete_handle.branch_delete("feature").await }); + branch_rv.wait_until_reached().await; + + { + let _fp = ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); + writer_db + .mutate( + "feature", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "late-sidecar")], &[("$age", 23)]), + ) + .await + .expect_err("writer must leave a confirmed sidecar in the branch-control gap"); + } + branch_rv.release(); + delete_task.await.unwrap().unwrap(); + assert!( + !delete_db + .branch_list() + .await + .unwrap() + .iter() + .any(|branch| branch == "feature"), + "branch delete must remove the authority that made the sidecar reachable" + ); + assert_eq!( + std::fs::read_dir(dir.path().join("__recovery")) + .unwrap() + .count(), + 1, + "delete leaves the orphaned intent for the audited recovery path" + ); + writer_db + .mutate( + "main", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "after-delete")], &[("$age", 24)]), + ) + .await + .expect("the next write must discard the dead-branch sidecar and proceed"); + assert_eq!( + std::fs::read_dir(dir.path().join("__recovery")) + .unwrap() + .count(), + 0, + "orphan-discard recovery must retire the late sidecar" + ); +} + +async fn setup_diverged_merge_branches(dir: &tempfile::TempDir) -> (String, usize) { + let uri = dir.path().to_str().unwrap().to_string(); + let db = helpers::init_and_load(dir).await; + let main_rows = helpers::count_rows(&db, "node:Person").await; + db.branch_create("source").await.unwrap(); + db.branch_create("target").await.unwrap(); + db.mutate( + "source", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "source-only")], &[("$age", 34)]), + ) + .await + .unwrap(); + db.mutate( + "target", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "old-target-only")], &[("$age", 35)]), + ) + .await + .unwrap(); + drop(db); + (uri, main_rows) +} + +/// A branch merge captures source/target authority before it builds a plan. +/// Once captured, native target delete+recreate must not reuse the same name +/// underneath that plan: both operations join the root-shared schema -> branch +/// gate order, so the control operation linearizes after the merge. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn branch_merge_fences_target_delete_recreate_aba() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let (uri, main_rows) = setup_diverged_merge_branches(&dir).await; + + // A recreated Lance ref can reuse the same branch name and numeric + // version; BranchIdentifier is the incarnation component that prevents + // that pair from masquerading as the authority captured by the merge. + let person_uri = node_table_uri(&uri, "Person"); + let old_target = lance::Dataset::open(&person_uri) + .await + .unwrap() + .checkout_branch("target") + .await + .unwrap(); + let old_target_version = old_target.version().version; + let old_target_identifier = old_target.branch_identifier().await.unwrap(); + + // Open both handles before the merge takes the schema gate. Open itself + // captures one coherent schema contract under that gate. + let merge_db = std::sync::Arc::new(Omnigraph::open(&uri).await.unwrap()); + let control_db = std::sync::Arc::new(Omnigraph::open(&uri).await.unwrap()); + + let merge_rv = + helpers::failpoint::Rendezvous::park_first(names::BRANCH_MERGE_POST_AUTHORITY_CAPTURE); + let control_rv = + helpers::failpoint::Rendezvous::park_first(names::BRANCH_CONTROL_POST_RECOVERY_BARRIER); + + let merge_handle = std::sync::Arc::clone(&merge_db); + let merge_task = tokio::spawn(async move { + merge_handle.branch_merge("source", "target").await + }); + merge_rv.wait_until_reached().await; + + let control_handle = std::sync::Arc::clone(&control_db); + let mut control_task = tokio::spawn(async move { + control_handle.branch_delete("target").await?; + control_handle.branch_create("target").await?; + control_handle + .mutate( + "target", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "replacement-only")], &[("$age", 36)]), + ) + .await?; + Ok::<(), OmniError>(()) + }); + control_rv.wait_until_reached().await; + control_rv.release(); + + // The control task is known to be immediately before its gate acquisition. + // It must remain blocked while merge holds the target-incarnation gate. + let control_blocked = tokio::time::timeout( + std::time::Duration::from_millis(250), + &mut control_task, + ) + .await + .is_err(); + let target_unchanged_while_parked = lance::Dataset::open(&person_uri) + .await + .unwrap() + .checkout_branch("target") + .await + .unwrap() + .branch_identifier() + .await + .unwrap() + == old_target_identifier; + // Always release before assertions so a failed oracle cannot strand the + // parked callback thread for its 30-second safety bound. + merge_rv.release(); + assert!( + control_blocked, + "target delete+recreate crossed a merge authority window (branch-name ABA)" + ); + assert!( + target_unchanged_while_parked, + "the target ref incarnation changed while merge authority was parked" + ); + + let outcome = merge_task.await.unwrap().unwrap(); + assert_eq!(outcome, omnigraph::db::MergeOutcome::Merged); + control_task.await.unwrap().unwrap(); + + let reopened = Omnigraph::open(&uri).await.unwrap(); + assert_eq!( + helpers::count_rows_branch(&reopened, "source", "node:Person").await, + main_rows + 1, + "source branch retains the source-only row" + ); + assert_eq!( + helpers::count_rows_branch(&reopened, "target", "node:Person").await, + main_rows + 1, + "the recreated target must contain only main plus its replacement row" + ); + let target_names = helpers::collect_column_strings( + &helpers::read_table_branch(&reopened, "target", "node:Person").await, + "name", + ); + assert!( + target_names.iter().any(|name| name == "replacement-only") + && !target_names.iter().any(|name| name == "source-only") + && !target_names.iter().any(|name| name == "old-target-only"), + "recreated target leaked state from the deleted target incarnation: {target_names:?}" + ); + let new_target = lance::Dataset::open(&person_uri) + .await + .unwrap() + .checkout_branch("target") + .await + .unwrap(); + assert_ne!( + new_target.branch_identifier().await.unwrap(), + old_target_identifier, + "delete+recreate must mint a new target incarnation" + ); + assert_eq!( + new_target.version().version, + old_target_version, + "the regression fixture must exercise same-name/same-version ABA" + ); +} + +/// `sync_branch` replaces a handle's active coordinator. It must join the same +/// schema authority gate held by branch merge, otherwise it can overwrite the +/// temporary target coordinator inside merge's swap -> publish -> restore +/// window and redirect physical effects or the merge commit. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn branch_merge_fences_concurrent_sync_on_same_handle() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let (uri, main_rows) = setup_diverged_merge_branches(&dir).await; + let db = Omnigraph::open(&uri).await.unwrap(); + db.branch_create("other").await.unwrap(); + let db = std::sync::Arc::new(db); + let merge_rv = + helpers::failpoint::Rendezvous::park_first(names::BRANCH_MERGE_POST_AUTHORITY_CAPTURE); + + let merge_handle = std::sync::Arc::clone(&db); + let merge_task = tokio::spawn(async move { + merge_handle.branch_merge("source", "target").await + }); + merge_rv.wait_until_reached().await; + + let sync_handle = std::sync::Arc::clone(&db); + let mut sync_task = tokio::spawn(async move { sync_handle.sync_branch("other").await }); + let sync_blocked = tokio::time::timeout( + std::time::Duration::from_millis(250), + &mut sync_task, + ) + .await + .is_err(); + merge_rv.release(); + assert!( + sync_blocked, + "sync replaced the active coordinator inside merge's authority window" + ); + + assert_eq!( + merge_task.await.unwrap().unwrap(), + omnigraph::db::MergeOutcome::Merged + ); + sync_task.await.unwrap().unwrap(); + assert_eq!( + helpers::count_rows_branch(&db, "target", "node:Person").await, + main_rows + 2, + "merge must publish both divergent rows to target before sync takes effect" + ); +} + +/// The post-table-gate merge check must read storage, not the swapped +/// coordinator's cached snapshot. Advance only the target manifest while merge +/// is parked after authority capture; the table version itself stays unchanged, +/// so only a fresh manifest-incarnation comparison catches the stale plan. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn branch_merge_rejects_fresh_target_manifest_change_before_effects() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let (uri, _) = setup_diverged_merge_branches(&dir).await; + let merge_db = std::sync::Arc::new(Omnigraph::open(&uri).await.unwrap()); + let mut target_writer = Omnigraph::open(&uri).await.unwrap(); + let merge_rv = + helpers::failpoint::Rendezvous::park_first(names::BRANCH_MERGE_POST_AUTHORITY_CAPTURE); + + let merge_handle = std::sync::Arc::clone(&merge_db); + let merge_task = tokio::spawn(async move { + merge_handle.branch_merge("source", "target").await + }); + merge_rv.wait_until_reached().await; + + let before = helpers::version_branch(&merge_db, "target").await.unwrap(); + // Advance the target's physical HEAD without changing row content, then + // publish that new pin through the legacy test seam. The merge handle's + // cached target snapshot remains at `before`. + let person_uri = node_table_uri(&uri, "Person"); + let mut raw_target = lance::Dataset::open(&person_uri) + .await + .unwrap() + .checkout_branch("target") + .await + .unwrap(); + helpers::lance_delete_inline(&mut raw_target, "1 = 2").await; + let publish_result = target_writer + .failpoint_publish_table_head_without_index_rebuild_for_test( + "target", + "node:Person", + Some("target"), + ) + .await; + let after_result = helpers::version_branch(&target_writer, "target").await; + // Release before asserting fixture setup so an unexpected setup error does + // not strand the parked callback thread. + merge_rv.release(); + publish_result.unwrap(); + let after = after_result.unwrap(); + assert!(after > before, "fixture must advance the target manifest"); + + let error = merge_task + .await + .unwrap() + .expect_err("merge must discard a plan prepared from the old target manifest"); + let OmniError::Manifest(manifest_error) = error else { + panic!("expected a typed read-set conflict"); + }; + assert!(matches!( + manifest_error.details, + Some(omnigraph::error::ManifestConflictDetails::ReadSetChanged { + ref member, + .. + }) if member == "branch_merge_target:target" + )); + assert!( + !dir.path().join("__recovery").exists() + || std::fs::read_dir(dir.path().join("__recovery")) + .unwrap() + .next() + .is_none(), + "pre-effect revalidation must fail before merge arms recovery" + ); +} + +/// A legacy writer can arm a relevant sidecar after merge's initial recovery +/// barrier. Merge acquires the complete source/target table envelope and lists +/// again before Phase A, so the late intent blocks it even when no table HEAD +/// moved and a version-only check would pass. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn branch_merge_rechecks_late_sidecar_after_table_gates() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let (uri, main_rows) = setup_diverged_merge_branches(&dir).await; + let merge_db = std::sync::Arc::new(Omnigraph::open(&uri).await.unwrap()); + let merge_rv = + helpers::failpoint::Rendezvous::park_first(names::BRANCH_MERGE_POST_AUTHORITY_CAPTURE); + + let merge_handle = std::sync::Arc::clone(&merge_db); + let merge_task = tokio::spawn(async move { + merge_handle.branch_merge("source", "target").await + }); + merge_rv.wait_until_reached().await; + + const OPERATION_ID: &str = "01H000000000000000000LATE"; + let recovery_dir = dir.path().join("__recovery"); + std::fs::create_dir_all(&recovery_dir).unwrap(); + std::fs::write( + recovery_dir.join(format!("{OPERATION_ID}.json")), + serde_json::to_string_pretty(&serde_json::json!({ + "schema_version": 1, + "operation_id": OPERATION_ID, + "started_at": "0", + "branch": "target", + "actor_id": null, + "writer_kind": "EnsureIndices", + "tables": [] + })) + .unwrap(), + ) + .unwrap(); + + merge_rv.release(); + let error = merge_task + .await + .unwrap() + .expect_err("late recovery ownership must block merge before effects"); + assert!(matches!( + error, + OmniError::RecoveryRequired { + ref operation_id, + .. + } if operation_id == OPERATION_ID + )); + assert_eq!( + helpers::count_rows_branch(&merge_db, "target", "node:Person").await, + main_rows + 1, + "blocked merge must leave the target image unchanged" + ); +} + +/// Cleanup's fast sidecar probe is only an optimization. A writer can fail +/// after that probe; the schema/branch/table GC envelope and final recheck must +/// refuse before Lance can delete recovery history. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn cleanup_rechecks_sidecars_under_gc_gates() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + drop(helpers::init_and_load(&dir).await); + + let cleanup_rv = helpers::failpoint::Rendezvous::park_first( + names::CLEANUP_POST_RECOVERY_CHECK_PRE_GATES, + ); + let cleanup_uri = uri.clone(); + let cleanup_task = tokio::spawn(async move { + let mut db = Omnigraph::open(&cleanup_uri).await.unwrap(); + db.cleanup(omnigraph::db::CleanupPolicyOptions { + keep_versions: Some(1), + older_than: None, + }) + .await + }); + cleanup_rv.wait_until_reached().await; + + let writer = Omnigraph::open(&uri).await.unwrap(); + { + let _fp = ScopedFailPoint::new(names::MUTATION_POST_FINALIZE_PRE_PUBLISHER, "return"); + writer + .mutate( + "main", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "gc-race")], &[("$age", 24)]), + ) + .await + .expect_err("writer must leave a sidecar after cleanup's fast probe"); + } + let person_uri = node_table_uri(&uri, "Person"); + let before_versions = lance::Dataset::open(&person_uri) + .await + .unwrap() + .versions() + .await + .unwrap() + .len(); + cleanup_rv.release(); + let error = cleanup_task + .await + .unwrap() + .expect_err("authoritative under-gate check must refuse cleanup"); + assert!(error.to_string().contains("after acquiring its GC gates")); + assert_eq!( + lance::Dataset::open(&person_uri) + .await + .unwrap() + .versions() + .await + .unwrap() + .len(), + before_versions, + "refused cleanup must not delete any recovery version history" + ); +} + +/// Full recovery must classify the sidecar body it re-reads after discovery, +/// not the stale parsed copy from its directory listing. Branch merge now holds +/// the schema gate for its complete authority window, so a second open cannot +/// discover its live sidecar underneath it. Instead, leave a confirmed crash +/// residual, park recovery after discovery, change that body to the valid +/// unconfirmed crash shape, and require the fresh body to drive rollback. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn full_recovery_rereads_sidecar_body_after_discovery() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + let db = helpers::init_and_load(&dir).await; + db.branch_create("feature").await.unwrap(); + db.mutate( + "feature", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "merge-feature")], &[("$age", 31)]), + ) + .await + .unwrap(); + db.mutate( + "main", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "merge-main")], &[("$age", 32)]), + ) + .await + .unwrap(); + drop(db); + + { + let _publish_failure = ScopedFailPoint::new( + names::BRANCH_MERGE_POST_PHASE_B_PRE_MANIFEST_COMMIT, + "return", + ); + Omnigraph::open(&uri) + .await + .unwrap() + .branch_merge("feature", "main") + .await + .expect_err("merge must leave a confirmed pre-publish sidecar"); + } + + let recovery_rv = + helpers::failpoint::Rendezvous::park_first(names::RECOVERY_POST_LIST_PRE_GATES); + let recovery_uri = uri.clone(); + let recovery_task = tokio::spawn(async move { Omnigraph::open(&recovery_uri).await }); + recovery_rv.wait_until_reached().await; + + let operation_id = single_sidecar_operation_id(dir.path()); + let sidecar_path = dir + .path() + .join("__recovery") + .join(format!("{operation_id}.json")); + let mut sidecar: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(&sidecar_path).unwrap()).unwrap(); + let tables = sidecar["tables"] + .as_array_mut() + .expect("branch-merge sidecar tables must be an array"); + assert!( + tables.iter().any(|table| !table["confirmed_version"].is_null()), + "fixture must begin with a confirmed BranchMerge residual" + ); + for table in tables { + table["confirmed_version"] = serde_json::Value::Null; + } + std::fs::write( + &sidecar_path, + serde_json::to_string_pretty(&sidecar).unwrap(), + ) + .unwrap(); + + recovery_rv.release(); + let recovered = recovery_task + .await + .unwrap() + .expect("full recovery must consume the freshly re-read body"); + assert_eq!( + helpers::count_rows(&recovered, "node:Person").await, + 5, + "fresh unconfirmed sidecar must roll the interrupted merge back; stale discovery would expose six rows" + ); +} diff --git a/crates/omnigraph/tests/helpers/cost.rs b/crates/omnigraph/tests/helpers/cost.rs index 05efe080..d9a51d76 100644 --- a/crates/omnigraph/tests/helpers/cost.rs +++ b/crates/omnigraph/tests/helpers/cost.rs @@ -202,7 +202,8 @@ impl ObjectStore for PrefixCountingStore { } fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, OSResult> { - self.counter.record_read(&prefix.cloned().unwrap_or_default()); + self.counter + .record_read(&prefix.cloned().unwrap_or_default()); self.target.list(prefix) } @@ -211,12 +212,14 @@ impl ObjectStore for PrefixCountingStore { prefix: Option<&Path>, offset: &Path, ) -> BoxStream<'static, OSResult> { - self.counter.record_read(&prefix.cloned().unwrap_or_default()); + self.counter + .record_read(&prefix.cloned().unwrap_or_default()); self.target.list_with_offset(prefix, offset) } async fn list_with_delimiter(&self, prefix: Option<&Path>) -> OSResult { - self.counter.record_read(&prefix.cloned().unwrap_or_default()); + self.counter + .record_read(&prefix.cloned().unwrap_or_default()); self.target.list_with_delimiter(prefix).await } diff --git a/crates/omnigraph/tests/helpers/mod.rs b/crates/omnigraph/tests/helpers/mod.rs index ada91266..d3490d37 100644 --- a/crates/omnigraph/tests/helpers/mod.rs +++ b/crates/omnigraph/tests/helpers/mod.rs @@ -61,6 +61,9 @@ pub async fn init_and_load(dir: &tempfile::TempDir) -> Omnigraph { load_jsonl(&mut db, TEST_DATA, LoadMode::Overwrite) .await .unwrap(); + // Mutation/load publish only exact data effects; physical indexes are + // reconciled separately as derived state. + db.ensure_indices().await.unwrap(); db } @@ -211,15 +214,18 @@ pub async fn commit_many(db: &mut Omnigraph, n: usize) { } } -/// Like [`commit_many`] but every commit carries an actor, so it grows -/// `_graph_commit_actors.lance` too — the authenticated (server/CLI) write path. +/// Like [`commit_many`] but every commit carries an actor in its inline +/// `__manifest` lineage row — the authenticated (server/CLI) write path. pub async fn commit_many_as(db: &mut Omnigraph, n: usize, actor: &str) { for i in 0..n { db.mutate_as( "main", MUTATION_QUERIES, "insert_person", - &mixed_params(&[("$name", &format!("commit_many_as_{i}"))], &[("$age", 30)]), + &mixed_params( + &[("$name", &format!("commit_many_as_{i}"))], + &[("$age", 30)], + ), Some(actor), ) .await diff --git a/crates/omnigraph/tests/helpers/recovery.rs b/crates/omnigraph/tests/helpers/recovery.rs index 4cb45e02..7db5efeb 100644 --- a/crates/omnigraph/tests/helpers/recovery.rs +++ b/crates/omnigraph/tests/helpers/recovery.rs @@ -13,8 +13,17 @@ const RECOVERY_ACTOR: &str = "omnigraph:recovery"; #[derive(Debug)] pub enum RecoveryExpectation { - RolledForward { tables: Vec }, - RolledBack { tables: Vec }, + RolledForward { + tables: Vec, + }, + /// Protocol-v3 mutation/load recovery republishes the writer's fixed + /// lineage intent rather than minting a synthetic recovery commit. + RolledForwardOriginalLineage { + tables: Vec, + }, + RolledBack { + tables: Vec, + }, Deferred, NoOp, } @@ -41,6 +50,7 @@ pub struct FollowUpMutation { struct RecoveryAuditRow { graph_commit_id: String, recovery_kind: String, + recovery_for_actor: Option, operation_id: String, sidecar_writer_kind: String, per_table_outcomes: Vec, @@ -201,7 +211,21 @@ pub async fn assert_post_recovery_invariants( ); assert_manifest_pins_match_lance_heads(graph_root, &tables).await?; assert_audit_to_versions_match_lance_heads(graph_root, &audit, &tables).await?; - assert_recovery_commit_shape(graph_root, &audit, &tables).await?; + assert_recovery_commit_shape(graph_root, &audit, &tables, false).await?; + assert_non_main_did_not_move_main(graph_root, &tables).await?; + assert_idempotent_reopen(graph_root, operation_id).await?; + run_follow_up_mutations(graph_root, tables).await?; + } + RecoveryExpectation::RolledForwardOriginalLineage { tables } => { + assert_sidecar_absent(graph_root, operation_id); + let audit = read_audit_row(graph_root, operation_id).await?; + assert_eq!( + audit.recovery_kind, "RolledForward", + "audit row for {operation_id} recorded the wrong recovery_kind", + ); + assert_manifest_pins_match_lance_heads(graph_root, &tables).await?; + assert_audit_to_versions_match_lance_heads(graph_root, &audit, &tables).await?; + assert_recovery_commit_shape(graph_root, &audit, &tables, true).await?; assert_non_main_did_not_move_main(graph_root, &tables).await?; assert_idempotent_reopen(graph_root, operation_id).await?; run_follow_up_mutations(graph_root, tables).await?; @@ -217,7 +241,7 @@ pub async fn assert_post_recovery_invariants( // Roll-back now publishes the restored HEAD, so manifest == Lance // HEAD afterward (symmetric with roll-forward) — no residual drift. assert_manifest_pins_match_lance_heads(graph_root, &tables).await?; - assert_recovery_commit_shape(graph_root, &audit, &tables).await?; + assert_recovery_commit_shape(graph_root, &audit, &tables, false).await?; assert_non_main_did_not_move_main(graph_root, &tables).await?; assert_idempotent_reopen(graph_root, operation_id).await?; run_follow_up_mutations(graph_root, tables).await?; @@ -366,19 +390,29 @@ async fn assert_recovery_commit_shape( graph_root: &Path, audit: &RecoveryAuditRow, tables: &[TableExpectation], + preserves_original_intent: bool, ) -> Result<()> { let branch = branch_context(tables); let expected_parent = expected_recovery_parent(tables)?; let branch = branch.as_deref(); let commit = read_recovery_commit(graph_root, audit, branch).await?; + // RFC-022 mutation/load roll-forward publishes the writer's original, + // pre-minted lineage intent. That preserves its original actor (including + // `None`) and makes a crash after publish distinguishable from a new + // synthetic recovery content commit. Rollback and legacy writer recovery + // still publish an explicit `omnigraph:recovery` commit. + let expected_actor = if preserves_original_intent { + audit.recovery_for_actor.as_deref() + } else { + Some(RECOVERY_ACTOR) + }; assert_eq!( commit.actor_id.as_deref(), - Some(RECOVERY_ACTOR), - "recovery commit {} for operation {} must use actor {}", + expected_actor, + "recovery commit {} for operation {} recorded the wrong actor", commit.graph_commit_id, audit.operation_id, - RECOVERY_ACTOR, ); if let Some(expected_parent) = expected_parent { @@ -573,6 +607,7 @@ async fn matching_audit_rows( for batch in batches { let graph_commit_ids = string_column(&batch, "graph_commit_id")?; let kinds = string_column(&batch, "recovery_kind")?; + let recovery_actors = string_column(&batch, "recovery_for_actor")?; let ops = string_column(&batch, "operation_id")?; let writers = string_column(&batch, "sidecar_writer_kind")?; let outcomes_json = string_column(&batch, "per_table_outcomes_json")?; @@ -589,6 +624,8 @@ async fn matching_audit_rows( rows.push(RecoveryAuditRow { graph_commit_id: graph_commit_ids.value(row).to_string(), recovery_kind: kinds.value(row).to_string(), + recovery_for_actor: (!recovery_actors.is_null(row)) + .then(|| recovery_actors.value(row).to_string()), operation_id: ops.value(row).to_string(), sidecar_writer_kind: writers.value(row).to_string(), per_table_outcomes, diff --git a/crates/omnigraph/tests/lineage_projection.rs b/crates/omnigraph/tests/lineage_projection.rs index 239beb16..c6b0b2c3 100644 --- a/crates/omnigraph/tests/lineage_projection.rs +++ b/crates/omnigraph/tests/lineage_projection.rs @@ -1,7 +1,7 @@ //! RFC-013 Phase 7 acceptance gate: graph lineage lives ONLY in `__manifest`. //! //! The `graph_commit` + `graph_head` rows ride the same publish CAS as the -//! table-version rows, so `_graph_commits.lance` carries NO commit rows. This +//! table-version rows, so no standalone commit-lineage dataset exists. This //! gate proves two things over a realistic history (commits on main, a branch, //! a merge, all with actors): //! diff --git a/crates/omnigraph/tests/maintenance.rs b/crates/omnigraph/tests/maintenance.rs index 448a7a2d..73a798a3 100644 --- a/crates/omnigraph/tests/maintenance.rs +++ b/crates/omnigraph/tests/maintenance.rs @@ -361,7 +361,8 @@ node Doc { let uri = dir.path().to_str().unwrap(); let mut db = Omnigraph::init(uri, SCHEMA).await.unwrap(); - // First load builds the id + rank BTREEs over the initial fragment. + // Loads publish only data effects; establish the initial id + rank BTREEs + // explicitly through the reconciler before creating partial coverage. load_jsonl( &mut db, "{\"type\":\"Doc\",\"data\":{\"slug\":\"d1\",\"rank\":1}}\n\ @@ -370,6 +371,7 @@ node Doc { ) .await .unwrap(); + db.ensure_indices().await.unwrap(); // A second load with NEW keys appends a fragment the existing BTREEs do not // cover (the existence gate skips re-building an index that already exists). @@ -411,13 +413,9 @@ node Doc { ); } -// Regression: `optimize` must not crash on a graph that has a `Blob` table. -// -// Lance `compact_files` forces `BlobHandling::AllBinary`, which mis-decodes -// blob-v2 columns ("more fields in the schema than provided column indices"), -// failing even a pristine uniform-V2_2 multi-fragment blob table. `optimize` -// must skip blob-bearing tables (and report the skip) rather than aborting the -// whole sweep. +// Regression: `optimize` must compact a graph that has a `Blob` table through +// the same positive path as every other data table; a reintroduced skip would +// hide both fragment growth and a Lance compatibility regression. // // History: through Lance 7.0.0 `compact_files` mis-decoded blob-v2 columns, so // `optimize` skipped blob tables (`SkipReason::BlobColumnsUnsupportedByLance`) @@ -431,16 +429,17 @@ node Doc { async fn optimize_compacts_blob_table_alongside_plain_table() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); - // One Blob node type (`Doc`) + one plain node type (`Tag`): proves the blob - // table is skipped while a non-blob table in the same sweep still compacts. + // One Blob node type (`Doc`) + one plain node type (`Tag`): proves both use + // the normal compaction path in the same sweep. let schema = "\ node Doc {\n slug: String @key\n content: Blob\n}\n\ node Tag {\n slug: String @key\n}\n"; let mut db = Omnigraph::init(uri, schema).await.unwrap(); // Multi-fragment blob table: Overwrite creates fragment 1; each Merge of - // new keys appends another. A >=2-fragment blob table is exactly what - // crashes `compact_files` today (single fragment would no-op and not crash). + // new keys appends another. A >=2-fragment blob table exercises the rewrite + // path that exposed the historical Lance regression (a single fragment + // would be a no-op). load_jsonl( &mut db, "{\"type\":\"Doc\",\"data\":{\"slug\":\"d1\",\"content\":\"base64:aGVsbG8x\"}}\n{\"type\":\"Doc\",\"data\":{\"slug\":\"d2\",\"content\":\"base64:aGVsbG8y\"}}", @@ -1133,15 +1132,12 @@ async fn cleanup_reconciles_live_branch_orphan_fork_but_keeps_legitimate_fork() } // Regression (iss-848): a table with rows but NULL vectors (the load-before- -// embed window) must not abort index building. The vector (IVF) index cannot -// train on 0 vectors, so `create_vector_index` errors with "KMeans cannot -// train 1 centroids with 0 vectors". `build_indices_on_dataset_for_catalog` -// is the chokepoint every caller funnels through (load/mutate via -// prepare_updates_for_commit, ensure_indices, optimize, schema apply, merge), -// so per-index fault isolation there must defer that one column (pending) and -// still build the sibling scalar indexes, instead of propagating the error. -// This exercises both the load path (which builds indices inline) and the -// ensure_indices reconciler. Pre-fix this fails at the load step. +// embed window) must remain writable and reconcilable. RFC-022-enrolled writes +// publish only their logical data effect; physical indexes are derived work. +// The vector (IVF) index cannot train on 0 vectors, so the index chokepoint must +// defer that column as pending while still building eligible sibling indexes. +// This exercises both halves of the contract: the logical load succeeds without +// inline index work, then `ensure_indices` tolerates the untrainable vector. #[tokio::test] async fn index_build_tolerates_null_vector_rows() { let dir = tempfile::tempdir().unwrap(); diff --git a/crates/omnigraph/tests/merge_fast_forward.rs b/crates/omnigraph/tests/merge_fast_forward.rs index 185f45d8..f52cd55f 100644 --- a/crates/omnigraph/tests/merge_fast_forward.rs +++ b/crates/omnigraph/tests/merge_fast_forward.rs @@ -127,8 +127,8 @@ async fn fast_forward_merge_defers_vector_index_to_reconciler() { let main = Omnigraph::init(uri, VEC_SCHEMA).await.unwrap(); main.branch_create("feature").await.unwrap(); - // Load embedding-bearing chunks onto the branch. The branch builds its own - // index here (outside the probe scope) — irrelevant to the merge's cost. + // Load embedding-bearing chunks onto the branch. Load publishes only the + // data effect, so the declared vector index remains pending here too. let mut rows = String::new(); for i in 0..24 { let v: Vec = (0..8).map(|j| format!("{}.0", (i + j) % 5)).collect(); @@ -140,7 +140,7 @@ async fn fast_forward_merge_defers_vector_index_to_reconciler() { let feature = Omnigraph::open(uri).await.unwrap(); feature.load("feature", &rows, LoadMode::Merge).await.unwrap(); - // Merge, counting inline vector-index builds the publish path performs. + // Merge, asserting that its publish path performs no inline vector-index build. let probes = MergeWriteProbes::default(); let outcome = with_merge_write_probes(probes.clone(), main.branch_merge("feature", "main")) .await diff --git a/crates/omnigraph/tests/recovery.rs b/crates/omnigraph/tests/recovery.rs index e1f47b2b..60c8cab8 100644 --- a/crates/omnigraph/tests/recovery.rs +++ b/crates/omnigraph/tests/recovery.rs @@ -1172,12 +1172,10 @@ async fn recovery_ensure_indices_steady_state_no_sidecar() { /// `count_rows == 0 → return false` short-circuit in `needs_index_work_*` /// is what makes this work. /// -/// A stronger assertion that captured the sidecar mid-flight and verified -/// the persisted JSON omits empty tables would require bypassing -/// `load_jsonl` (which auto-builds indices via -/// `prepare_updates_for_commit`); pinning that with a unit test on the -/// helpers directly would require bootstrapping an engine plus raw Lance -/// writes — left as a follow-up. +/// A stronger assertion that captured the sidecar after arming but before the +/// first index effect could inspect the persisted pin set directly. That needs +/// a dedicated pre-effect EnsureIndices rendezvous; the current failpoint is +/// after its effects, so this remains an end-to-end behavioral assertion. #[tokio::test] async fn recovery_ensure_indices_handles_empty_tables() { let dir = tempfile::tempdir().unwrap(); diff --git a/crates/omnigraph/tests/scalar_indexes.rs b/crates/omnigraph/tests/scalar_indexes.rs index 8d8a3f0a..bfa31054 100644 --- a/crates/omnigraph/tests/scalar_indexes.rs +++ b/crates/omnigraph/tests/scalar_indexes.rs @@ -30,8 +30,8 @@ const DATA: &str = r#"{"type":"Item","data":{"slug":"a","status":"active","publi {"type":"Item","data":{"slug":"b","status":"archived","published":"2023-01-01T00:00:00Z","rank":2,"title":"beta","note":"n2"}} {"type":"Item","data":{"slug":"c","status":"active","published":"2025-02-02T00:00:00Z","rank":3,"title":"gamma","note":"n3"}}"#; -// Enums and orderable scalars (DateTime, numeric) get a BTREE from load's -// build-indices pass, so a `=`/range filter on them uses the index. Free-text +// Enums and orderable scalars (DateTime, numeric) get a BTREE from the index +// reconciler, so a `=`/range filter on them uses the index. Free-text // String `@index` keeps FTS (no BTREE), and an un-annotated column has no // scalar index — both report `Degraded`, which is the negative control that // keeps this test from being vacuously green. @@ -41,6 +41,7 @@ async fn node_scalar_and_enum_index_columns_get_btree() { let uri = dir.path().to_str().unwrap(); let mut db = Omnigraph::init(uri, SCHEMA).await.unwrap(); load_jsonl(&mut db, DATA, LoadMode::Overwrite).await.unwrap(); + db.ensure_indices().await.unwrap(); let snap = snapshot_main(&db).await.unwrap(); let ds = snap.open("node:Item").await.unwrap(); diff --git a/crates/omnigraph/tests/schema_apply.rs b/crates/omnigraph/tests/schema_apply.rs index 83ba0ff3..0f70c01f 100644 --- a/crates/omnigraph/tests/schema_apply.rs +++ b/crates/omnigraph/tests/schema_apply.rs @@ -1,14 +1,17 @@ mod helpers; use std::fs; +#[cfg(feature = "failpoints")] +use std::sync::Arc; -use omnigraph::db::{Omnigraph, ReadTarget}; +use omnigraph::db::{MergeOutcome, Omnigraph, ReadTarget}; use omnigraph::loader::{LoadMode, load_jsonl}; use omnigraph_compiler::{SchemaMigrationStep, SchemaTypeKind}; use helpers::*; #[tokio::test] +#[cfg_attr(feature = "failpoints", serial_test::parallel)] async fn plan_schema_reports_supported_additive_change() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); @@ -33,6 +36,322 @@ async fn plan_schema_reports_supported_additive_change() { } #[tokio::test] +#[cfg_attr(feature = "failpoints", serial_test::parallel)] +async fn long_lived_handle_uses_the_schema_catalog_bound_to_its_write_token() { + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap(); + let schema_owner = Omnigraph::init(uri, TEST_SCHEMA).await.unwrap(); + // Open before the migration: this handle's process-local ArcSwap catalog is + // intentionally stale after `schema_owner` completes the apply. + let stale_handle = Omnigraph::open(uri).await.unwrap(); + + let desired = format!( + "{}\nnode Project {{\n name: String @key\n}}\n", + TEST_SCHEMA.replace( + " age: I32?\n}", + " age: I32?\n nickname: String?\n}", + ) + ); + schema_owner.apply_schema(&desired).await.unwrap(); + + let mutation = r#" +query insert_with_nickname($name: String, $age: I32, $nickname: String) { + insert Person { name: $name, age: $age, nickname: $nickname } +} +"#; + let inserted = stale_handle + .mutate( + "main", + mutation, + "insert_with_nickname", + &mixed_params( + &[("$name", "mutated-after-schema"), ("$nickname", "fresh")], + &[("$age", 31)], + ), + ) + .await + .expect("mutation must typecheck and build its batch with the token-bound catalog"); + assert_eq!(inserted.affected_nodes, 1); + + let loaded = stale_handle + .load( + "main", + r#"{"type":"Person","data":{"name":"loaded-after-schema","age":32,"nickname":"fresh"}}"#, + LoadMode::Merge, + ) + .await + .expect("load parsing and validation must use the same token-bound catalog"); + assert_eq!(loaded.nodes_loaded.get("Person"), Some(&1)); + assert_eq!(count_rows(&stale_handle, "node:Person").await, 2); + + // The same stale handle must bind branch-merge planning and conservative + // branch-control table gates to the accepted contract captured under the + // schema gate. The warm handle catalog predates Project; consulting it here + // would fail with `unknown node type` (or omit Project's control queue). + stale_handle.branch_create("source").await.unwrap(); + stale_handle.branch_create("target").await.unwrap(); + let project_mutation = r#" +query insert_project($name: String) { + insert Project { name: $name } +} +"#; + stale_handle + .mutate( + "source", + project_mutation, + "insert_project", + ¶ms(&[("$name", "fresh-catalog-project")]), + ) + .await + .expect("source write must use the token-bound post-apply catalog"); + assert_eq!( + stale_handle + .branch_merge("source", "target") + .await + .expect("merge planning must use the schema-gated post-apply catalog"), + MergeOutcome::FastForward + ); + assert_eq!( + count_rows_branch(&stale_handle, "target", "node:Project").await, + 1 + ); +} + +/// Native branch controls must enumerate their conservative table envelope +/// from the accepted catalog captured under the schema gate, not a long-lived +/// handle's pre-apply ArcSwap. Park delete after that envelope is held and prove +/// a legacy Project-only index reconciler cannot cross its table queue. +#[cfg(feature = "failpoints")] +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial_test::serial] +async fn stale_handle_branch_delete_gates_tables_added_by_schema_apply() { + use omnigraph::failpoints::names; + + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap(); + let schema_owner = Omnigraph::init(uri, TEST_SCHEMA).await.unwrap(); + let stale_control = Arc::new(Omnigraph::open(uri).await.unwrap()); + let desired = format!("{TEST_SCHEMA}\nnode Project {{\n name: String @key\n}}\n"); + schema_owner.apply_schema(&desired).await.unwrap(); + schema_owner.branch_create("target").await.unwrap(); + schema_owner + .load( + "target", + r#"{"type":"Project","data":{"name":"pending-index"}}"#, + LoadMode::Merge, + ) + .await + .unwrap(); + let index_reconciler = Arc::new(Omnigraph::open(uri).await.unwrap()); + + let delete_rv = + helpers::failpoint::Rendezvous::park_first(names::BRANCH_DELETE_POST_TABLE_GATES); + let delete_handle = Arc::clone(&stale_control); + let delete_task = tokio::spawn(async move { delete_handle.branch_delete("target").await }); + delete_rv.wait_until_reached().await; + + let index_handle = Arc::clone(&index_reconciler); + let mut index_task = tokio::spawn(async move { index_handle.ensure_indices_on("target").await }); + let index_blocked = tokio::time::timeout( + std::time::Duration::from_millis(250), + &mut index_task, + ) + .await + .is_err(); + delete_rv.release(); + assert!( + index_blocked, + "stale control catalog omitted the newly-added Project table gate" + ); + delete_task.await.unwrap().unwrap(); + + if tokio::time::timeout(std::time::Duration::from_secs(10), &mut index_task) + .await + .is_err() + { + index_task.abort(); + let _ = index_task.await; + panic!("index reconciler did not finish after branch delete released its table gate"); + } +} + +#[cfg(feature = "failpoints")] +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial_test::serial] +async fn mutation_waits_for_mid_apply_schema_gate_then_reprepares() { + use omnigraph::failpoints::names; + + let dir = tempfile::tempdir().unwrap(); + let db = Arc::new(init_and_load(&dir).await); + let desired = TEST_SCHEMA.replace( + " age: I32?\n}", + " age: I32?\n nickname: String?\n}", + ); + + // First park the mutation after all validation/staging but before it enters + // the schema→branch→table effect gates. This fixes the otherwise tiny race + // window deterministically. + let mutation_rv = + helpers::failpoint::Rendezvous::park_first(names::MUTATION_POST_STAGE_PRE_EFFECT_GATE); + let mutation_db = Arc::clone(&db); + let mutation_task = tokio::spawn(async move { + mutation_db + .mutate( + "main", + MUTATION_QUERIES, + "insert_person", + &mixed_params(&[("$name", "schema-gated")], &[("$age", 33)]), + ) + .await + }); + mutation_rv.wait_until_reached().await; + + // Start schema apply and park it after its staging files (and any table + // rewrite) exist but before manifest/schema promotion. The outer apply owns + // the schema-control gate throughout this window. + let schema_rv = + helpers::failpoint::Rendezvous::park_first(names::SCHEMA_APPLY_AFTER_STAGING_WRITE); + let schema_db = Arc::clone(&db); + let schema_task = tokio::spawn(async move { schema_db.apply_schema(&desired).await }); + schema_rv.wait_until_reached().await; + + mutation_rv.release(); + // Give the already-runnable mutation repeated scheduler turns. It must stay + // pending on the schema gate; completing here means it either advanced under + // an in-flight migration or returned a spurious post-prepare failure. + for _ in 0..128 { + tokio::task::yield_now().await; + if mutation_task.is_finished() { + break; + } + } + assert!( + !mutation_task.is_finished(), + "mutation must remain behind the schema-control gate while apply is in flight", + ); + + schema_rv.release(); + schema_task.await.unwrap().unwrap(); + let result = mutation_task + .await + .unwrap() + .expect("insert-only mutation must reprepare under the promoted schema"); + assert_eq!(result.affected_nodes, 1); + assert_eq!(count_rows(&db, "node:Person").await, 5); +} + +/// ReadOnly opens participate in the process-local schema publication gate even +/// though they perform no recovery writes. Park an open immediately before its +/// source/IR/state read: schema apply must not reach staging until that coherent +/// catalog capture finishes. +#[cfg(feature = "failpoints")] +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial_test::serial] +async fn read_only_open_holds_schema_gate_through_catalog_capture() { + use omnigraph::failpoints::names; + + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + let owner = Arc::new(init_and_load(&dir).await); + let desired = TEST_SCHEMA.replace( + " age: I32?\n}", + " age: I32?\n nickname: String?\n}", + ); + + let open_rv = + helpers::failpoint::Rendezvous::park_first(names::OPEN_BEFORE_SCHEMA_CONTRACT_READ); + let open_uri = uri.clone(); + let open_task = tokio::spawn(async move { Omnigraph::open_read_only(&open_uri).await }); + open_rv.wait_until_reached().await; + + let apply_rv = + helpers::failpoint::Rendezvous::park_first(names::SCHEMA_APPLY_AFTER_STAGING_WRITE); + let apply_owner = Arc::clone(&owner); + let apply_task = tokio::spawn(async move { apply_owner.apply_schema(&desired).await }); + assert!( + tokio::time::timeout( + std::time::Duration::from_millis(200), + apply_rv.wait_until_reached(), + ) + .await + .is_err(), + "schema apply must remain behind the ReadOnly catalog-capture gate", + ); + + open_rv.release(); + let opened = open_task.await.unwrap().unwrap(); + assert!( + !opened.catalog().node_types["Person"] + .properties + .contains_key("nickname"), + "the serialized open must publish the complete pre-apply catalog" + ); + apply_rv.wait_until_reached().await; + apply_rv.release(); + apply_task.await.unwrap().unwrap(); + let current = Omnigraph::open_read_only(&uri).await.unwrap(); + assert!( + current.catalog().node_types["Person"] + .properties + .contains_key("nickname"), + "the next open must publish the complete post-apply catalog" + ); +} + +/// Refresh must reacquire the schema gate after sidecar healing and retain it +/// through the ArcSwap publication. Otherwise a concurrent three-file schema +/// promotion can be interleaved with its contract read. +#[cfg(feature = "failpoints")] +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial_test::serial] +async fn refresh_holds_schema_gate_through_catalog_publication() { + use omnigraph::failpoints::names; + + let dir = tempfile::tempdir().unwrap(); + let owner = Arc::new(init_and_load(&dir).await); + let stale = Arc::new(Omnigraph::open(dir.path().to_str().unwrap()).await.unwrap()); + let desired = TEST_SCHEMA.replace( + " age: I32?\n}", + " age: I32?\n nickname: String?\n}", + ); + + let reload_rv = + helpers::failpoint::Rendezvous::park_first(names::SCHEMA_RELOAD_BEFORE_CONTRACT_READ); + let refresh_handle = Arc::clone(&stale); + let refresh_task = tokio::spawn(async move { refresh_handle.refresh().await }); + reload_rv.wait_until_reached().await; + + let apply_rv = + helpers::failpoint::Rendezvous::park_first(names::SCHEMA_APPLY_AFTER_STAGING_WRITE); + let apply_owner = Arc::clone(&owner); + let apply_task = tokio::spawn(async move { apply_owner.apply_schema(&desired).await }); + assert!( + tokio::time::timeout( + std::time::Duration::from_millis(200), + apply_rv.wait_until_reached(), + ) + .await + .is_err(), + "schema apply must remain behind refresh's catalog-publication gate", + ); + + reload_rv.release(); + refresh_task.await.unwrap().unwrap(); + apply_rv.wait_until_reached().await; + apply_rv.release(); + apply_task.await.unwrap().unwrap(); + + stale.refresh().await.unwrap(); + assert!( + stale.catalog().node_types["Person"] + .properties + .contains_key("nickname"), + "a post-apply refresh must publish the complete new catalog" + ); +} + +#[tokio::test] +#[cfg_attr(feature = "failpoints", serial_test::parallel)] async fn plan_schema_rejects_when_schema_contract_has_drifted() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); @@ -49,6 +368,7 @@ async fn plan_schema_rejects_when_schema_contract_has_drifted() { } #[tokio::test] +#[cfg_attr(feature = "failpoints", serial_test::parallel)] async fn apply_schema_noop_returns_not_applied() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); @@ -61,6 +381,7 @@ async fn apply_schema_noop_returns_not_applied() { } #[tokio::test] +#[cfg_attr(feature = "failpoints", serial_test::parallel)] async fn apply_schema_rejects_when_non_main_branch_exists() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); @@ -79,6 +400,7 @@ async fn apply_schema_rejects_when_non_main_branch_exists() { } #[tokio::test] +#[cfg_attr(feature = "failpoints", serial_test::parallel)] async fn apply_schema_unsupported_plan_does_not_advance_manifest() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); @@ -117,6 +439,7 @@ async fn apply_schema_unsupported_plan_does_not_advance_manifest() { // contract so a regression in the planner can't silently change behavior. #[tokio::test] +#[cfg_attr(feature = "failpoints", serial_test::parallel)] async fn apply_schema_drops_a_nullable_property_softly_preserves_prior_version() { let dir = tempfile::tempdir().unwrap(); let mut db = init_and_load(&dir).await; @@ -221,6 +544,7 @@ async fn apply_schema_drops_a_nullable_property_softly_preserves_prior_version() } #[tokio::test] +#[cfg_attr(feature = "failpoints", serial_test::parallel)] async fn apply_schema_drops_node_and_referencing_edge_softly() { let dir = tempfile::tempdir().unwrap(); let mut db = init_and_load(&dir).await; @@ -334,6 +658,7 @@ edge Knows: Person -> Person { } #[tokio::test] +#[cfg_attr(feature = "failpoints", serial_test::parallel)] async fn apply_schema_drops_an_edge_type_softly() { let dir = tempfile::tempdir().unwrap(); let mut db = init_and_load(&dir).await; @@ -391,6 +716,7 @@ async fn apply_schema_drops_an_edge_type_softly() { } #[tokio::test] +#[cfg_attr(feature = "failpoints", serial_test::parallel)] async fn apply_schema_rejects_adding_a_required_property_without_backfill() { let dir = tempfile::tempdir().unwrap(); let mut db = init_and_load(&dir).await; @@ -419,6 +745,7 @@ async fn apply_schema_rejects_adding_a_required_property_without_backfill() { } #[tokio::test] +#[cfg_attr(feature = "failpoints", serial_test::parallel)] async fn plan_schema_for_property_type_narrowing_is_not_supported() { // Symmetric companion to `apply_schema_unsupported_plan_does_not_advance_manifest`, // which exercises widening (I32 -> I64). Narrowing (I64 -> I32) is also @@ -446,6 +773,7 @@ async fn plan_schema_for_property_type_narrowing_is_not_supported() { } #[tokio::test] +#[cfg_attr(feature = "failpoints", serial_test::parallel)] async fn apply_schema_renames_node_type_via_rename_from_and_preserves_rows() { // Covers the stable-type-id contract: renaming a type preserves the // underlying Lance dataset (by stable id), so existing rows survive the @@ -535,6 +863,7 @@ edge WorksAt: Human -> Company // persists. Full orphan-dataset deletion is a separate follow-up. #[tokio::test] +#[cfg_attr(feature = "failpoints", serial_test::parallel)] async fn apply_schema_with_allow_data_loss_promotes_drops_to_hard() { let dir = tempfile::tempdir().unwrap(); let mut db = init_and_load(&dir).await; @@ -601,6 +930,7 @@ async fn apply_schema_with_allow_data_loss_promotes_drops_to_hard() { } #[tokio::test] +#[cfg_attr(feature = "failpoints", serial_test::parallel)] async fn apply_schema_hard_drops_property_makes_prior_version_unreachable() { let dir = tempfile::tempdir().unwrap(); let mut db = init_and_load(&dir).await; @@ -653,6 +983,7 @@ async fn apply_schema_hard_drops_property_makes_prior_version_unreachable() { } #[tokio::test] +#[cfg_attr(feature = "failpoints", serial_test::parallel)] async fn apply_schema_hard_drops_node_and_edge_with_flag_succeeds() { let dir = tempfile::tempdir().unwrap(); let mut db = init_and_load(&dir).await; @@ -737,19 +1068,14 @@ edge Knows: Person -> Person { // (no manifest entry), which is the user-facing guarantee. } -// Regression (bug 3 / dev-graph iss-848): a `Vector @index` on a 0-row table -// must not abort an otherwise-valid schema apply. A vector (IVF) index trains -// k-means centroids over the column's vectors, so Lance cannot build it on 0 -// vectors — it errors with "Creating empty vector indices with train=False is -// not yet implemented". When a *later* migration touches that table (here, an -// unrelated scalar `@index` on `body`), schema apply reconciles the table's -// whole index set, which previously tried to materialize the dormant vector -// index and aborted the entire migration (all-or-nothing). The build is now -// deferred (pending) when the column is untrainable, instead of failing the -// migration. The dormant index is materialized by a later `ensure_indices` / -// `optimize` once the table has rows. Full decoupling — intent recorded at -// apply, an async reconciler converges physical coverage — is iss-848. +// Regression (bug 3 / dev-graph iss-848): schema apply records index intent but +// performs no physical index work. That decoupling is load-bearing for a +// `Vector @index` on a 0-row table: Lance cannot train IVF centroids on no +// vectors, yet the logical migration must still succeed. A later +// `ensure_indices` / `optimize` materializes every buildable declaration once +// data exists and reports an untrainable vector column as pending meanwhile. #[tokio::test] +#[cfg_attr(feature = "failpoints", serial_test::parallel)] async fn apply_schema_defers_vector_index_on_empty_table() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); @@ -766,9 +1092,8 @@ async fn apply_schema_defers_vector_index_on_empty_table() { }\n"; let mut db = Omnigraph::init(uri, v1).await.unwrap(); - // Add an *unrelated* scalar @index on `body`. This routes Doc through - // schema apply's index reconcile, which must NOT abort on the untrainable - // empty vector index. + // Add an unrelated scalar @index on `body`. Schema apply must record both + // declarations without trying to build either one or train the empty vector. let v2 = "node Doc {\n \ slug: String @key\n \ body: String? @index\n \ @@ -779,11 +1104,8 @@ async fn apply_schema_defers_vector_index_on_empty_table() { ); assert!(result.applied, "the scalar @index change must apply"); - // The deferred vector index is not dropped — once the table has a - // trainable vector, `ensure_indices` materializes it without error. (If - // the guard wrongly skipped a non-empty column, this would still be - // unindexed; if it wrongly tried to build on empty, the apply above would - // have failed.) + // The deferred declarations are not dropped: after data arrives, the + // explicit reconciler materializes every buildable index without error. load_jsonl( &mut db, r#"{"type":"Doc","data":{"slug":"d1","body":"hello","embedding":[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8]}}"#, @@ -803,6 +1125,7 @@ async fn apply_schema_defers_vector_index_on_empty_table() { // optimize. Pre-iss-848 the indexed_tables block built the index inline and // bumped the table version. #[tokio::test] +#[cfg_attr(feature = "failpoints", serial_test::parallel)] async fn index_only_constraint_apply_touches_no_table_data() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); @@ -823,6 +1146,7 @@ async fn index_only_constraint_apply_touches_no_table_data() { .entry("node:Doc") .unwrap() .table_version; + let before_commits = db.list_commits(None).await.unwrap(); // Add an @index on the existing `n` column. let v2 = "node Doc {\n slug: String @key\n n: I64 @index\n}\n"; @@ -840,6 +1164,12 @@ async fn index_only_constraint_apply_touches_no_table_data() { before, after, "adding an @index must not bump the table version (no inline index build)" ); + let after_commits = db.list_commits(None).await.unwrap(); + assert_eq!( + after_commits.len(), + before_commits.len() + 1, + "metadata-only schema apply must still advance graph_head so it arbitrates concurrent prepared writes" + ); } // Enum widening (iss-enum-widening-migration): adding variants to an enum is @@ -847,6 +1177,7 @@ async fn index_only_constraint_apply_touches_no_table_data() { // touched, and the widened set is enforced immediately on writes. Narrowing // stays OG-MF-106-refused. #[tokio::test] +#[cfg_attr(feature = "failpoints", serial_test::parallel)] async fn enum_widening_apply_is_metadata_only_and_accepts_new_variant() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); @@ -867,6 +1198,7 @@ async fn enum_widening_apply_is_metadata_only_and_accepts_new_variant() { .entry("node:Ticket") .unwrap() .table_version; + let before_commits = db.list_commits(None).await.unwrap(); let v2 = "node Ticket {\n slug: String @key\n status: enum(todo, doing, done, blocked)\n}\n"; @@ -885,6 +1217,12 @@ async fn enum_widening_apply_is_metadata_only_and_accepts_new_variant() { before, after, "enum widening must not bump the table version (metadata-only)" ); + let after_commits = db.list_commits(None).await.unwrap(); + assert_eq!( + after_commits.len(), + before_commits.len() + 1, + "metadata-only enum widening must still advance graph_head" + ); // The NEW variant is accepted on the write path... load_jsonl( @@ -914,6 +1252,7 @@ async fn enum_widening_apply_is_metadata_only_and_accepts_new_variant() { } #[tokio::test] +#[cfg_attr(feature = "failpoints", serial_test::parallel)] async fn enum_narrowing_apply_is_refused() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); diff --git a/crates/omnigraph/tests/search.rs b/crates/omnigraph/tests/search.rs index 3d078772..4e7f39b2 100644 --- a/crates/omnigraph/tests/search.rs +++ b/crates/omnigraph/tests/search.rs @@ -86,6 +86,7 @@ async fn init_search_db(dir: &tempfile::TempDir) -> Omnigraph { load_jsonl(&mut db, SEARCH_DATA, LoadMode::Overwrite) .await .unwrap(); + db.ensure_indices().await.unwrap(); db } @@ -95,6 +96,7 @@ async fn init_mock_embedding_search_db(dir: &tempfile::TempDir) -> Omnigraph { load_jsonl(&mut db, &mock_embedding_seed_data(), LoadMode::Overwrite) .await .unwrap(); + db.ensure_indices().await.unwrap(); db } @@ -104,6 +106,7 @@ async fn init_model_recorded_search_db(dir: &tempfile::TempDir) -> Omnigraph { load_jsonl(&mut db, &mock_embedding_seed_data(), LoadMode::Overwrite) .await .unwrap(); + db.ensure_indices().await.unwrap(); db } @@ -202,6 +205,40 @@ async fn doc_user_index_count(db: &Omnigraph) -> usize { .count() } +/// RFC-022 data writes publish only their exact table effects. Declared FTS +/// and vector indexes may therefore still be pending immediately after load; +/// both retrieval modes (and their RRF composition) must remain logically +/// correct through Lance's flat-search paths. +#[tokio::test] +#[serial] +async fn deferred_indexes_do_not_block_hybrid_reads() { + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap(); + let mut db = Omnigraph::init(uri, MOCK_SEARCH_SCHEMA).await.unwrap(); + load_jsonl( + &mut db, + &mock_embedding_seed_data(), + LoadMode::Overwrite, + ) + .await + .unwrap(); + + assert_eq!( + doc_user_index_count(&db).await, + 0, + "load must leave declared physical indexes to the reconciler" + ); + let result = query_main( + &mut db, + MOCK_SEARCH_QUERIES, + "hybrid_search_vector", + &vector_and_string_params("$vq", &mock_embedding("alpha", 4), "$tq", "alpha"), + ) + .await + .expect("pending FTS/vector indexes must degrade to flat search"); + assert_eq!(result_slugs(&result)[0], "alpha-doc"); +} + struct EnvGuard { saved: Vec<(&'static str, Option)>, } @@ -853,7 +890,7 @@ async fn rrf_fuses_two_vector_queries() { #[tokio::test] #[serial] -async fn mutation_commit_refreshes_search_indices_without_manual_ensure() { +async fn mutation_with_deferred_index_coverage_remains_searchable() { let dir = tempfile::tempdir().unwrap(); let mut db = init_search_db(&dir).await; assert_eq!(doc_user_index_count(&db).await, 4); @@ -879,7 +916,7 @@ async fn mutation_commit_refreshes_search_indices_without_manual_ensure() { assert_eq!( doc_user_index_count(&db).await, 4, - "mutation commit should refresh required indices without duplicating them" + "mutation must leave physical index materialization to the reconciler" ); let result = query_main( @@ -892,7 +929,7 @@ async fn mutation_commit_refreshes_search_indices_without_manual_ensure() { .unwrap(); assert!( result_slugs(&result).contains(&"quasar-notes".to_string()), - "newly inserted row should be searchable without an explicit ensure_indices step" + "a row outside current index coverage must remain searchable via fallback scan" ); } @@ -919,7 +956,7 @@ async fn rrf_fuses_vector_and_text() { #[tokio::test] #[serial] -async fn load_commit_creates_vector_index_for_vector_annotations() { +async fn index_reconciler_creates_vector_index_for_vector_annotations() { let schema = r#" node Doc { slug: String @key @@ -935,6 +972,12 @@ node Doc { load_jsonl(&mut db, data, LoadMode::Overwrite) .await .unwrap(); + assert_eq!( + doc_user_index_count(&db).await, + 0, + "load publishes exact data effects and leaves physical indexes pending" + ); + db.ensure_indices().await.unwrap(); let ds = snapshot_main(&db) .await diff --git a/crates/omnigraph/tests/staged_writes.rs b/crates/omnigraph/tests/staged_writes.rs index 579d3655..52142305 100644 --- a/crates/omnigraph/tests/staged_writes.rs +++ b/crates/omnigraph/tests/staged_writes.rs @@ -450,6 +450,56 @@ async fn stage_merge_insert_commit_rebases_over_disjoint_committed_delete() { assert_eq!(collect_age_for_id(&batches, "p10"), None); } +#[tokio::test] +async fn exact_commit_exposes_lance_preflight_rebase_version() { + let dir = tempfile::tempdir().unwrap(); + let uri = format!("{}/people.lance", dir.path().to_str().unwrap()); + let store = TableStore::new(dir.path().to_str().unwrap(), test_session()); + + let ds = TableStore::write_dataset(&uri, person_batch(&[("alice", Some(30))])) + .await + .unwrap(); + let staged = store + .stage_append(&ds, person_batch(&[("bob", Some(25))]), &[]) + .await + .unwrap(); + let planned = staged.transaction_identity(); + assert_eq!(planned.read_version, 1); + + // A foreign append lands after staging. Even with max_retries(0), Lance + // performs one preflight conflict-resolution pass and can rebase this + // append before its sole manifest-write attempt. Lance preserves the + // transaction fields, so the exact path must expose the later achieved + // version as the second half of the identity check. + let mut foreign = ds.clone(); + lance_append_inline_local(&mut foreign, person_batch(&[("carol", Some(40))])).await; + let (committed, observed) = store + .commit_staged_exact(Arc::new(ds), staged) + .await + .unwrap(); + assert_eq!(committed.version().version, 3); + assert_eq!(observed, planned); + assert_ne!( + committed.version().version, + planned.read_version + 1, + "Lance preserves transaction identity during this preflight rebase; \ + the achieved version is the second half of exact-effect identity" + ); + + // Without a concurrent advance, the same one-attempt path lands the exact + // staged transaction identity. + let next = store + .stage_append(&committed, person_batch(&[("dave", Some(50))]), &[]) + .await + .unwrap(); + let next_planned = next.transaction_identity(); + let (_committed, next_observed) = store + .commit_staged_exact(Arc::new(committed), next) + .await + .unwrap(); + assert_eq!(next_observed, next_planned); +} + /// **Documented limitation** (see `scan_with_staged` doc): when a filter /// is supplied, Lance's stats-based pruning drops the staged fragment from /// the filtered scan because uncommitted fragments produced by @@ -1075,10 +1125,9 @@ async fn lance_restore_appends_one_commit_with_checked_out_content() { ); } -/// Empirical pin of the `Dataset::restore` concurrency hazard that -/// motivates the recovery sweep's open-time-only invocation strategy -/// and any future continuous-recovery reconciler's queue-acquisition -/// requirement. +/// Empirical pin of the `Dataset::restore` concurrency hazard that requires +/// Full recovery to join the root-scoped writer gates and leaves a real +/// cross-process fencing boundary. /// /// `Dataset::restore`'s `check_restore_txn` (lance-6.0.1 /// `src/io/commit/conflict_resolver.rs:986`) returns `Ok(())` against @@ -1091,18 +1140,17 @@ async fn lance_restore_appends_one_commit_with_checked_out_content() { /// rewind commit AFTER the legitimate concurrent Append, silently /// orphaning that Append's data from the active timeline. /// -/// The recovery sweep sidesteps this by running only at `Omnigraph::open` -/// (before any other writers can race). A future continuous-recovery -/// reconciler must acquire per-(table_key, branch) queues for sidecar -/// tables before invoking restore — otherwise this hazard becomes -/// reachable during in-flight tenant traffic. +/// Full recovery may invoke Restore on a read-write open, but first joins the +/// same root-scoped schema → branch → sorted-table gates as live writers and +/// rechecks that the sidecar still exists after waiting. The in-process healer +/// is roll-forward-only and never restores beneath live traffic. Together those +/// rules keep this substrate asymmetry unreachable against another handle in +/// the same process. /// -/// MR-686 introduces those per-(table_key, branch) writer queues as the -/// application-layer mechanism that closes this hazard once continuous -/// in-process recovery (MR-870) lands. Until MR-686's queue is wired into -/// the recovery path, the open-time-only invocation strategy is the -/// only thing keeping this hazard out of production. See -/// `docs/invariants.md` §VI.30, §VI.32, §VI.33. +/// The gates remain process-local. A recovery pass cannot fence a live writer +/// in another process, so general multi-process write/recovery topologies remain +/// outside the supported boundary until a distributed fence exists. See +/// `docs/dev/invariants.md` and `docs/dev/writes.md`. /// /// This test is the load-bearing constraint any future reconciler must /// honor. @@ -1159,10 +1207,10 @@ async fn lance_restore_loses_to_concurrent_append_via_orphaning() { ids, vec!["alice".to_string()], "Concurrent Append's row 'bob' was silently orphaned by the \ - Restore. Active-timeline contents == v1's contents. The recovery \ - sweep sidesteps this hazard via open-time-only invocation; any \ - future continuous-recovery reconciler must guard against it via \ - per-(table, branch) queue acquisition. Got: {:?}", + Restore. Active-timeline contents == v1's contents. Full recovery \ + must join the root-scoped writer gates before Restore; those gates \ + remain process-local, so a distributed fence is required before \ + multi-process recovery can be supported. Got: {:?}", ids, ); diff --git a/crates/omnigraph/tests/traversal_indexed.rs b/crates/omnigraph/tests/traversal_indexed.rs index ae500078..4c9404ca 100644 --- a/crates/omnigraph/tests/traversal_indexed.rs +++ b/crates/omnigraph/tests/traversal_indexed.rs @@ -58,7 +58,8 @@ async fn key_column_index_coverage_detects_btree_presence() { let db = init_and_load(&dir).await; let snap = snapshot_main(&db).await.unwrap(); - // Edge `src` gets a BTREE from ensure_indices on load → Indexed. + // The shared fixture explicitly reconciles indexes after loading, so the + // edge `src` BTREE is present and fully covered here. let edge_ds = snap.open("edge:Knows").await.unwrap(); let src_cov = TableStore::key_column_index_coverage(&edge_ds, "src") .await @@ -86,7 +87,8 @@ async fn coverage_degrades_for_appended_unindexed_fragment() { let dir = tempfile::tempdir().unwrap(); let mut db = init_and_load(&dir).await; - // Fresh load: the Knows BTREE covers every fragment → Indexed. + // The fixture's explicit post-load `ensure_indices` covers every current + // Knows fragment → Indexed. let snap = snapshot_main(&db).await.unwrap(); let edge_ds = snap.open("edge:Knows").await.unwrap(); assert_eq!( diff --git a/crates/omnigraph/tests/write_cost.rs b/crates/omnigraph/tests/write_cost.rs index f58d9d49..18edaf1e 100644 --- a/crates/omnigraph/tests/write_cost.rs +++ b/crates/omnigraph/tests/write_cost.rs @@ -5,8 +5,9 @@ //! boundary. Guards invariant 15 (cost bounded by work, not history) on writes. //! //! **Backend split (see docs/dev/testing.md / RFC-013).** This file runs on -//! **local FS** and gates the **internal-table** term (`__manifest`/`_graph_commits` -//! fragment scans, ~+18/depth — O(fragments) on any backend, step 2's target). +//! **local FS** and gates the **internal-table** term (`__manifest` fragment +//! scans, including inline lineage/actor rows — O(fragments) on any backend, +//! step 2's target). The former standalone lineage tables are retired. //! //! The **data-table opener** term (step 3a's win) is a per-object-store-RPC //! phenomenon and is NOT gated here: local-FS latest-resolution is cheap whether @@ -270,14 +271,16 @@ async fn keyed_insert_routes_through_merge_insert_only() { // ── (D) Step-3b capture-once fitness asserts (RED today → GREEN after WriteTxn) ── -/// A write must validate the schema contract EXACTLY ONCE (3 `read_text` + 2 `exists`). -/// Today the write path re-validates at every resolve point (entry, per-table -/// `resolved_branch_target`, commit-time `fresh_snapshot_for_branch`), so the delta is -/// a multiple of that. Step 3b's `WriteTxn` validates once and threads it. The shape is +/// A write performs one full schema validation while capturing the catalog-bound +/// `WriteTxn`, one trailing state-marker read that fences torn head/schema capture, +/// and one full validation under the pre-effect gates (7 `read_text` + 4 `exists` +/// total). Per-table resolves must not add more validation. The gate read is +/// correctness work: it arbitrates schema identity after preparation and before +/// the recovery sidecar or any Lance HEAD movement. The shape is /// the write twin of `warm_read_cost.rs::warm_query_validates_schema_contract_once`, /// built with ZERO production change via the counting storage adapter. #[tokio::test] -async fn write_validates_schema_contract_once() { +async fn write_schema_io_is_bounded_to_capture_fence_and_effect_gate() { use omnigraph::instrumentation::CountingStorageAdapter; use omnigraph::storage::storage_for_uri; @@ -291,6 +294,8 @@ async fn write_validates_schema_contract_once() { let before_read_text = counts.read_text(); let before_exists = counts.exists(); + let before_write_text = counts.write_text(); + let before_delete = counts.delete(); db.mutate( "main", MUTATION_QUERIES, @@ -302,14 +307,24 @@ async fn write_validates_schema_contract_once() { let read_text_delta = counts.read_text() - before_read_text; let exists_delta = counts.exists() - before_exists; + let write_text_delta = counts.write_text() - before_write_text; + let delete_delta = counts.delete() - before_delete; eprintln!("schema-contract reads on one write: read_text={read_text_delta} exists={exists_delta}"); assert_eq!( - read_text_delta, 3, - "a write must validate the schema contract once (3 reads), not N times", + read_text_delta, 7, + "a write must do capture validation + trailing identity fence + pre-effect validation (7 reads), not per table", ); assert_eq!( - exists_delta, 2, - "a write must probe contract-file existence once (2 probes), not N times", + exists_delta, 4, + "a write must probe contract-file existence at capture + pre-effect revalidation (4 probes)", + ); + assert_eq!( + write_text_delta, 2, + "an enrolled write must write its recovery sidecar exactly twice (arm + exact confirmation)", + ); + assert_eq!( + delete_delta, 1, + "a successful enrolled write must delete its confirmed sidecar once", ); } diff --git a/crates/omnigraph/tests/writes.rs b/crates/omnigraph/tests/writes.rs index aac44bf7..2181fd9d 100644 --- a/crates/omnigraph/tests/writes.rs +++ b/crates/omnigraph/tests/writes.rs @@ -1,13 +1,17 @@ -//! Tests for the direct-publish write path: mutations and loads write -//! directly to target tables and commit once via the publisher's -//! `expected_table_versions` CAS. (History: this replaced the removed Run +//! Tests for the direct-publish write path. Mutations and loads capture one +//! branch-wide authority token, prepare exact per-table transactions, then +//! acquire the root-shared schema → branch → sorted-table gates, revalidate, +//! arm schema-v3 recovery, commit the table effects, and publish once under the +//! same exact-head/table precondition. (History: this replaced the removed Run //! state machine / `__run__` staging branches / RunRecord — MR-771.) //! //! What this file covers: //! - No `__run__*` branches are created by load or mutate. //! - Cancellation of a mutation future leaves no graph-level state. -//! - Concurrent non-strict inserts/merges rebase under the per-table queue; -//! strict updates/deletes surface `ExpectedVersionMismatch` on stale state. +//! - A pre-effect branch-authority change makes an insert-only mutation or +//! Append/Merge load discard and fully reprepare with a bounded retry; strict +//! Update/Delete/Overwrite surfaces `ReadSetChanged`. Post-effect failures +//! require recovery. //! - Failed mutations and loads leave the target unchanged. //! - Multi-statement mutations are atomic (one commit per query). //! - actor_id propagates through to the commit graph. @@ -242,11 +246,12 @@ async fn partial_failure_leaves_target_queryable_and_unblocks_next_mutation() { assert_eq!(frank.num_rows(), 1, "Frank must be visible after publish"); } -/// Stale non-strict writers rebase to the live manifest pin under the -/// per-table queue instead of folding raw drift or returning a false 409. -/// Strict update/delete semantics are covered by the consistency/server tests. +/// Stale non-strict writers discard and reprepare their whole logical attempt +/// from the live branch authority instead of rebasing an already-validated +/// staged transaction or returning a false 409. Strict update/delete semantics +/// are covered by the consistency/server tests. #[tokio::test] -async fn stale_non_strict_insert_rebases_to_live_manifest_pin() { +async fn stale_non_strict_insert_reprepares_from_live_branch_state() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_string_lossy().into_owned(); @@ -276,9 +281,10 @@ async fn stale_non_strict_insert_rebases_to_live_manifest_pin() { } // Writer B's coordinator is still at the pre-A snapshot, but Insert is - // non-strict: commit_all re-reads the live manifest pin under the queue, - // verifies Lance HEAD equals that pin, and then lets Lance rebase the - // staged append. + // retryable: the RFC-022 adapter notices the authority change under the + // branch gate, discards B's prepared/staged attempt, and reruns the full + // operation from A's committed state. Lance never rebases a plan whose + // validation inputs are stale. db_b.mutate( "main", MUTATION_QUERIES, @@ -598,8 +604,16 @@ async fn overlapping_delete_predicates_do_not_double_count_affected() { ); // The data is correct regardless of the count: Bob + Diana remain. - assert_eq!(count_rows(&db, "node:Person").await, 2, "Bob and Diana remain"); - assert_eq!(count_rows(&db, "edge:Knows").await, 1, "only Bob→Diana remains"); + assert_eq!( + count_rows(&db, "node:Person").await, + 2, + "Bob and Diana remain" + ); + assert_eq!( + count_rows(&db, "edge:Knows").await, + 1, + "only Bob→Diana remains" + ); assert_eq!( count_rows(&db, "edge:WorksAt").await, 1, @@ -633,7 +647,9 @@ edge Knows: Person -> Person {"type":"Person","data":{"name":"Zoe"}} {"edge":"Knows","from":"Zoe","to":"Charlie"}"#; let mut db = Omnigraph::init(uri, schema).await.unwrap(); - load_jsonl(&mut db, data, LoadMode::Overwrite).await.unwrap(); + load_jsonl(&mut db, data, LoadMode::Overwrite) + .await + .unwrap(); let q = r#" query del_age_then_name($threshold: I32, $name: String) { @@ -668,7 +684,7 @@ query del_age_then_name($threshold: I32, $name: String) { /// `insert Person 'X'; update Person where name='X' set age=...` — both /// ops produce content on `node:Person` and coalesce into one /// `stage_merge_insert` at end-of-query. The accumulator's last-write-wins -/// dedupe (in `MutationStaging::finalize`) ensures the update's value +/// dedupe (during `MutationStaging::stage_all`) ensures the update's value /// wins. Single Lance commit per table per query. #[tokio::test] async fn mixed_insert_and_update_on_same_person_coalesces_to_one_merge() { @@ -692,7 +708,7 @@ async fn mixed_insert_and_update_on_same_person_coalesces_to_one_merge() { assert_eq!(result.affected_nodes, 2, "1 insert + 1 update reported"); // The end-state row carries the update value (last-write-wins via - // dedupe in finalize), proving the staged merge_insert ran with the + // end-of-query dedupe), proving the staged merge_insert ran with the // correct source dedupe. Read the underlying Person table directly // and assert age=99 for the row we just inserted+updated. let batches = read_table(&db, "node:Person").await; @@ -1063,7 +1079,7 @@ edge WorksAt: Person -> Company @card(0..1) /// `scan_with_pending`, the second update sees the stale committed value /// (the first update's row still appears in the Lance scan because the /// pending side hasn't committed), the predicate matches it, and the -/// dedupe-last-wins step at finalize ends up applying the second update +/// end-of-query dedupe-last-wins step ends up applying the second update /// to a row whose pending value should have shielded it. /// /// Concretely: Alice starts at age=30 in TEST_DATA. Op-1 sets Alice to @@ -1347,7 +1363,7 @@ edge WorksAt: Person -> Company @card(0..1) } /// A Merge load whose input has TWO rows with the same edge id must be -/// deduped at cardinality-count time, not just at finalize. Without +/// deduped at cardinality-count time, not just during end-of-query staging. Without /// dedup, two pending rows count twice → spurious `@card` violation. /// With dedup (last-occurrence-wins, mirroring /// `dedupe_merge_batches_by_id`), the pending side counts once. @@ -1382,7 +1398,7 @@ edge WorksAt: Person -> Company @card(0..1) .unwrap(); // Merge load with the SAME edge id twice — the second row supersedes - // the first in the finalize-time dedupe. If pending-counting doesn't + // the first in the end-of-query dedupe. If pending-counting doesn't // dedupe, Alice has 2 pending edges → @card(0..1) trips → load // fails. With dedupe, Alice has 1 → load succeeds. let dup_data = r#"{"edge": "WorksAt", "from": "Alice", "to": "Acme", "data": {"id": "w1"}} @@ -1490,21 +1506,13 @@ async fn scan_with_pending_rejects_key_column_missing_from_projection() { ); } -/// `PendingTable.schema` is captured from the first `append_batch` call -/// and never updated. On a blob-bearing table, an `insert` produces a -/// full-schema batch (blob columns included) and an `update` that -/// doesn't assign every blob produces a subset-schema batch. Mixed in -/// one query, the second `append_batch` would silently push an -/// incompatible batch — the mismatch surfaced eventually at -/// `concat_batches`/MemTable construction inside finalize, but the -/// failure point was distant from the offending op. -/// -/// `append_batch` validates the new batch's schema against the existing -/// accumulator's schema and returns a typed error directing the caller -/// to split the mutation. The error fires at the second op (the -/// update), not at end-of-query. +/// A blob-table insert followed by a non-blob update must remain one +/// full-schema merge stream. The update reads the just-inserted pending row, +/// copies its logical blob column through, and replaces the pending row under +/// the existing last-write-wins shadow semantics. This used to produce a +/// partial update batch and fail schema validation at the second op. #[tokio::test] -async fn append_batch_rejects_mismatched_schema_in_blob_table_at_offending_op() { +async fn blob_table_insert_then_non_blob_update_preserves_full_schema() { use omnigraph::loader::{LoadMode, load_jsonl}; const BLOB_SCHEMA: &str = r#" @@ -1527,10 +1535,9 @@ query insert_then_update_note( let uri = dir.path().to_str().unwrap(); let mut db = Omnigraph::init(uri, BLOB_SCHEMA).await.unwrap(); - // Seed with a Document so the update has something to match (the - // mid-query case is the chained-update scenario where the update's - // predicate matches the just-inserted row, exercising the in-memory - // pending union). + // Keep one committed blob row as well as the just-inserted pending row so + // the table has the real blob-v2 physical representation while this query + // exercises the pending union. load_jsonl( &mut db, r#"{"type":"Document","data":{"title":"seed","content":"base64:AQID"}}"#, @@ -1539,7 +1546,7 @@ query insert_then_update_note( .await .unwrap(); - let err = db + let result = db .mutate( "main", BLOB_QUERIES, @@ -1551,36 +1558,35 @@ query insert_then_update_note( ]), ) .await - .expect_err("blob-table mixed insert+update with non-fully-assigned blob must error early"); - let OmniError::Manifest(manifest_err) = err else { - panic!("expected Manifest error, got {err:?}"); - }; - assert!( - manifest_err.message.contains("mismatched schemas") - && manifest_err.message.contains("Split the mutation"), - "error must direct user to split: {}", - manifest_err.message, + .expect("blob-table insert + non-blob update must share a full-schema merge batch"); + assert_eq!( + result.affected_nodes, 2, + "one insert plus one pending-row update" ); - // Confirm the manifest didn't advance — early error must be - // before any commit. + let blob = db + .read_blob("Document", "letter", "content") + .await + .expect("inserted blob must remain readable after the update"); + assert_eq!(&blob.read().await.unwrap()[..], &[4, 5, 6]); + let qr = db .query( ReadTarget::branch("main"), r#"query get_doc($title: String) { match { $d: Document { title: $title } } - return { $d.title } + return { $d.title, $d.note } }"#, "get_doc", ¶ms(&[("$title", "letter")]), ) .await .unwrap(); - assert_eq!( - qr.num_rows(), - 0, - "letter must not be visible after early error" - ); + assert_eq!(qr.num_rows(), 1); + let json = qr.to_sdk_json(); + let row = json.as_array().unwrap().first().unwrap(); + assert_eq!(row["d.title"], "letter"); + assert_eq!(row["d.note"], "draft 1"); } /// MR-920 regression: two sequential `update T set {f:v} where x=y` @@ -1794,7 +1800,9 @@ async fn camelcase_mutation_predicate_updates_and_deletes() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); let mut db = Omnigraph::init(uri, CC_SCHEMA).await.unwrap(); - load_jsonl(&mut db, CC_DATA, LoadMode::Overwrite).await.unwrap(); + load_jsonl(&mut db, CC_DATA, LoadMode::Overwrite) + .await + .unwrap(); let m = r#" query set_status($repo: String, $st: String) { update Doc set { status: $st } where repoName = $repo } @@ -1802,7 +1810,12 @@ query del($repo: String) { delete Doc where repoName = $repo } "#; let upd = db - .mutate("main", m, "set_status", ¶ms(&[("$repo", "acme"), ("$st", "closed")])) + .mutate( + "main", + m, + "set_status", + ¶ms(&[("$repo", "acme"), ("$st", "closed")]), + ) .await .expect("update with a camelCase predicate must execute"); assert_eq!(upd.affected_nodes, 1, "exactly the acme Doc should update"); @@ -1811,9 +1824,16 @@ query del($repo: String) { delete Doc where repoName = $repo } .mutate("main", m, "del", ¶ms(&[("$repo", "globex")])) .await .expect("delete with a camelCase predicate must execute"); - assert_eq!(del.affected_nodes, 1, "exactly the globex Doc should delete"); + assert_eq!( + del.affected_nodes, 1, + "exactly the globex Doc should delete" + ); - assert_eq!(count_rows(&db, "node:Doc").await, 1, "one Doc (acme) should remain"); + assert_eq!( + count_rows(&db, "node:Doc").await, + 1, + "one Doc (acme) should remain" + ); } // #283 (pending side): a chained mutation whose 2nd op filters a camelCase @@ -1825,7 +1845,9 @@ async fn camelcase_chained_mutation_reads_pending_by_camelcase() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); let mut db = Omnigraph::init(uri, CC_SCHEMA).await.unwrap(); - load_jsonl(&mut db, CC_DATA, LoadMode::Overwrite).await.unwrap(); + load_jsonl(&mut db, CC_DATA, LoadMode::Overwrite) + .await + .unwrap(); // op-1 stages a status change to the acme Doc; op-2 re-filters the same // camelCase column, so it must match op-1's pending row. @@ -1838,8 +1860,13 @@ query chain($repo: String) { let r = db .mutate("main", m, "chain", ¶ms(&[("$repo", "acme")])) .await - .expect("chained camelCase mutation must read the pending row, not fail at the MemTable SELECT"); - assert_eq!(r.affected_nodes, 2, "both ops should touch the acme Doc (read-your-writes)"); + .expect( + "chained camelCase mutation must read the pending row, not fail at the MemTable SELECT", + ); + assert_eq!( + r.affected_nodes, 2, + "both ops should touch the acme Doc (read-your-writes)" + ); } /// A zero-row cascade delete must not advance an edge table's Lance HEAD past @@ -1992,7 +2019,9 @@ async fn filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent() ) }) .collect(); - load_jsonl(&mut db, &updates, LoadMode::Merge).await.unwrap(); + load_jsonl(&mut db, &updates, LoadMode::Merge) + .await + .unwrap(); // The delete adds a deletion vector, so the overlapping region no longer // densely tiles its id range — the shape lance#7444 choked on. diff --git a/docs/dev/architecture.md b/docs/dev/architecture.md index 2469c84c..84d80321 100644 --- a/docs/dev/architecture.md +++ b/docs/dev/architecture.md @@ -170,30 +170,36 @@ Code paths: - Graph index: `crates/omnigraph/src/graph_index/` - Loader: `Omnigraph::ingest` at `crates/omnigraph/src/loader/mod.rs:74` -### Mutation atomicity — in-memory accumulator (MR-794) +### Mutation atomicity — in-memory accumulator + branch-wide OCC (MR-794 / RFC-022) -Inserts and updates inside `mutate_as` and the bulk loader's -Append/Merge modes go through `MutationStaging` +Mutations inside `mutate_as` and every bulk-load mode go through `MutationStaging` ([`crates/omnigraph/src/exec/staging.rs`](../../crates/omnigraph/src/exec/staging.rs)), a per-query in-memory accumulator. No Lance HEAD advance happens during -op execution; one `stage_*` + `commit_staged` per touched table runs -at end-of-query, then the publisher commits the manifest atomically. +op execution. The attempt captures one native branch identity, exact graph head, +accepted schema identity/catalog, and base snapshot. At end-of-query it stages +one exact Lance transaction per touched table outside the effect gates, then +acquires the root-shared schema → branch → sorted-table gates and revalidates the +complete authority before arming recovery or advancing any table HEAD. ``` op-1 (insert/update) → push RecordBatch → MutationStaging.pending[table] op-2 (insert/update) → read committed via Lance + pending via DataFusion MemTable (read-your-writes) → push batch op-N → push batch -─── end of query ─────────────────────────────────────── -finalize: per pending table: - concat batches → stage_append OR stage_merge_insert OR stage_overwrite - → commit_staged -publisher: ManifestBatchPublisher::publish (one cross-table CAS) +─── end of query ──────────────────────────────────────────────── +stage_all: concat batches → one exact stage_* transaction per table (no HEAD move) +commit_all: acquire schema → branch → sorted-table gates + → reject a new recovery intent → revalidate the complete branch token + → arm schema-v3 recovery → commit_staged per table +publisher: publish pre-minted lineage under the exact native-branch/head + table precondition ``` -A failed op leaves Lance HEAD untouched on the staged tables: the next -mutation proceeds normally with no drift to reconcile. Concrete -contracts: +A failed op leaves Lance HEAD untouched on the staged tables. If authority +changes before effects, insert-only mutations and Append/Merge loads discard the +whole attempt and fully reprepare with a bounded retry; Update/Delete/Overwrite +returns `ReadSetChanged`. After any physical effect, any later error returns +`RecoveryRequired` and leaves the v3 sidecar to converge the fixed outcome. +Concrete contracts: - `D₂` parse-time rule: a query is either insert/update-only or delete-only. Mixed → reject. Deletes now stage like inserts/updates @@ -262,7 +268,7 @@ flowchart LR rec --> diff --> wp ``` -Today, indexes are built explicitly via `ensure_indices`. Reads degrade gracefully when index coverage is partial — Lance's scanner unions indexed and scan paths automatically. The roadmap reconciler observes manifest state and converges coverage in the background. +Today, physical indexes are built explicitly via `ensure_indices`/`optimize`; schema apply, mutation, and load only record intent or publish their exact data effects and never build indexes inline. Reads degrade gracefully when index coverage is missing or partial — Lance's scanner unions indexed and scan paths automatically (vector search falls back to brute force). A future background reconciler may automate the same explicit convergence path. ### Server / CLI @@ -276,14 +282,14 @@ flowchart LR pol[Cedar policy gate
per request]:::l2 wl[WorkloadController
per-actor admission]:::l2 eng[engine API
Arc<Omnigraph>]:::l2 - wq[WriteQueueManager
per-(table, branch)]:::l2 + wq[WriteQueueManager
root-scoped schema → branch → table gates]:::l2 cli -.-> eng srv_in --> auth --> pol --> wl --> eng eng --> wq ``` -The server applies Cedar policy at the HTTP boundary today. The roadmap, called out in [docs/dev/invariants.md](invariants.md) as a known gap, is to push policy into the planner as predicates. After Cedar, mutating handlers go through `WorkloadController` (per-actor admission cap + byte budget; PR 2 / MR-686) before reaching the engine. The engine itself holds an `Arc` so concurrent mutations on the same `(table, branch)` serialize at the queue, while disjoint keys run in parallel — see [docs/user/server.md](../user/operations/server.md) "Per-actor admission control" and [docs/dev/writes.md](writes.md). The CLI bypasses the HTTP layer (and admission) and calls the engine API directly. +The server applies Cedar policy at the HTTP boundary today. The roadmap, called out in [docs/dev/invariants.md](invariants.md) as a known gap, is to push policy into the planner as predicates. After Cedar, mutating handlers go through `WorkloadController` (per-actor admission cap + byte budget; PR 2 / MR-686) before reaching the engine. Every handle for one canonical graph root shares an `Arc` in-process. RFC-022-enrolled mutation/load attempts prepare outside the effect gates, then acquire the exclusive root schema gate, the target branch-effect gate, and sorted `(table, branch)` gates and hold them through publish. The root schema gate means enrolled effect windows on one graph currently serialize even across different branches; readers remain ungated, and pre-effect parsing, validation, and fragment staging can overlap. The branch gate preserves the complete per-branch validation authority, while table gates protect each concrete Lance effect and legacy adapter. These gates prevent same-process races and impose one deadlock-free order; the exact publisher precondition and durable recovery intent remain the correctness authorities, and the gates are not a cross-process lock. See [docs/user/server.md](../user/operations/server.md) "Per-actor admission control" and [docs/dev/writes.md](writes.md). The CLI bypasses the HTTP layer (and admission) and calls the engine API directly. Code paths: @@ -303,8 +309,8 @@ Throughout the docs, capabilities are split into: - **MVCC**: every Lance write bumps a per-dataset version; the OmniGraph manifest version coordinates which sub-table versions are visible together. - **Snapshot isolation**: a query holds one `Snapshot` for its lifetime; concurrent writes don't leak in. -- **Cross-branch isolation**: copy-on-write means readers and writers on different branches don't block each other. -- **Per-query staging**: `mutate_as` and `load` (Append/Merge) accumulate insert/update batches in an in-memory `MutationStaging`; one `stage_*` + `commit_staged` per touched table runs at end-of-query, then the publisher commits the manifest atomically. A mid-query failure leaves Lance HEAD untouched on staged tables. (MR-794; pre-v0.4.0 used a `__run__` staging branch + Run state machine, removed in MR-771.) +- **Cross-branch isolation**: Lance copy-on-write keeps branch data independent, and readers never acquire write gates. Enrolled writer preparation can overlap across branches, but their effect/publish windows currently share one exclusive root schema gate before the branch/table gates, so those windows serialize in-process even for different branches. +- **Per-query staging**: `mutate_as` and every `load` mode accumulate their work in an in-memory `MutationStaging`; `stage_all` prepares exact per-table transactions without moving HEAD. At commit, the root schema → branch → sorted-table gates protect complete-token revalidation, v3 recovery arming, zero-retry table effects, and one atomic manifest publish. A mid-query failure leaves Lance HEAD untouched on staged tables. (MR-794 / RFC-022; pre-v0.4.0 used a `__run__` staging branch + Run state machine, removed in MR-771.) - **Schema-apply lock**: `__schema_apply_lock__` system branch serializes schema migrations. - **Fail-points** (`failpoints` cargo feature): `failpoints::maybe_fail("operation.step")?` in `branch_create`, publish, etc., for deterministic failure injection in tests. diff --git a/docs/dev/execution.md b/docs/dev/execution.md index 0430f25d..386aa751 100644 --- a/docs/dev/execution.md +++ b/docs/dev/execution.md @@ -79,16 +79,16 @@ Hybrid example: `order { rrf(nearest($d.embedding, $q), bm25($d.body, $q_text)) ## Mutation execution (`exec/mutation.rs`) -Resolves expression values to literals, converts to typed Arrow arrays (`literal_to_typed_array(lit, DataType, num_rows)`), then writes via Lance's two-phase distributed-write API at end-of-query: +Resolves expression values to literals, converts to typed Arrow arrays (`literal_to_typed_array(lit, DataType, num_rows)`), then writes via Lance's two-phase distributed-write API at end-of-query. Before lowering/execution, one `WriteTxn` captures the target's Lance-native branch identity, exact optional graph head, accepted schema identity/catalog, and base table snapshot; every step in the attempt uses that immutable authority. -- `insert` (no `@key`, edges) → accumulate into `MutationStaging.pending` (Append mode); finalize calls `stage_append` once per touched table. -- `insert` (`@key` node) → accumulate into `pending` (Merge mode); finalize calls `stage_merge_insert` once per touched table. +- `insert` (no `@key`, edges) → accumulate into `MutationStaging.pending` (Append mode); `stage_all` later calls `stage_append` once per touched table. +- `insert` (`@key` node) → accumulate into `pending` (Merge mode); `stage_all` later calls `stage_merge_insert` once per touched table. - `update` → scan committed via Lance + pending via DataFusion `MemTable` (read-your-writes), apply assignments, accumulate into `pending` (Merge mode). -- `delete` → records a predicate into `MutationStaging.delete_predicates` (count matching committed rows now for `affected_*`); finalize combines a table's predicates into one `stage_delete` (Lance 7.0 `DeleteBuilder::execute_uncommitted`, a deletion-vector transaction) committed via `commit_staged` — no inline HEAD advance (MR-A). +- `delete` → records a predicate into `MutationStaging.delete_predicates` (count matching committed rows now for `affected_*`); `stage_all` combines a table's predicates into one `stage_delete` (Lance 7.0 `DeleteBuilder::execute_uncommitted`, a deletion-vector transaction) — no inline HEAD advance (MR-A). **D₂ parse-time rule.** A single mutation query is either insert/update-only or delete-only. Mixed → reject before any I/O. The check fires in `enforce_no_mixed_destructive_constructive(&ir)` inside `execute_named_mutation`. -Multi-statement mutations are atomic at the publisher commit boundary: every insert/update batch lives in memory until end-of-query, then exactly one `stage_*` + `commit_staged` runs per touched table, then `ManifestBatchPublisher::publish` commits the manifest atomically with per-table `expected_table_versions` CAS. +Multi-statement mutations are atomic at the publisher commit boundary. Every batch lives in memory until all statements and validation succeed; `stage_all` then prepares one exact transaction per touched table without advancing HEAD. `commit_all` acquires the root-shared schema → branch → sorted-table gates, rechecks for recovery intent, revalidates the complete branch authority, writes the schema-v3 recovery sidecar, and commits the table transactions with zero transparent conflict retries. The guards remain held while `ManifestBatchPublisher` publishes the pre-minted lineage under the same exact native-branch/head and table-version precondition. ### Mutation flow — sequence @@ -100,11 +100,12 @@ sequenceDiagram participant cmp as omnigraph-compiler participant stg as MutationStaging
(exec/staging.rs) participant ts as table_store - participant lance as Lance dataset + participant rec as schema-v3 recovery sidecar participant pub as ManifestBatchPublisher client->>og: mutate_as(branch, source, name, params, actor_id) - og->>cmp: parse + typecheck + lower_mutation_query + og->>og: heal/reject recovery intent; open_write_txn + og->>cmp: parse + typecheck + lower using txn catalog cmp-->>og: MutationIR og->>og: enforce_no_mixed_destructive_constructive (D₂) loop for each mutation op @@ -116,23 +117,43 @@ sequenceDiagram og->>ts: scan_with_pending (Lance + DataFusion MemTable union) ts-->>og: matched batches end - else delete (stage, D₂ keeps separate) + else delete (record predicate; D₂ keeps separate) og->>ts: count_rows (committed match → affected_*) og->>stg: ensure_path + record_delete (predicate) end end - og->>stg: finalize(db, branch) - loop per pending table - stg->>ts: stage_append OR stage_merge_insert (one per table) - ts-->>stg: StagedWrite (transaction + commit metadata + fragments) - stg->>ts: commit_staged (advances Lance HEAD) - ts-->>stg: new Dataset + og->>og: validate complete staged change-set against txn base + og->>stg: stage_all(db, branch) + loop per touched table + stg->>ts: stage_append OR stage_merge_insert OR stage_delete (one per table) + ts-->>stg: exact staged transaction (no HEAD movement) + end + stg->>stg: acquire schema → branch → sorted-table gates + stg->>og: recheck recovery barrier + revalidate complete WriteTxn + alt authority changed before effects + stg-->>og: ReadSetChanged + alt insert-only mutation + og->>og: discard complete attempt; bounded full reprepare + else Update/Delete + og-->>client: ReadSetChanged (409) + end + else authority unchanged + stg->>rec: persist fixed lineage + exact transaction identities + loop per touched table + stg->>ts: commit_staged (zero transparent retries) + ts-->>stg: confirm achieved transaction + table update + end + stg-->>og: updates + expected versions + sidecar + held gates + og->>pub: publish exact graph-head/table precondition + alt publish succeeds + pub-->>og: new manifest version + og->>rec: delete sidecar + og-->>client: MutationResult + else any error after an effect + pub-->>og: error + og-->>client: RecoveryRequired (sidecar remains authoritative) + end end - stg-->>og: (updates: Vec, expected_versions) - og->>pub: commit_updates_on_branch_with_expected - pub->>pub: publisher CAS (cross-table OCC on __manifest) - pub-->>og: new manifest version - og-->>client: MutationResult ``` **Code paths:** @@ -143,11 +164,11 @@ sequenceDiagram - Per-op execution: `execute_insert`, `execute_update`, `execute_delete_node`, `execute_delete_edge` - Pending-aware reads: `TableStore::scan_with_pending` / `count_rows_with_pending` at `crates/omnigraph/src/table_store.rs` - Edge cardinality with pending: `validate_edge_cardinality_with_pending` at `crates/omnigraph/src/exec/mutation.rs` -- Per-query accumulator: `crates/omnigraph/src/exec/staging.rs` (`MutationStaging`, `PendingTable`, `PendingMode`, `finalize`) -- End-of-query Lance commit: `TableStore::stage_append`, `stage_merge_insert`, `commit_staged` at `crates/omnigraph/src/table_store.rs` -- Manifest commit primitive: `commit_updates_on_branch_with_expected` at `crates/omnigraph/src/db/omnigraph/table_ops.rs` +- Per-query accumulator and protocol adapter: `crates/omnigraph/src/exec/staging.rs` (`MutationStaging::stage_all`, `StagedMutation::commit_all`) +- End-of-query Lance operations: `TableStore::stage_append`, `stage_merge_insert`, `stage_delete`, `commit_staged` at `crates/omnigraph/src/table_store.rs` +- Manifest commit primitive: `commit_updates_on_branch_with_expected` at `crates/omnigraph/src/db/omnigraph/table_ops.rs` (exact native-branch/head precondition plus expected table versions) -Atomicity guarantee for multi-statement mutations: a mid-query failure leaves Lance HEAD untouched on staged tables (no inline commit happened during op execution), so the next mutation proceeds normally with no `ExpectedVersionMismatch`. The publisher CAS at the very end either succeeds (manifest advances atomically across all touched sub-tables) or fails with a typed `ManifestConflictDetails::ExpectedVersionMismatch` (no partial publish). See [docs/dev/invariants.md](invariants.md) and [docs/dev/writes.md](writes.md). +Atomicity guarantee for multi-statement mutations: a mid-query failure leaves Lance HEAD untouched because no effect occurs during statement execution or staging. A pre-effect authority mismatch discards the complete attempt: insert-only mutations fully reprepare with a bounded retry, while Update/Delete returns typed `ReadSetChanged`. Once any `commit_staged` effect is durable, the attempt either makes every update visible together or returns `RecoveryRequired` with the fixed sidecar left for recovery; the engine never treats that post-effect error as an ordinary rebase. See [docs/dev/invariants.md](invariants.md) and [docs/dev/writes.md](writes.md). ## Bulk loader (`loader/mod.rs`) @@ -162,11 +183,11 @@ Atomicity guarantee for multi-statement mutations: a mid-query failure leaves La | Mode | Semantics | Path (post-MR-794) | |---|---|---| -| `Overwrite` | Replace all data in the target tables on the branch | Same accumulator; one `stage_overwrite` + `commit_staged` per touched table at end-of-load (a staged Lance `Operation::Overwrite` transaction — HEAD does not advance until commit; MR-793 Phase 2); publisher CAS. | -| `Append` | Strict insert; duplicates error | In-memory `MutationStaging` accumulator; one `stage_append` + `commit_staged` per touched table at end-of-load; publisher CAS. | -| `Merge` | Upsert by `id` (`merge_insert`) | Same accumulator; one `stage_merge_insert` per touched table at end-of-load (Merge mode dedupes by `id`, last-write-wins); publisher CAS. | +| `Overwrite` | Replace all data in the target tables on the branch | Same accumulator; one staged Lance `Operation::Overwrite` transaction per touched table. A pre-effect authority change is strict `ReadSetChanged`; no automatic replay. | +| `Append` | Strict insert; duplicates error | One `stage_append` transaction per touched table. A pre-effect authority change discards the whole parsed/validated attempt and fully reprepares with a bounded retry. | +| `Merge` | Upsert by `id` (`merge_insert`) | One `stage_merge_insert` transaction per touched table (deduped by `id`, last-write-wins). The same bounded full-reprepare rule as Append applies before effects. | -For all three modes, a mid-load failure (RI / cardinality violation, validation error) leaves Lance HEAD untouched on the staged tables — the next load on the same tables proceeds normally with no `ExpectedVersionMismatch`. +All three modes then use the same schema → branch → sorted-table gate, v3 recovery, zero-retry table commit, and exact publisher-precondition path as mutation. A parse, RI, cardinality, or validation failure leaves Lance HEAD untouched. After any table effect, any later error is `RecoveryRequired`. Load, mutation, and schema apply build no physical indexes inline; explicit `ensure_indices`/`optimize` reconciliation materializes declared intent later. ## `load` and the deprecated `ingest` shims diff --git a/docs/dev/handoff-rfc-013-write-path.md b/docs/dev/handoff-rfc-013-write-path.md index 9ae296b6..12c66603 100644 --- a/docs/dev/handoff-rfc-013-write-path.md +++ b/docs/dev/handoff-rfc-013-write-path.md @@ -61,11 +61,14 @@ amplification = PR2's target.) ### A.2 What is LANDED on `main` -- **Step 2a** — `optimize` compacts the internal tables too (`__manifest` / `_graph_commits` / - `_graph_commit_actors`), so a *periodically-compacted* graph keeps Term-1 flat. (Cleanup/version-GC - of them is the still-open PR1.) +- **Step 2a** — #291 originally added compaction for all three then-current + internal tables (`__manifest`, `_graph_commits`, and `_graph_commit_actors`). + After strand-and-retire removed the two lineage datasets, the live behavior is + `__manifest`-only internal compaction. A periodically-compacted graph therefore + keeps the remaining Term-1 flat. (`__manifest` cleanup/version-GC is still open.) - **Phase 7 / #299** (`1c5cb874`) — graph lineage lives in `__manifest` (`graph_commit` + - `graph_head:` rows in the same publish merge-insert; `_graph_commits` is now a projection; + `graph_head:` rows in the same publish merge-insert; `CommitGraph` is + now a projection and the former lineage datasets are gone; v3→v4 internal-schema migration; schema-version floor). This removed the per-write commit-graph scan and closed the manifest→commit-graph atomicity + commit-graph-parent-under-concurrency gaps. **This is the base everything below builds on.** @@ -202,12 +205,14 @@ for the canonical list. Current reality: - **Step 3a** — opener bypass: write opens go direct (`Dataset::open` by URI + version) instead of the Lance-namespace builder (#288). **This already banked the dominant depth win** — see §2 below; it reframes everything. -- **Step 2a** — internal-table compaction: `optimize` now compacts `__manifest` / - `_graph_commits` / `_graph_commit_actors` (#291). Plus the RFC latency-model - correction (#292). +- **Step 2a** — internal-table compaction: #291 originally covered + `__manifest` plus the two then-live lineage datasets. The current + post-retirement implementation compacts only `__manifest`. Plus the RFC + latency-model correction (#292). - **Step 4 / Phase 7** — graph lineage moved into `__manifest` (#299 `1c5cb874`): `graph_commit` + mutable `graph_head:` in the publish merge-insert, - `_graph_commits` now a projection. **The base for the live branch (§A).** + with `CommitGraph` projected directly from those rows and no separate lineage + dataset. **The base for the live branch (§A).** - **Optimize-vs-write race** — optimize survives a cross-process write race on the same table (#297, **LANDED** — origin/main `6d4606a8`; see §6 for why it's not redundant with Design A). Step 3b stacks on top of this. @@ -312,7 +317,7 @@ single staleness feeds **two distinct failure modes**, both surfaced this cycle: `writes.rs::served_strict_delete_after_external_optimize_advance_auto_refreshes` (`#[ignore]` on branch `fix/write-path-stale-view-probe`). **The naive "just probe" fix is proven wrong** — a blanket probe silently refreshes past *logical* advances too, breaking - `consistency::stale_handle_public_mutation_must_refresh_then_retry` (the deliberate + `consistency::stale_handle_strict_mutation_returns_read_set_changed_then_refresh_retry` (the deliberate cross-process lost-update OCC primitive). The fix must **discriminate by op class**. **Both fold into Design A (step 5), same as §1c.** `open_txn`'s one warm probe makes the base @@ -425,13 +430,14 @@ reopen/replan **semantics** are permanent. (Noted in RFC §6.6.) ## 4. DONE: Step 3b — capture-once `WriteTxn` (shipped on `rfc-013-step-3b-writetxn-v2`) **Delivered:** on the **table-touch hot path**, a single `mutate`/`load` validates the schema -contract **once** and opens each touched data table **at most once** — a constant-factor/RTT -win (not a depth-slope win; 1a). Two cost gates in `write_cost.rs` lock it (both on a node -insert): `write_validates_schema_contract_once` (3 `read_text` / 2 `exists`, was 12/9) and +contract once at capture and once at the RFC-022 pre-effect gate, with one cheap trailing +identity-marker read fencing capture (never per table), and opens each touched data table +**at most once** — a constant-factor/RTT win (not a depth-slope win; +1a). Two cost gates in `write_cost.rs` lock it (both on a node insert): +`write_schema_io_is_bounded_to_capture_fence_and_effect_gate` (7 `read_text` / 4 `exists`) and `keyed_insert_opens_table_at_most_once` (`data_open_count <= 1`, was 4). The carrier is the -minimal `WriteTxn { branch, base }`, threaded as `Option<&WriteTxn>` (`Some` on the hot -mutate/load path, `None` byte-identical everywhere else); it **converges into** step 5's -`PublishPlan`. +operation-local `WriteTxn { branch, base, authority, catalog }`; the catalog is built from the +exact accepted IR named by the authority token. It **converges into** step 5's `PublishPlan`. **Not "once" everywhere (scope, not regression):** edge endpoint / cardinality RI validation (`ensure_node_id_exists`, the loader's RI + cardinality) still resolves through @@ -444,8 +450,9 @@ gate yet (only the node gates above). Commits (off merged-#297 main): - **Stage 0** — scope `open_count` → `data_open_count`/`internal_open_count` by URI class - (the review fix: `open_dataset_tracked` also opens `__manifest`/`_graph_commits`, so the - raw counter conflated them and the gate was unreachable). Re-baselined RED 4. + (historically, before lineage-table retirement, `open_dataset_tracked` also opened + `__manifest`/`_graph_commits`, so the raw counter conflated them and the gate was + unreachable). Re-baselined RED 4. - **Commit A (schema-once)** — capture `txn` once at entry (the single validation); the 4 validation sites collapse: S1 (entry `ensure_schema_state_valid`) removed; S3a (`open_for_mutation_on_branch`) + S3b (`prepare_updates_for_commit`) source `txn.base`; @@ -587,7 +594,8 @@ for #298** (which built none of those constructs) but are **load-bearing constra residual concurrent TOCTOU is the §7.1 gap (step 4) — un-widen here, don't over-reach. - **Step 4 / Phase 7** (`iss-991`): **LANDED on `main` as #299 (`1c5cb874`).** Lineage now lives in `__manifest` (`graph_commit` + mutable `graph_head:` in the same merge-insert; - `_graph_commits` is a projection). Removed the per-write `commit_graph.refresh`; closed the + `CommitGraph` is projected from those rows and the old lineage dataset is + gone). Removed the per-write `commit_graph.refresh`; closed the manifest→commit-graph atomicity + commit-graph-parent-under-concurrency gaps. *(Historical note, kept for the §7.1 framing it carried:)* it carries the §7.1 *concurrent* write-skew fix (needs the `graph_head` contention row) — @@ -600,7 +608,7 @@ for #298** (which built none of those constructs) but are **load-bearing constra false-fail** (§1d.2) — the op-class-aware precondition + `open_txn` probe. The contract is two tests passing *together*: un-ignore `writes.rs::served_strict_delete_after_external_optimize_advance_auto_refreshes` (goes green) - *while* `consistency::stale_handle_public_mutation_must_refresh_then_retry` stays green + *while* `consistency::stale_handle_strict_mutation_returns_read_set_changed_then_refresh_retry` stays green (maintenance fast-forwards; logical fails loudly). Self-contained enough to ship standalone like #297 if prod pain is acute; otherwise fold into the single PublishPlan delta-interpreter. - **Step 2b** — internal-table cleanup + the Q8 monotonic watermark (a Lance boundary tag). diff --git a/docs/dev/index.md b/docs/dev/index.md index dc51e836..5311456f 100644 --- a/docs/dev/index.md +++ b/docs/dev/index.md @@ -74,6 +74,15 @@ pattern, not just the outcome. |---|---| | camelCase property filters lowercased at runtime (#283) — two engine→Lance boundaries, two different fixes | [bug-case-fix.md](bug-case-fix.md) | +## Active Design Reviews + +Review ledgers record open findings against proposed architecture. They remain +as durable disposition history after closure, so RFC backlinks stay valid. + +| Area | Read | +|---|---| +| RFC-022–027 split architecture review — open correctness blockers, dependency corrections, and required acceptance evidence | [rfc-022-027-architecture-review.md](rfc-022-027-architecture-review.md) | + ## Active Implementation Plans Working documents for in-flight feature work. Removed when the work lands. diff --git a/docs/dev/invariants.md b/docs/dev/invariants.md index b151ea56..1da9fd6e 100644 --- a/docs/dev/invariants.md +++ b/docs/dev/invariants.md @@ -146,7 +146,11 @@ converge the physical state. drifts" and "manifest-derivable reconciler" items, are instances; so is bounding a read's cost to its working set rather than the commit count. This is the structural face of "engineering is programming integrated over time": - both failure modes are liabilities that compound as the system grows. + both failure modes are liabilities that compound as the system grows. At a + write/control boundary, schema identity, compiled catalog, and manifest + authority must come from one operation-local accepted view: validating a + fresh on-disk marker while planning or enumerating gates from a stale warm + catalog is not a coherent derivation. ## Current Truth Matrix @@ -155,11 +159,11 @@ converge the physical state. | Multi-table commit | Manifest CAS plus recovery sidecars; not a single Lance primitive | [writes.md](writes.md), [architecture.md](architecture.md) | | Constructive mutations | In-memory `MutationStaging`, one end-of-query table commit per touched table, then one manifest publish | [writes.md](writes.md), [execution.md](execution.md) | | Deletes | Staged like inserts/updates (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, MR-A) — no inline HEAD advance; mixed insert/update/delete in one query rejected by D2 as a deliberate boundary (constructive XOR destructive per query; compose via separate mutations or a branch) | [query-language.md](../user/queries/index.md), [writes.md](writes.md) | -| Branch delete | Manifest is the single authority, flipped atomically first; per-table forks are derived state, reclaimed best-effort (`force_delete_branch`) with the `cleanup` reconciler as the guaranteed backstop. Reusing a name whose reclaim failed before `cleanup` surfaces an actionable error | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md) | +| Branch delete | Manifest is the single authority, flipped atomically first; per-table forks are derived state, reclaimed best-effort (`force_delete_branch`) with the `cleanup` reconciler as the guaranteed backstop. Under schema/target/all-table gates, a target-scoped unresolved sidecar may be made unreachable by deletion and is then audit-discarded by recovery; graph-global SchemaApply still blocks. Reusing a name whose fork reclaim failed before `cleanup` surfaces an actionable error | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) | | Schema validation | Type checks, required fields, defaults, edge endpoint checks, and edge cardinality are enforced on write paths | [schema-language.md](../user/schema/index.md), [execution.md](execution.md) | -| Unique constraints | Value/enum, uniqueness, edge-RI, and cardinality route through ONE unified, catalog-derived evaluator (`crate::validate`) on ALL THREE write surfaces — branch-merge, mutation, and bulk load: Δ-scoped (checks the delta, not the whole graph) and index-backed (probes committed state through its BTREE rather than full-scanning every catalog table), reusing the leaf checks (`loader::validate_value_constraints`/`validate_enum_constraints`/`composite_unique_key`) so the surfaces cannot drift. This closed the prior merge bug (merge validated `@range`/`@check` but not enum) AND the **cross-version uniqueness gap** on the mutation and load paths (a duplicate of a committed `@unique` value is now rejected; the merge path always enforced it). The committed view is the merge target snapshot (merge), the write's pinned `txn.base` (mutation), or the pinned pre-load base (load — `Overwrite` validates the batch as the whole new image, committed view empty); `@card` reads LIVE HEAD per edge table on the mutation path only (the #298 stale-handle fix). `@key` is id-backed, so it is checked intra-delta only (a committed holder of a key value is always the same row — an upsert), skipping a wasted O(Δ) probe per keyed row; `@unique` (non-key) groups do the committed lookup. | [schema-language.md](../user/schema/index.md) | +| Unique constraints | Value/enum, uniqueness, edge-RI, and cardinality route through ONE unified, catalog-derived evaluator (`crate::validate`) on ALL THREE write surfaces — branch-merge, mutation, and bulk load: Δ-scoped (checks the delta, not the whole graph) and structured-filter-backed (committed probes use a BTREE when reconciled and remain correct by scanning while it is pending), reusing the leaf checks (`loader::validate_value_constraints`/`validate_enum_constraints`/`composite_unique_key`) so the surfaces cannot drift. This closed the prior merge bug (merge validated `@range`/`@check` but not enum) AND the **cross-version uniqueness gap** on the mutation and load paths (a duplicate of a committed `@unique` value is now rejected; the merge path always enforced it). The committed view is the merge target snapshot (merge), the write's pinned `txn.base` (mutation), or the pinned pre-load base (load — `Overwrite` validates the batch as the whole new image, committed view empty); `@card` refreshes the manifest-visible graph-branch snapshot on the mutation path only (the #298 stale-handle fix), then follows each entry's actual inherited/owned Lance ref. `@key` is id-backed, so it is checked intra-delta only (a committed holder of a key value is always the same row — an upsert), skipping a wasted O(Δ) probe per keyed row; `@unique` (non-key) groups do the committed lookup. | [schema-language.md](../user/schema/index.md) | | Storage trait | `TableStorage` (via `db.storage()`) is staged-only; the sole inline-commit residual (`create_vector_index`) is split onto a separate sealed `InlineCommitResidual` trait reached via `db.storage_inline_residual()` (MR-854), so §1 holds by construction; capability/stat surfaces are roadmap | [writes.md](writes.md), [architecture.md](architecture.md) | -| Index lifecycle | `@index`/`@key` declares *intent*; the physical index is derived state and never fails a logical op. `schema apply` builds no indexes (records intent only; index-only changes touch no table data). `load`/`mutate` build inline through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector) that fault-isolates an untrainable Vector column into a *pending* index instead of aborting. `optimize`/`ensure_indices` is the reconciler: it creates declared-but-missing indexes and folds appended/rewritten fragments into existing ones (`optimize_indices`), reporting still-pending columns. Explicit maintenance call, not yet a background loop | [indexes.md](../user/search/indexes.md), [maintenance.md](../user/operations/maintenance.md) | +| Index lifecycle | `@index`/`@key` declares *intent*; the physical index is derived state and never fails a logical op. `schema apply`, `load`, and `mutate` build no indexes inline: RFC-022 mutation/load sidecars describe only their exact data effects, and index availability must never become a correctness prerequisite. `optimize`/`ensure_indices` is the reconciler: through the single `build_indices_on_dataset_for_catalog` chokepoint it creates declared-but-missing indexes (enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector), folds appended/rewritten fragments into existing ones (`optimize_indices`), and reports untrainable Vector columns as pending. Explicit maintenance call, not yet a background loop | [indexes.md](../user/search/indexes.md), [maintenance.md](../user/operations/maintenance.md) | | Traversal IDs | Runtime still builds `TypeIndex`; Lance stable row-id based graph IDs are roadmap | [architecture.md](architecture.md), [query-language.md](../user/queries/index.md) | | Auth | Bearer token hashing and server-side actor resolution are implemented at the HTTP boundary | [server.md](../user/operations/server.md), [policy.md](../user/operations/policy.md) | | Tests | Tempdir-backed Lance tests are the current substrate; the storage adapter has an in-memory backend for adapter-level contract tests, but Lance datasets bypass it | [testing.md](testing.md) | @@ -215,12 +219,15 @@ them explicit. `maintenance.rs::optimize_compacts_blob_table_alongside_plain_table` pin the positive behavior (a red there means blob compaction regressed — restore the skip machinery from git history). -- **Recovery is serialized against live writers in-process only:** the - write-entry heal (and `refresh`) serialize against a live writer's sidecar - lifetime via the per-`(table, branch)` write queues plus the schema-apply - serialization key — all in-process primitives. A recovery pass in one - process cannot serialize against a live writer in another (the open-time - sweep has the same exposure, and always has): it may roll a live foreign +- **Recovery is serialized against live writers in-process only:** every + `Omnigraph` handle for one canonical local root identity (lexically absolute, + with existing symlink ancestors resolved) shares a root-scoped queue manager; + object-store/custom scheme identities remain their normalized opaque URIs. + The write-entry heal, `refresh`, and Full ReadWrite-open sweep serialize + against a live writer's complete sidecar lifetime in schema → branch → sorted + table order and re-check sidecar existence after waiting. These remain + in-process primitives: a recovery pass cannot serialize against a live writer + in another process. It may roll a live foreign writer's sidecar forward, which degrades to publisher-CAS contention for data writes but can race the schema-staging promotion for a foreign live schema apply. The roll-**forward** CAS contention is now @@ -238,22 +245,26 @@ them explicit. cross-process serialization primitive (e.g. lease-based use of the schema-apply lock branch) — design it before promoting multi-process write topologies. -- **Fork reclaim is in-process-safe only:** the first write to a table on a - branch forks it (a Lance `create_branch` that advances state before the - manifest publish). An interrupted fork (crash, or a cancelled request - future) leaves a manifest-unreferenced branch ref. The next write self-heals - it — `reclaim_orphaned_fork_and_refork` (`force_delete_branch` + re-fork) - — but reclaim is only safe because the writer holds the per-`(table, - branch)` write queue from before the fork through the publish AND re-checks - the live manifest under it, so no *in-process* writer can be mid-fork. A - reclaim cannot serialize against a foreign-*process* in-flight fork: it may - force-delete a peer's just-created ref, which makes that peer's commit fail - and retry — the same one-winner-CAS exposure as above, not corruption. The - reclaim never fires unless in-process-queue + manifest authority both prove - the ref is manifest-unreferenced. `cleanup`'s per-table reconciler - (`reconcile_orphaned_branches`) is the guaranteed backstop for any fork the - write path never revisits. Both degrade to a no-op if Lance ships an atomic - multi-dataset branch op. +- **Fork ownership is durable, but Lance ref deletion is not conditional:** a + first-touch Mutation/Load table no longer creates its target ref while + preparing. Under schema → branch → table gates it revalidates, durably arms a + schema-v3 sidecar naming the target, creates the ref, then stages branch-local + files and commits. Both `reclaim_orphaned_fork_and_refork` and + `reconcile_orphaned_branches` consult pending sidecars before destruction; a + foreign claim is conflict/indeterminate, never permission to delete. Full + recovery accepts sidecar-before-ref crashes and deletes an unpublished fork + only when it is still exactly at the inherited version and no other sidecar + claims `(table_path, target ref)`. A no-effect sidecar with a competitor + discards only itself; the last survivor either cleans the untouched ref or + recovers its owned effect. Partial rollback performs no-effect cleanup before + publishing its fixed outcome. This closes the cross-handle live-ref + deletion bug and keeps cleanup as the backstop for truly unclaimed refs. + The remaining multi-process gap is narrower but real: Lance exposes + `force_delete_branch`, not compare-and-delete by `BranchIdentifier`, so a + foreign process can create intent/ref between the final list/check and the + delete. The documented single-writer-process support boundary remains until + Lance provides a conditional ref primitive (or OmniGraph adds a distributed + fence); process-local queues are not credited as that primitive. - **Local `write_text_if_match` is not a cross-process CAS:** object-store backends use a true conditional put (ETag If-Match; the in-memory test backend too), but upstream `object_store` leaves `PutMode::Update` diff --git a/docs/dev/rfc-004-cluster-graph-schema-apply.md b/docs/dev/rfc-004-cluster-graph-schema-apply.md index e9c0336d..c6a6d925 100644 --- a/docs/dev/rfc-004-cluster-graph-schema-apply.md +++ b/docs/dev/rfc-004-cluster-graph-schema-apply.md @@ -36,7 +36,7 @@ The implementation spec's hard gate for this phase — failpoint recovery tests What Phase 4 builds on (all shipped): -- **The engine's recovery discipline.** Writers that can advance Lance HEAD before manifest publish write `__recovery/{ulid}.json` sidecars carrying per-table pins (`expected_version`, `post_commit_pin`); `Omnigraph::open` in read-write mode classifies every pinned table (`NoMovement` / `RolledPastExpected` / `UnexpectedAtP1` / `UnexpectedMultistep` / `InvariantViolation`) and decides all-or-nothing: roll forward via one manifest publish, or roll back via `Dataset::restore`, recording an audit row attributed to `omnigraph:recovery`. The cluster inherits the *vocabulary* of this design but not its mechanics — see the roll-forward-only argument below. +- **The engine's recovery discipline.** Writers that can advance Lance HEAD before manifest publish write `__recovery/{ulid}.json` sidecars carrying per-table pins (`expected_version`, `post_commit_pin`); `Omnigraph::open` in read-write mode classifies every pinned table (`NoMovement` / `RolledPastExpected` / `UnexpectedAtP1` / `UnexpectedMultistep` / `InvariantViolation`) and decides all-or-nothing: roll forward via one manifest publish, or roll back via `Dataset::restore`. Every completed action records an audit row with the interrupted writer's actor in `recovery_for_actor`. A schema-v3 roll-forward publishes the sidecar's pre-minted lineage and therefore preserves that writer's commit id and actor; rollback and legacy recovery commits use `omnigraph:recovery`. The cluster inherits the *vocabulary* of this design but not its mechanics — see the roll-forward-only argument below. - **The engine's schema-apply surface.** `apply_schema_as(desired_source, SchemaApplyOptions { allow_data_loss }, actor)` returns `SchemaApplyResult { supported, applied, manifest_version, steps }`; `preview_schema_apply_with_options` returns the migration plan plus desired catalog without applying; the `__schema_apply_lock__` branch serializes schema applies graph-wide and refuses to run while user branches exist. Policy enforcement (`enforce(SchemaApply, TargetBranch("main"), actor)`) happens before the lock. - **Graph init.** `Omnigraph::init(uri, schema_source)` with a strict preflight (errors if schema artifacts exist) and an atomic `_schema.pg` claim. A documented gap: a failed init does not clean up Lance datasets or `__manifest/` it already created. - **No engine graph-delete primitive.** Deleting a graph today means removing its object-store prefix. This RFC works with that fact rather than waiting on a primitive. diff --git a/docs/dev/rfc-022-027-architecture-review.md b/docs/dev/rfc-022-027-architecture-review.md new file mode 100644 index 00000000..7dbd5941 --- /dev/null +++ b/docs/dev/rfc-022-027-architecture-review.md @@ -0,0 +1,549 @@ +# Architecture review: RFC-022 through RFC-027 + +**Status:** Open review findings +**Date:** 2026-07-11 +**Audience:** RFC authors, engine/storage maintainers, and release reviewers +**Reviewed against:** OmniGraph 0.8.1; Lance 9.0.0-beta.15 at +`f24e42c11a742581365e1cbe17c906ea2dac1bc6`; full Lance transaction, +branch/tag, MemWAL, row-lineage, read/write, compaction, and cleanup +specifications; pinned Rust implementation where the public specification is +not precise enough + +This is a review ledger, not a competing specification. The RFCs remain the +normative proposals. A finding is closed only when the affected RFC either: + +1. incorporates the required contract and tests; or +2. explicitly rejects the finding with evidence and records the resulting + support boundary. + +Items marked **BLOCKER** must be dispositioned before the RFC that owns them is +accepted. Items marked **TIGHTENING** may be resolved during revision, but they +must not silently disappear from review. + +This ledger is durable review history, not a throwaway implementation plan. +After every finding is dispositioned, change its status to closed and record +the resolving RFC sections/commits; keep the RFC backlinks valid. + +## Overall architectural judgment + +The central architecture remains the right one: + +- Lance is the physical storage/versioning truth. +- `__manifest` is the graph-visible authority and one graph publish is the + visibility point for a logical graph commit. +- RFC-022 should define one correctness protocol with explicit physical-effect + adapters, not pretend that Lance offers one universal staged primitive. +- Key fencing, durable heads, retention, MemWAL, and lineage acceleration are + separate irreversible decisions and should keep separate evidence and format + gates. + +The current blockers are boundary problems, not a reason to replace that core. +They concern crash classification, actual fencing, foreign-branch authority, +and capability activation. + +> 💬 **Second-pass verification (2026-07-11):** I independently re-verified this +> ledger's two sharpest new substrate claims against the pinned checkout before +> commenting: BLOCKER-01's two-phase branch create is confirmed in Lance's own +> doc comment, and BLOCKER-04's lazy-branch exposure is confirmed +> architecturally — and appears to be a live bug in the shipped `cleanup`, not +> only an RFC gap (see the comment there). Overall judgment: concur with this +> ledger; three of its findings supersede corrections I applied to the RFCs on +> 2026-07-11 (noted inline at BLOCKER-07, BLOCKER-11, and tightening 5). + +The latest revisions improved several important points and those changes should +remain: + +- exact recovery-goal convergence is distinguished from adopting an unrelated + newer version; +- MemWAL is treated as Lance's strategic streaming architecture, with API and + format evolution as the integration risk; +- keyed fast-forward merge now has an explicit embedding-table memory gate; +- the legacy `__manifest` PK representation is preserved rather than + “normalized” into an incompatible form; +- cleanup's required operating cadence and the cost of indefinite deferral are + explicit; +- native branch refs are no longer described as cross-process-safe merely + because a process-local gate exists. + +## Dependency correction + +Stable table identity and incarnation are consumed by more than durable heads. +The intended dependency shape is therefore: + +```text +RFC-022 unified write protocol + | + +-- stable identity/incarnation capability + | +-- RFC-024 durable heads + | +-- RFC-025 checkpoint rows + | `-- RFC-026 stream/reject authority + | + +-- RFC-023 key fencing + | `-- RFC-026 keyed streaming mode + | + +-- RFC-025 retention + +-- RFC-026 MemWAL + `-- RFC-027 merge-delta research +``` + +The public RFC process accepts whole RFCs, not independently accepted sections. +The identity capability must therefore be extracted into its own RFC, or +RFC-024 must be accepted in full before identity-dependent siblings. An +implementation may phase an accepted RFC internally, but it cannot treat a +sub-contract of an unaccepted RFC as a separately accepted decision. + +## Blocking findings + +### BLOCKER-01 — native graph-branch control is multi-phase + +**Affected:** [RFC-022 §7](../rfcs/rfc-022-unified-write-path.md#7-native-graph-branch-ref-control-protocol) + +RFC-022 currently models branch create/delete as one native ref mutation whose +completion is the visibility point. The pinned Lance implementation is more +specific: + +- [`Dataset::create_branch`](https://github.com/lance-format/lance/blob/f24e42c11a742581365e1cbe17c906ea2dac1bc6/rust/lance/src/dataset.rs#L488-L535) + first commits a shallow-cloned branch dataset and only then writes + `BranchContents`. Lance explicitly calls this a non-atomic two-phase + operation and calls `BranchContents` the source of truth. +- [branch deletion](https://github.com/lance-format/lance/blob/f24e42c11a742581365e1cbe17c906ea2dac1bc6/rust/lance/src/dataset/refs.rs#L548-L584) + removes `BranchContents` before cleaning the branch directory. + +Therefore create can crash with a zombie branch dataset and no authoritative +ref. A retry with the same name can fail until the zombie is reclaimed. Delete +can make the branch logically absent and then return a cleanup error. + +**Required disposition:** define an idempotent control-operation classifier for +at least these states: + +| Operation | Physical state | Logical result | +|---|---|---| +| create | no clone, no `BranchContents` | not started | +| create | clone only | zombie; reclaim before retry, or wait for an upstream completion API | +| create | clone plus matching `BranchContents` | complete | +| delete | `BranchContents` present | not deleted | +| delete | `BranchContents` absent, directory remains | deleted; reclaim pending | +| delete | both absent | complete | + +The pinned public API cannot adopt the clone-only state: `BranchIdentifier` is +generated later by Lance's private `BranchContents` creation phase. The RFC +must therefore say whether a durable operation marker/sidecar is written before +the shallow clone, how retry distinguishes its own zombie from foreign state, +and whether it reclaims and retries or waits for a new upstream completion API. +It must also explain why cleanup failure after authoritative deletion is not +reported as if the branch still existed. Add failpoints at both create phases +and between delete authority removal and directory cleanup. + +> 💬 **Verified (2026-07-11):** confirmed verbatim in the pinned source — +> `dataset.rs:488` documents "This is a two-phase operation… These two phases +> are not atomic. We consider `BranchContents` as the source of truth… may +> leave a zombie branch dataset", including the zombie-blocks-retry +> consequence and the cleanup duties this finding transcribes. The 2026-07-11 +> RFC-022 §7 revision (single-writer-process boundary + conditional-ref +> upstream ask) addressed the *conditional-put* gap but not this crash +> classification — the two dispositions compose; both are needed. + +### BLOCKER-02 — create-if-absent ownership is not a fencing lease + +**Affected:** RFC-023 migration claim, RFC-024 migration claim, and +[RFC-025 §2.3](../rfcs/rfc-025-checkpoint-retention.md#23-cross-process-retention-claim) + +`PutMode::Create` correctly elects one initial owner. A random token in that +object does not by itself fence a paused owner, because Lance tag, branch, +migration, and cleanup effects do not condition their commits on that token. +RFC-025 proposes a check-then-delete release; RFC-023 does not yet specify a +release protocol at all. For the check-then-delete shape, the race is: + +1. old owner reads and verifies token A; +2. old owner pauses; +3. takeover removes A and creates token B; +4. old owner resumes and deletes B, or resumes a destructive physical effect. + +“The stale process checks again” does not close the window after its final +check. Calling the random owner token a fencing token overstates the substrate +guarantee. + +**Required disposition:** choose one honest contract: + +- no takeover while the prior process can resume; operator recovery proves the + process/host is dead before removing the claim; +- a substrate-enforced lease/fencing primitive whose monotonically newer token + is checked by every protected effect, plus conditional compare-and-delete; +- or a non-takeoverable claim that requires explicit offline repair. + +The `StorageAdapter` does not currently expose conditional delete, and its local +`write_text_if_match` is not a cross-process CAS. Any selected design must work +on local and S3 or explicitly narrow the supported backend/topology. For a +substrate-enforced lease, tests pause an owner after its last token check, +perform takeover, then resume it; the old owner must be unable to mutate state +or release the new claim. A host-death or non-takeoverable design instead proves +that takeover is refused until external fencing/death proof is established. + +> 💬 **Concur (2026-07-11):** the paused-owner race is real and the "fencing +> token" label overstated the guarantee. Given the substrate as it stands (no +> conditional delete on the adapter; the documented local `write_text_if_match` +> gap), the only honest near-term contract is the boring one — no takeover +> while the prior process can resume; operator proves process/host death or +> uses explicit offline repair. The substrate-enforced lease is the right end +> state but should be written as an upstream/adapter work item, not assumed. + +### BLOCKER-03 — key-conflict retry must first resolve partial effects + +**Affected:** [RFC-022 §9](../rfcs/rfc-022-unified-write-path.md#9-concurrency-and-retry-semantics), +[RFC-023 §6](../rfcs/rfc-023-key-conflict-fencing.md#6-retry-and-validation-semantics), +and [RFC-026 §6](../rfcs/rfc-026-memwal-streaming-ingest.md#6-fold-protocol) + +A retryable concurrent inserted-row-filter conflict is detected when a table +transaction attempts to commit. (`WhenMatched::Fail` may instead report a +pre-existing match during merge execution.) In a multi-table graph write or +MemWAL fold, table A may already have advanced under the armed sidecar before +table B reports the concurrent key conflict. Immediately “restart the entire +logical attempt” would replan around unresolved physical state, which RFC-022 +forbids. + +**Required disposition:** distinguish: + +- a conflict before any physical table commit — finalize the already-durable + empty sidecar, then perform a safe full semantic restart; +- a conflict after any physical commit — keep the sidecar and resolve the + RFC-022 recovery outcome first. Normally the partial attempt must roll back + before a new semantic attempt can be prepared. If rollback is not safe, return + a typed recovery-required failure rather than retrying around it. + +For strict insert, finalize an empty intent or retain a partial-effect sidecar, +as applicable, and return the terminal `KeyConflict`; RFC-022 permits partial +recovery to finish at the next synchronous barrier. Do not start another +attempt around that sidecar. For non-strict upsert, the barrier must resolve the +sidecar before the bounded automatic whole-operation retry. Add a failpoint/race +in which table N conflicts after table 1 has committed. + +### BLOCKER-04 — live graph branches need physical GC protection + +**Affected:** [RFC-025 §6](../rfcs/rfc-025-checkpoint-retention.md#6-cleanup-protocol-and-pruned-through-boundary) + +The cleanup root set includes live graph branches, but RFC-025 creates Lance +tags only for named checkpoints. This is insufficient for lazy graph branches. +A graph branch may store, in its `__manifest`, a foreign reference to an old +version on a data table's `main` branch. Lance cleanup on that data table cannot +see the foreign OmniGraph row; it protects versions through its own branch and +tag references. + +**Required disposition:** for every live graph-branch table reference, either: + +- materialize a deterministic Lance tag/native ref that physically protects the + exact `(table branch, version)`; or +- choose a per-dataset cleanup cutoff no newer than the oldest live reference, + accepting that all intervening versions remain retained. + +Checkpoint tags alone do not solve this. Add a test where a lazy branch pins an +old main-table version, main advances beyond the retention window, cleanup +runs, and the lazy branch still opens and reads the exact pinned state. + +> 💬 **Escalation (2026-07-11):** this is very likely a live bug in the shipped +> `cleanup`, independent of RFC-025. Verified in code: `cleanup_all_tables` +> (`optimize.rs:802`) runs Lance `cleanup_old_versions` per data table with no +> lazy-branch pin logic, and a lazy graph branch creates **no Lance-native ref +> on the data table** until its first write to that table — its pin exists only +> as a foreign `__manifest` row Lance cannot see. Repro shape: create a branch, +> never write table X on it, advance main past the keep window, `cleanup +> --keep N` → the branch's pinned version of X is collected and the branch +> breaks. Per the repo's test-first rule this deserves a red regression test +> and an issue *now*; RFC-025's disposition should then build on that fix +> rather than owning the discovery. (Snapshot/time-travel pins share the +> mechanism but are history-trimming under an operator-confirmed policy; a +> live branch's working state is not history.) + +### BLOCKER-05 — durable-head OCC must compare the full head token + +**Affected:** [RFC-024 §5](../rfcs/rfc-024-durable-table-heads.md#5-atomic-publisher-contract) + +The RFC says expected table versions are compared against table heads. Lance +version numbers can repeat across recreated datasets/branches, so version-only +comparison admits drop/recreate ABA. + +**Required disposition:** define the OCC token as the complete logical head +identity, at least: + +```text +(state, stable_table_id, incarnation_id, table_path, + table_branch, physical_ref_incarnation, table_version, schema_hash) +``` + +The publisher may encode that token differently, but retry/revalidation must +not reduce it to `table_version`. `physical_ref_incarnation` means an e_tag when +the backend supplies one, or another proven token that changes when a dataset +or native ref is deleted and recreated at the same path/branch/version. This is +distinct from the logical `incarnation_id`, which RFC-024 deliberately preserves +across an owner handoff. Add stale-writer races for both logical drop/recreate +and physical ref recreation at the same numeric version. + +### BLOCKER-06 — MemWAL needs a capability/format activation barrier + +**Affected:** [RFC-026 §3](../rfcs/rfc-026-memwal-streaming-ingest.md#3-enrollment-is-a-recoverable-inline-commit), +[§5](../rfcs/rfc-026-memwal-streaming-ingest.md#5-ack-path-validation-and-writer-lifecycle), +[§6](../rfcs/rfc-026-memwal-streaming-ingest.md#6-fold-protocol), +[§8](../rfcs/rfc-026-memwal-streaming-ingest.md#8-epoch-fenced-quiescence-barrier), +and [§12](../rfcs/rfc-026-memwal-streaming-ingest.md#12-phasing) + +Enrollment creates persistent MemWAL metadata and `stream_state` changes the +correctness preconditions for schema, branch, maintenance, and data operations. +An older binary that understands key fencing but not stream lifecycle can ignore +`OPEN | DRAINING | SEALED`, mutate the base table, or perform a schema/branch +operation without draining acknowledged rows. + +**Required disposition:** define: + +- a graph capability/internal-format stamp written only after every enrolled + table and lifecycle authority is valid; +- old-binary/new-format and new-binary/partial-format refusal behavior; +- migration and rollback/roll-forward ordering; +- preservation rules for later heads/retention formats; +- genuine cross-version tests, not a stamp-rewind simulation. + +Replica scope must also match RFC-023's recovery support boundary. Multiple +replicas may route acknowledgement traffic to one shard owner, but enrollment, +fold, and sidecar recovery remain single-writer-process operations until +foreign-process sidecar ownership is fenced. Do not advertise general replica +failover for those operations merely because MemWAL has a shard epoch. + +### BLOCKER-07 — stable identity ownership contradicts sibling dependencies + +**Affected:** [RFC-024 §3.1](../rfcs/rfc-024-durable-table-heads.md#31-identity-and-object-key), +[RFC-025 §2.1](../rfcs/rfc-025-checkpoint-retention.md#21-logical-authority-manifest-rows), +[RFC-026 §7](../rfcs/rfc-026-memwal-streaming-ingest.md#7-fold-time-rejection-is-atomic), +and [RFC-026 §8](../rfcs/rfc-026-memwal-streaming-ingest.md#8-epoch-fenced-quiescence-barrier) + +RFC-024 now says it exclusively owns stable table identity and incarnation and +that identity-dependent sibling claims must wait for it. RFC-025 still calls +RFC-024 optional, and RFC-026 persists `stable-table-id` without declaring the +dependency. + +**Required disposition:** extract the identity/incarnation format and migration +as a shared prerequisite RFC, or accept RFC-024 in full before RFC-025 and +RFC-026. Durable-head implementation can remain a later phase only after its +owning RFC and format contract are accepted. + +The `_ingest_rejects` deterministic key must use stable table ID plus +incarnation rather than mutable `table_key`, or rename/recreate can change reject +identity and break replay idempotence. + +> 💬 **Concur; supersedes a 2026-07-11 correction (2026-07-11):** the earlier +> pass placed identity ownership *inside* RFC-024 §3.1, which created exactly +> the contradiction described here (RFC-025 calls heads optional while +> consuming the identity RFC-024 claims to own). The extracted +> identity/incarnation RFC is the cleaner shape and should replace that +> ownership note. The `_ingest_rejects` key observation is an additional catch +> the earlier pass missed — endorsed. + +### BLOCKER-08 — retention activation cannot precede migration selection + +**Affected:** [RFC-025 §8](../rfcs/rfc-025-checkpoint-retention.md#8-migration-and-compatibility) +and [§11](../rfcs/rfc-025-checkpoint-retention.md#11-phasing) + +RFC-025 leaves in-place migration versus export/import undecided. Its phase +table nevertheless puts “format activation” in Phase A and “selected migration +path” in Phase D. Activation cannot be implemented or reviewed before the +upgrade contract is known. + +**Required disposition:** select the migration before Phase A, then specify +quiescence, partial-state refusal, main-stamp ordering, branch coverage, crash +recovery, and later capability preservation. If export/import is selected, +state the irreversible loss of branches, commit DAG, snapshots, and historical +checkpoints as part of the acceptance decision. + +### BLOCKER-09 — decide whether these are public or internal RFCs + +**Affected:** [public RFC process](../rfcs/README.md), RFC-022 through RFC-027 + +The files currently live in the public RFC track but use internal-style +`rfc-022-*` names, noncanonical `draft`/`research-blocked` statuses, and empty +owner metadata. In the public process, merging an RFC is acceptance; a merged +public RFC cannot simultaneously retain undispositioned acceptance blockers. + +**Required disposition:** choose the track before merge: + +- for the public track, rename to `NNNN-title.md`, use `Status: Proposed`, fill + the template's author/discussion/implementation metadata, and resolve every + blocker before the RFC PR merges; or +- move the proposals to `docs/dev/` and follow the maintainer-internal process, + leaving the public RFC directory for externally authorable accepted records. + +RFC-027 may state “research-blocked” prominently as its technical state while +retaining the public lifecycle status `Proposed`. + +> 💬 **Concur (2026-07-11):** same recommendation as the original family +> review — these are maintainer-internal proposals mid-revision; `docs/dev/` +> with the internal process is the low-friction disposition, keeping +> `docs/rfcs/` for its defined merge-equals-acceptance lifecycle. Whichever +> track is chosen, decide it before any of the set merges, since it defines +> what "dispositioned before acceptance" means for every other finding here. + +## Protocol clarifications + +### BLOCKER-10 — do not put foreign-branch facts in an atomic `ReadSet` + +RFC-022 requires every `ReadSet` member to be arbitrated atomically by the +publish CAS. A CAS on reserved main cannot arbitrate a row on a named source +branch; a merge-target CAS cannot arbitrate a source-branch row. + +Use the right category for each fact: + +- checkpoint creation captures an immutable source version, revalidates it + before tagging, and then relies on the physical tag. A later source-head + advance is intentionally harmless, so the source head is an effect + precondition, not target-CAS authority; +- offline cleanup facts are protected by the selected fleet/retention barrier, + not by pretending one main-branch CAS covers every named branch; +- branch merge should define captured-source-commit semantics. If “latest + source at target publish” is required instead, it needs a real source-branch + fence held through the target CAS. + +Keep target-branch values that must remain stable in the atomic `ReadSet`. + +### BLOCKER-11 — RFC-022 and no-heads MemWAL need coarse OCC + +**Affected:** [RFC-022 §10](../rfcs/rfc-022-unified-write-path.md#10-rollout-and-compatibility) +and [RFC-026 §6](../rfcs/rfc-026-memwal-streaming-ingest.md#6-fold-protocol) + +[RFC-022's rollout](../rfcs/rfc-022-unified-write-path.md#10-rollout-and-compatibility) +currently says mutation/load read-set arbitration is likely blocked on +RFC-024 because probed-but-untouched tables lack mutable head rows. + +For graph-content writes, `(branch incarnation, optional graph head)` is already +a conservative branch-wide authority token. An established branch changes its +`graph_head:` row on every graph commit; a fresh branch may initially +have no branch-specific head row, so absence plus incarnation is part of the +captured token. Schema identity needs its own atomically contended authority. +Any concurrent logical graph change forces full revalidation. RFC-024 table +heads can later reduce false contention by narrowing that token to the tables +actually read. + +Today's publisher retries head-row contention by re-reading the live head and +reparenting the prepared write. That is insufficient for a +validation-sensitive plan. On every publisher retry, it must compare the +captured token and return control to full revalidation rather than reparenting +and continuing with stale validation. + +The lower-liability rollout is therefore: + +1. ship correct coarse OCC with `graph_head` plus schema identity; +2. introduce table-head tokens only after RFC-024 passes its independent format + and cost gates. + +The no-heads RFC-026 fold path must carry the same captured branch token in its +`ReadSet`. Do not recouple RFC-022 or RFC-026 correctness to RFC-024 +performance. + +> 💬 **Concur; supersedes a 2026-07-11 correction (2026-07-11):** the premise +> is independently verified — `graph_head:` is one mutable row updated +> in every graph-content publish (the deliberate contention point pinned by +> the concurrent-disjoint-writers tests), so the coarse token genuinely +> arbitrates probed-but-untouched same-branch tables. This is a better design +> than the earlier RFC-022 §10 wording ("likely blocked on RFC-024"), which +> should be revised to the two-step rollout described here. One cost worth +> stating in the revision: coarse OCC makes *any* concurrent commit on a busy +> branch force full revalidation of in-flight writers — real throughput cost +> on agent-fleet branches, and precisely the honest motivation for RFC-024's +> later narrowing (rather than a correctness argument for it). + +> **Implementation disposition (2026-07-11):** mutation/load now capture the +> native Lance `BranchIdentifier`, exact optional `graph_head`, and accepted +> schema identity; revalidate under a branch-then-table gate; and pass the same +> token to every publisher retry and schema-v3 recovery decision. Metadata-only +> schema-apply tests pin the required invariant that supported schema changes +> move `graph_head` even when no data-table version changes. This closes the +> coarse mutation/load cell without claiming a general schema-authority row or +> multi-process native-ref fencing. RFC-024 remains the false-contention +> narrowing step. + +### TIGHTENING-01 — symmetric Lance conflicts are not obviously an activation gate + +RFC-023 correctly identifies Lance's directional filtered/unfiltered behavior. +However, its own mandatory fleet outage, recovery drain, historical-duplicate +validation, stamp-last activation, and old-binary refusal are intended to make +unfiltered writers unreachable after activation. + +The current RFC-023 routing table still permits a direct existing-row `Update`, +whose Lance operation has no inserted-row filter. Therefore unfiltered +transactions do remain reachable after activation. Symmetry can be demoted only +if every insertion-bearing keyed path routes through filtered merge-insert and +the RFC proves a direct update cannot insert a row or change `id`; affected-row +conflict metadata must continue to cover updates to the same existing row. + +After that narrowing, upstream symmetry is useful defense in depth and a +valuable Lance surface guard, but it need not block activation for transaction +pairs that cannot violate key insertion. Two old unfiltered writers are not +protected by symmetry in either case. + +> 💬 **Concur, with the deciding check named (2026-07-11):** the concrete +> unfiltered-`Update` reachable after activation is the matched-only +> partial-schema update merge (`WhenMatched::UpdateAll` + +> `WhenNotMatched::DoNothing`, the field-level-updates path in PR #342): it +> classifies zero inserts, so whether it emits an **empty** filter (`Some` +> with no keys — symmetric-safe) or **no** filter (`None` — the reachable +> unfiltered transaction) is exactly what decides whether this demotion is +> sound. That is a one-test question against the pinned revision and should be +> pinned as a surface guard before the demotion is accepted. + +## Specification and acceptance tightening + +These are smaller than the blockers above, but should be resolved before the +RFC set is merged: + +1. **Checkpoint-name normalization:** define allowed bytes, Unicode + normalization, case sensitivity, maximum encoded length, reserved prefixes, + and normalization-version compatibility. Test collisions and reuse across + versions. +2. **RFC lifecycle values:** [`docs/rfcs/README.md`](../rfcs/README.md#status-values) + defines `Proposed`, `Accepted`, `Declined`, `Superseded by NNNN`, and + `Implemented`. Either use those values or amend the process before using + `draft` and `research-blocked` as machine-readable statuses. Research-blocked + can remain prominent in RFC-027's body while its lifecycle status is + `Proposed`. +3. **Provisional version naming:** sibling RFCs should say “RFC-024 heads + format” rather than treating `v5` as permanent while RFC-024 says the numeral + will change if another format lands first. +4. **Capability ordering:** RFC-025 must specify retention-first, heads-first, + and co-release preservation rules. RFC-026 is a conditional dependency when + streams are enrolled because cleanup must persistently quiesce them. +5. **Memory measurement:** `helpers::cost` measures I/O, not peak RSS. RFC-023's + adopted-merge and RFC-027's lineage memory gates should use the subprocess + `scenarios.rs` harness or an equivalent `wait4`/`ru_maxrss` instrument and + name dataset sizes, baseline, cap, and pass threshold. + + > 💬 **Concur; catches a gap in a 2026-07-11 correction:** the RFC-023 + > §11.4 memory gate added that day states the bound ("peak memory bounded by + > batch size") without naming an instrument that can measure it — this item + > is what makes that gate enforceable rather than aspirational. +6. **Derived-index fallbacks:** acceptance tests must force index-absent and + partially covered states for RFC-024 head lookup and RFC-027 candidate + discovery, assert identical logical results, and assert the promised + degraded-cost/fallback telemetry. + +## Acceptance order after disposition + +The review does not require all RFCs to land together. A safe order is: + +1. accept RFC-022 after branch-control recovery, coarse `graph_head` OCC, and + foreign-branch fact classification are explicit; +2. accept and land the stable identity RFC/capability; +3. accept RFC-023 once partial-effect retry and its format/fleet barrier are + complete; +4. evaluate RFC-024 independently on its physical lookup cost gate; +5. accept RFC-025 after physical protection of both checkpoints and live graph + branches plus a selected upgrade path; +6. accept RFC-026 after its capability barrier and writer-ownership scope are + explicit; +7. keep RFC-027 in research until deletion-delta discovery passes its stated + correctness and flat-cost gates. + +This ordering preserves the split's main benefit: a blocked performance +optimization or research result does not hold correctness work hostage. + +> 💬 **Concur with the order; two sequencing notes (2026-07-11):** +> BLOCKER-04's escalation means step 5 has a prerequisite outside this list — +> the live lazy-branch cleanup exposure needs its red regression test and fix +> in the shipped `cleanup` first, so RFC-025 builds on a correct baseline. And +> step 3's TIGHTENING-01 demotion hinges on the partial-update filter check +> noted there; if that check lands `None`, upstream symmetry returns to the +> activation gate and RFC-023's timeline moves accordingly. diff --git a/docs/dev/testing.md b/docs/dev/testing.md index b038e9fd..8042cb58 100644 --- a/docs/dev/testing.md +++ b/docs/dev/testing.md @@ -22,7 +22,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav | `branching.rs` | Branch create / list / delete, lazy fork | | `merge_truth_table.rs` | Merge-pair truth table (MR-786): all 9×9 `(left_op, right_op)` cells from `{noop, addNode, removeNode, addEdge, removeEdge, setProperty, dropProperty, addLabel, removeLabel}`. Adding a new op to `OpVariant` forces a compile error in `build_case` until the new row + column are dispositioned. 36 executable cells run through real `branch_merge` with a structured oracle (`MergeOutcome` / `MergeConflictKind` + graph-state assert); 45 cells involving `dropProperty`/`addLabel`/`removeLabel` are recorded as `Unsupported` until the mutation grammar grows. | | `merge_fast_forward.rs` | Fast-forward branch-merge cost + correctness: an append-only adopted-source merge routes *new* rows through `stage_append` instead of one whole-delta `stage_merge_insert` (the full-outer join that exhausted the DataFusion memory pool on embedding-bearing tables). The regression gate is structural — it asserts WHICH staged-write primitive the merge invokes via the task-local write probes (`omnigraph::instrumentation`), not a brittle size threshold; also: merge yields source state, defers vector indexes to the reconciler, streams blob columns | -| `writes.rs` | Direct-publish writes: cancellation, non-strict insert/merge rebase under the per-table queue, strict stale-write conflicts, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery); the lance#7444 row-id-overlap regression (`filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent` — merge-load → same-key merge-load → delete → keyed point lookup, green only under the vendored lance-table patch — plus its append-only control) | +| `writes.rs` | Direct-publish writes: cancellation, RFC-022 non-strict full-attempt reprepare from fresh branch authority, strict stale-write conflicts, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery); the lance#7444 row-id-overlap regression (`filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent` — merge-load → same-key merge-load → delete → keyed point lookup, green only under the vendored lance-table patch — plus its append-only control) | | `staged_writes.rs` | TableStore staged-write primitives (`stage_append`, `stage_merge_insert`, `commit_staged`, `scan_with_staged`, `count_rows_with_staged`) — primitive-level only; engine code uses the in-memory `MutationStaging` accumulator instead | | `forbidden_apis.rs` | Defense-in-depth source-walk guard: engine code (`exec/`, `db/omnigraph/`, `loader/`, `changes/`) must not reach around the sealed storage trait to Lance inline-commit APIs, nor open datasets directly (`Dataset::open` / `DatasetBuilder::from_uri`/`from_namespace`) — reads route through `Snapshot::open` and the held-handle cache; `// forbidden-api-allow: ` sentinel exempts reviewed lines | | `failpoint_names_guard.rs` | Source-walk guard (same defense-in-depth shape as `forbidden_apis.rs`): every failpoint call site (`maybe_fail`, `ScopedFailPoint::new`/`with_callback`, `Rendezvous::park_first`) must reference a `failpoints::names` const, never a bare string literal — a typo'd literal compiles and silently never fires | @@ -36,7 +36,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav | `changes.rs` | `diff_between` / `diff_commits` | | `consistency.rs` | Cross-table snapshot isolation, atomic publish | | `lineage_projection.rs` | RFC-013 Phase 7 acceptance gate: graph lineage lives ONLY in `__manifest` — over a realistic history (main commits, a branch, a merge, actors), the production `CommitGraph::open` projection reconstructs the full DAG (commit set, parents, merge parents + merge actor, per-branch heads, inline actors) from the `graph_commit`/`graph_head` rows, and the `_graph_commits.lance` / `_graph_commit_actors.lance` dataset directories are never created at all | -| `schema_apply.rs` | Migration plan + apply, schema-apply lock; index materialization deferred to the reconciler (iss-848): `apply_schema_defers_vector_index_on_empty_table` (an empty-table Vector `@index` never aborts the apply) and `index_only_constraint_apply_touches_no_table_data` (adding an `@index` is metadata-only — no table-version bump); enum widening (iss-enum-widening-migration): `enum_widening_apply_is_metadata_only_and_accepts_new_variant` (no table-version bump; new variant accepted, out-of-set still rejected) + `enum_narrowing_apply_is_refused` (OG-MF-106 with the graph left writable). The planner's widening/narrowing matrix lives in `schema_plan.rs`'s in-source tests | +| `schema_apply.rs` | Migration plan + apply, schema-apply lock; schema-contract publication is pinned by `read_only_open_holds_schema_gate_through_catalog_capture` and `refresh_holds_schema_gate_through_catalog_publication` (source, accepted IR/state, and compiled catalog are captured under one root schema gate). `long_lived_handle_uses_the_schema_catalog_bound_to_its_write_token` covers mutation/load plus a post-apply new node type merged through the pre-apply handle; `stale_handle_branch_delete_gates_tables_added_by_schema_apply` parks delete over that new type while a legacy index reconciler waits, proving merge planning and native-control table envelopes use an operation-local accepted catalog rather than stale ArcSwap state. Index materialization is deferred to the reconciler (iss-848): `apply_schema_defers_vector_index_on_empty_table` (an empty-table Vector `@index` never aborts the apply) and `index_only_constraint_apply_touches_no_table_data` (adding an `@index` is metadata-only — no table-version bump); enum widening (iss-enum-widening-migration): `enum_widening_apply_is_metadata_only_and_accepts_new_variant` (no table-version bump; new variant accepted, out-of-set still rejected) + `enum_narrowing_apply_is_refused` (OG-MF-106 with the graph left writable). The planner's widening/narrowing matrix lives in `schema_plan.rs`'s in-source tests | | `search.rs` | FTS / vector / hybrid (`bm25`, `nearest`, `rrf`) | | `scalar_indexes.rs` | Per-property index dispatch of `build_indices_on_dataset_for_catalog`: enums + orderable scalars get a BTREE (so `=`/range/IN/IS NULL are index-accelerated), free-text Strings keep FTS — observed via `TableStore::key_column_index_coverage`, the same helper the traversal chooser uses | | `traversal.rs` | `Expand`, variable-length hops, anti-join, undirected traversal (`$a $b`, `Direction::Both` — out ∪ in with set-semantics dedup, both-direction anti-join) (CSR path — `OMNIGRAPH_TRAVERSAL_MODE` unset) | @@ -51,10 +51,10 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav | `validators.rs` | Schema constraint enforcement (enum, range, unique, cardinality) across JSONL load, mutation insert/update. ALL THREE write surfaces — mutation, bulk load, AND merge — route through the unified `crate::validate` evaluator (Δ-scoped, index-backed, reusing these leaf checks). Cross-version-uniqueness closure: `cross_version_unique_rejected_on_mutation_insert` + `reinsert_existing_key_is_upsert_not_unique_violation` (mutation path); `cross_version_unique_rejected_on_append_load` + `merge_load_reupsert_existing_key_is_not_unique_violation` (load path). Per-table `Overwrite`: `overwrite_load_validates_ri_against_new_image` (an edges-only overwrite still resolves RI against retained committed nodes) + `append_load_rejects_orphan_edge`. The evaluator's own unit tests live in `src/validate.rs` (`#[cfg(test)]`); its merge-conflict equivalence is pinned by `merge_truth_table.rs` (OrphanEdge) + `branching.rs` (Unique/Cardinality merge tests). Intra-batch duplicate-`@key` rejection on every load mode is pinned by `consistency.rs::loader_rejects_intra_batch_duplicate_keys`; the mutation-coalesce counterpart (insert+update / chained updates of one id are NOT a self-collision) by `writes.rs`. Non-String `@unique` columns probe committed state with a TYPED literal (not a stringified key): `cross_version_unique_rejected_on_date_column` + `noncolliding_write_to_date_unique_column_succeeds` (a `Date @unique` collision is a proper `@unique` violation, and a distinct value does not raise a Date32-vs-Utf8 coercion error). Cardinality is keyed by edge id, last-wins (matching commit's `dedupe_merge_batches_by_id`): `merge_load_edge_src_move_rechecks_vacated_src_cardinality` (a Merge-load moving an edge recounts the vacated src for `@card` min) + `merge_load_duplicate_edge_id_counts_once_per_card` (a dup edge id under two srcs in one batch counts once, no spurious max violation). Direct deletes capture the ids they remove (from the delete op's own scan) into the change-set's `deleted_ids`, so a delete emptying a src is validated: `mutation_delete_edge_below_card_min_rejected` (a `delete Edge` dropping a src below `@card` min is rejected, not silently committed). | | `merge_cost.rs` | Cost-budget tests for branch MERGE on the shared `helpers::cost` harness: `merge_validation_is_delta_scoped` (a 1-row-delta merge opens ≤3 data tables — Δ-scoped, not the whole catalog; was ~6 pre-#5) and `merge_manifest_cost_grows_with_history` (the cross-branch `__manifest` open amplification still grows with commit depth — a separate, not-yet-addressed term — while validation `data_open_count` stays flat) | | `policy_engine_chassis.rs` | Engine-layer Cedar enforcement (MR-722): allow + deny through every `_as` writer via the SDK directly — no HTTP — proving embedded and CLI callers hit the same gate as the server, with action × scope shapes matching `authorize_request` | -| `maintenance.rs` | `optimize` (compaction), `repair` (explicit uncovered-drift publish), and `cleanup` (version GC): empty/idempotent/no-op edges, policy validation, head preservation; `optimize` publishes its own compaction (`optimize_publishes_compaction_to_manifest_so_schema_apply_succeeds`), skips pre-existing uncovered drift (`optimize_skips_preexisting_manifest_head_drift`), and refuses to run while a `__recovery` sidecar is pending (`optimize_defers_when_recovery_sidecar_is_pending`); `repair` previews/heals verified maintenance drift, refuses raw semantic drift without `--force`, and forced repair publishes only by explicit operator choice; the index reconciler (iss-848): `index_build_tolerates_null_vector_rows` (an untrainable Vector column defers instead of aborting the build, sibling indexes still build) and `optimize_materializes_index_declared_but_unbuilt` (optimize creates a declared-but-deferred index) | -| `failpoints.rs` | Failure-injection coverage (gated on `failpoints` feature). Includes the five per-writer Phase B → recovery integration tests (`recovery_rolls_forward_after_finalize_publisher_failure`, `schema_apply_phase_b_failure_recovered_on_next_open`, `branch_merge_phase_b_failure_recovered_on_next_open`, `ensure_indices_phase_b_failure_recovered_on_next_open`, `optimize_phase_b_failure_recovered_on_next_open`) and the write-entry in-process heal contract (the four `*_after_finalize_publisher_failure_heals_without_reopen` tests — load, mutation, schema apply, branch merge: a follow-up write on the same handle rolls a sidecar-covered residual forward without reopen/refresh) and the storage-fault matrix for the sidecar lifecycle (`recovery.sidecar_{write,delete,list}` / `recovery.record_audit` failpoints: Phase A put failure aborts with zero drift, Phase D delete failure is swallowed and healed by the next write, list failures are loud at heal and open, audit-append failures are retried to exactly one audit row; plus the bucket-gated `s3_load_recovers_after_publisher_failure_without_reopen`). And the convergence-idempotent roll-forward regression (`open_sweep_roll_forward_converges_when_manifest_advances_concurrently`: two concurrent open-sweeps race one sidecar at the `recovery.before_roll_forward_publish` rendezvous; the CAS loser must converge, not fail the open — iss-schema-apply-reopen-recovery-race). | +| `maintenance.rs` | `optimize` (compaction), `repair` (explicit uncovered-drift publish), and `cleanup` (version GC): empty/idempotent/no-op edges, policy validation, head preservation; `optimize` publishes its own compaction (`optimize_publishes_compaction_to_manifest_so_schema_apply_succeeds`), skips pre-existing uncovered drift (`optimize_skips_preexisting_manifest_head_drift`), refuses to run while a `__recovery` sidecar is pending (`optimize_defers_when_recovery_sidecar_is_pending`), and compacts blob-v2 tables through the normal path (`optimize_compacts_blob_table_alongside_plain_table`); `repair` previews/heals verified maintenance drift, refuses raw semantic drift without `--force`, and forced repair publishes only by explicit operator choice; the index reconciler (iss-848): `index_build_tolerates_null_vector_rows` (logical load succeeds without inline index work; an untrainable Vector column then defers during reconciliation) and `optimize_materializes_index_declared_but_unbuilt` (optimize creates a declared-but-deferred index) | +| `failpoints.rs` | Failure-injection coverage (gated on `failpoints` feature). RFC-022 includes deterministic post-stage/pre-effect races for mutation/load uniqueness and strict disjoint-head changes, plus the cross-handle post-effect `RecoveryRequired` → read-write-open rollback cell. Control/recovery race cells include `first_touch_post_create_open_error_keeps_recovery_ownership`, `branch_delete_orphans_sidecar_armed_after_initial_barrier`, `branch_merge_fences_target_delete_recreate_aba`, `branch_merge_fences_concurrent_sync_on_same_handle`, `branch_merge_rejects_fresh_target_manifest_change_before_effects`, `branch_merge_rechecks_late_sidecar_after_table_gates`, `cleanup_rechecks_sidecars_under_gc_gates`, and `full_recovery_rereads_sidecar_body_after_discovery`. The suite also includes the five per-writer effect → manifest-CAS recovery tests, write-entry in-process heal contract, storage-fault matrix, S3 recovery twin, and convergence-idempotent roll-forward regression. | | `failpoint_names_guard.rs` | Source-walk guard: every `maybe_fail(...)` call site (engine + cluster) must reference the compile-checked `failpoints::names` consts, never a bare string literal — a typo'd literal compiles but silently never fires; same defense-in-depth shape as `forbidden_apis.rs` | -| `recovery.rs` | Open-time recovery sweep — sidecar I/O, classifier dispatch (NoMovement / RolledPastExpected / UnexpectedAtP1 / UnexpectedMultistep / InvariantViolation), all-or-nothing decision, roll-forward via `ManifestBatchPublisher::publish`, roll-back via `Dataset::restore`, audit row in `_graph_commit_recoveries.lance`, `OpenMode::ReadOnly` skip path | +| `recovery.rs` | Open-time recovery sweep — legacy sidecars plus schema-v3 Mutation/Load exact transaction identity, fixed logical/rollback outcome IDs, branch-token comparison, fresh under-gate sidecar reread/reparse, all-or-nothing roll-forward/rollback/refusal, audit row in `_graph_commit_recoveries.lance`, and `OpenMode::ReadOnly` skip path | | `composite_flow.rs` | Compositional/narrative end-to-end stories — multi-step flows that compose mechanics covered by other test files. Catches integration regressions where individual operations all pass their unit tests but their composition breaks (sequential merges, post-merge main writes, time-travel through merge DAG, reopen consistency over multi-merge histories, post-optimize and post-cleanup strict writes). | ## Fixtures diff --git a/docs/dev/writes.md b/docs/dev/writes.md index e22e85b1..8bb6d720 100644 --- a/docs/dev/writes.md +++ b/docs/dev/writes.md @@ -4,11 +4,14 @@ > removed in MR-771 (shipped v0.4.0). Writes now go directly to the target > table; this document specifies that direct-publish path. -`mutate_as` and `load` write **directly to the target table** -and call `ManifestBatchPublisher::publish` once at the end with -`expected_table_versions` (the per-table manifest versions captured before -the first write). Cross-table OCC is enforced inside the publisher; the -publisher's row-level CAS on `__manifest` is the single fence. +`mutate_as` and `load` prepare against one immutable branch-authority token, +write **directly to the target table**, and call +`ManifestBatchPublisher::publish` once at the end. The token is +`(Lance branch identifier, exact optional graph_head, accepted schema identity)`; +the exact `graph_head` check protects validation dependencies on tables the +write does not touch. Publisher row-level CAS on `__manifest` is the visibility +fence. Process-local branch/table queues reduce retries but are not distributed +authority. ## What this means in practice @@ -19,15 +22,93 @@ publisher's row-level CAS on `__manifest` is the single fence. `__run__*` branch on an upgraded graph is swept off `__manifest` by the v2→v3 internal-schema migration on first read-write open. (The inert `_graph_runs.lance` bytes remain until a `delete_prefix` primitive lands.) -- Cancelled mutation futures leave **no graph-visible state** — the manifest - is never advanced. They can leave two kinds of unreferenced residue, both - self-healing: orphaned Lance fragments (reclaimed by `omnigraph cleanup`), - and — on the *first* write to a table on a branch, which forks it before the - publish — a manifest-unreferenced branch ref. The next write to that table - reclaims the stale fork and re-forks (`reclaim_orphaned_fork_and_refork`), - and `cleanup`'s per-table reconciler is the guaranteed backstop; see the +- Cancelled mutation futures leave **no graph-visible state** unless the + manifest publish already completed. Before that point they can leave + reclaimable uncommitted Lance files, or sidecar-covered committed table + effects that the next quiesced recovery rolls forward/compensates. A + first-touch named-branch write can also leave a target table ref, but it is + never created without ownership: the schema-v3 sidecar is durable first and + names that `(table_path, target ref)`. Reclaim and `cleanup` treat any + matching pending sidecar as a hard stop. Quiesced full recovery accepts both + crash shapes — sidecar durable with no ref yet, or an exact untouched ref at + the inherited version — and removes the latter before deleting the empty + intent. If several pending intents claim one ref, a no-effect intent discards + only itself while any competitor remains; the last no-effect survivor cleans + an untouched ref, or the effect-owning survivor recovers normally. `cleanup` + remains the backstop for genuinely unclaimed legacy/stale refs; see the fork-reclaim note in [invariants.md](invariants.md). +## Mutation/load coarse OCC (RFC-022 first adapter) + +Mutation and load use a closed prepare → effect → publish attempt: + +1. run the branch-aware recovery barrier, then capture the target branch's native Lance + `BranchIdentifier`, exact `graph_head:` (including absence on a + fresh branch), accepted schema identity, and table snapshot; +2. run the complete validator and prepare every effect outside the effect gate. + Existing-table transactions may stage reclaimable files here. A first-touch + named-branch table retains its batch/predicate and pre-mints the transaction + identity instead: Lance branch-local files cannot be staged until its target + ref exists; +3. acquire the schema gate, branch gate, then sorted table queues; re-check for + a relevant sidecar armed since step 1, then revalidate the token. Any + unresolved relevant intent returns typed `RecoveryRequired` before effects; +4. on a pre-effect mismatch, discard the complete attempt. Append/Insert/Merge + reprepare with a bounded retry; strict Update/Delete/Overwrite return typed + `ReadSetChanged`; +5. arm a schema-v3 recovery sidecar. For each deferred first-touch table, create + its target ref, stage branch-local files on that ref, and bind the staged + transaction to the pre-minted UUID. Then commit every planned transaction + with zero transparent conflict retries, confirm exact transaction UUIDs and + table updates, and publish the pre-minted lineage intent under the same token. + +The publisher checks the exact head and native branch identity on every CAS +attempt. It never reparents a validation-sensitive intent after contention. A +mismatch after any physical effect returns `RecoveryRequired` and leaves the +sidecar intact; it is not an ordinary retry loop. + +This adapter preserves the documented single-writer-process support boundary. +The native branch identifier detects delete/recreate ABA but is not a Lance +conditional-ref fence, and destructive recovery remains unsafe beside a live +foreign process. + +### Branch-merge authority fence (adapter bridge) + +Branch merge still uses its writer-specific multi-commit table effects and +confirmation sidecar; it has not yet been converted to the RFC-022 exact-effect +adapter. It does, however, join the closed control boundary needed by this first +slice: after the strict recovery barrier it acquires the root-shared schema gate +and the sorted source/target branch gates, performs the final sidecar check, +loads one operation-local catalog from the accepted contract, captures both graph +heads plus the base/source/target snapshots, and holds those gates through table +effects and manifest publication. Planning stays outside table queues. Before +Phase A, merge acquires the conservative all-catalog table envelope for both +source and target, re-lists sidecars, and compares fresh source/target manifest +versions with the captured snapshots. A stale warm handle catalog or coordinator +snapshot is never accepted as that revalidation. + +That fence prevents a same-process target delete/recreate from reusing the branch +name underneath a merge plan. The race test deliberately recreates a target with +the same name and numeric Lance version but a different `BranchIdentifier`, so +version-only checking cannot accidentally satisfy it. This is a process-local +bridge, not a cross-process conditional-ref primitive and not a substitute for +the later full branch-merge read-set/reprepare adapter. `sync_branch` joins the +same root schema gate before replacing a handle's coordinator, so it cannot +overwrite merge's temporary target coordinator or change a native control's +active-branch authority mid-operation. + +### Branch-delete orphaning exception + +Branch deletion runs the healer first and then holds schema, the target branch, +and every accepted-catalog table gate through the native ref removal. An +unresolved sidecar scoped to that target does not permanently block deletion: +once those gates prove its in-process owner is no longer live, removing the +manifest branch makes its physical effects unreachable. The next write/open +records the orphan-discard recovery audit and deletes the sidecar. A +`SchemaApply` sidecar remains graph-global and blocks deletion. This exception +is specific to removing the authority that made the intent reachable; create, +merge, mutation, and load still reject relevant unresolved ownership. + ## Read-your-writes within a multi-statement mutation A `.gq` query with multiple ops (e.g. `insert Person … insert Knows …`) @@ -47,12 +128,20 @@ shared by both `mutate_as` and the bulk loader: `TableStore::scan_with_pending`, which scans committed via Lance and applies the same SQL filter to the pending batches via DataFusion `MemTable`. Same-query writes are visible to subsequent reads. -- At end-of-query, `MutationStaging::finalize` issues exactly one - `stage_*` + `commit_staged` per touched table (concatenating - accumulated batches; merge-mode dedupes by `id`, last-write-wins), - and the publisher publishes the manifest atomically across all - touched sub-tables. Cross-table conflicts surface as - `ManifestConflictDetails::ExpectedVersionMismatch`. +- Blob-bearing updates use the materializing variant: Lance reads only matched + committed blob payloads as binary, the engine normalizes them back to the + logical blob schema, and the full rows join the same pending-shadow union. + Rewriting matched blob bytes costs I/O proportional to those bytes, but makes + correctness independent of whether a physical index selects a different + Lance merge plan. +- At end-of-query, `MutationStaging::stage_all` prepares exactly one staged + transaction per touched table and `commit_all` commits it (concatenating accumulated + batches; merge-mode dedupes by `id`, last-write-wins), and the publisher + publishes the manifest atomically across all touched sub-tables. Existing + tables stage before gate acquisition; a first-touch named-branch table stages + after sidecar + fork under the gates so its uncommitted files live in the + correct Lance branch tree. Cross-table conflicts surface as typed read-set or + manifest conflicts. - **Deletes stage too (MR-A).** Lance 7.0's `DeleteBuilder::execute_uncommitted` (#6658) makes delete a two-phase op, so deletes no longer inline-commit. Each delete records a predicate in @@ -153,9 +242,9 @@ integrity, and edge cardinality before any Lance HEAD movement, stages each touched table with Lance `Operation::Overwrite`, then runs `commit_staged` under the normal `SidecarKind::Load` recovery sidecar before publishing `__manifest`. `OMNIGRAPH_LOAD_CONCURRENCY` applies to the -fragment-writing stage only; the commit and manifest publish still run -under the per-table write queues. Empty-table overwrite is represented as -a valid zero-fragment Lance `Overwrite` transaction, not as +fragment-writing stage only; the commit and manifest publish run while holding +the root-shared schema → branch → sorted-table gates. Empty-table overwrite is +represented as a valid zero-fragment Lance `Overwrite` transaction, not as truncate-then-append. ### Open-time recovery sweep @@ -170,8 +259,8 @@ test pins. A second, narrower drift class — the **finalize → publisher window** — is closed across one open cycle by the open-time recovery sweep: -`MutationStaging::finalize` runs `stage_*` + `commit_staged` per touched -table sequentially, then the publisher commits the manifest. Lance has +`MutationStaging::stage_all` prepares the table transactions and `commit_all` +runs their independent HEAD advances before the publisher commits the manifest. Lance has no multi-dataset atomic commit, so the per-table `commit_staged` calls are independent operations: if commit_staged on table N+1 fails *after* commit_staged on tables 1..N succeeded, or if the publisher's CAS @@ -179,30 +268,37 @@ pre-check rejects *after* every commit_staged succeeded, tables 1..N are left at `Lance HEAD = manifest_pinned + 1`. **Recovery protocol** (lifecycle of every staged-write writer — -`MutationStaging::finalize`, `schema_apply::apply_schema_with_lock`, +`MutationStaging::commit_all`, `schema_apply::apply_schema_with_lock`, `branch_merge_on_current_target`, `ensure_indices_for_branch`, `optimize_all_tables`): 1. **Phase A**: writer writes a sidecar JSON to - `__recovery/{ulid}.json` BEFORE its first HEAD-advancing commit + `__recovery/{ulid}.json` BEFORE its first independently durable physical + effect (including a first-touch Lance branch ref) or HEAD-advancing commit (`commit_staged`, or `compact_files` for `optimize_all_tables`, which advances the Lance HEAD via a reserve-fragments + rewrite commit rather than a staged write). The sidecar names every `(table_key, table_path, expected_version, post_commit_pin)` it intends to commit + the writer kind + actor_id. + For a first-touch named-branch Mutation/Load table, Phase A is followed by + target-ref creation and branch-local `stage_*`; the sidecar already carries + its pre-minted transaction identity. 2. **Phase B**: writer's per-table `commit_staged` loop runs. - - **Phase-B confirmation (`BranchMerge` only)**: a `BranchMerge` writer + - **Phase-B confirmation:** a `BranchMerge` writer advances each table's HEAD by *several* commits (append → upsert → delete), so a bare "HEAD moved" is ambiguous — it could be a complete publish or one crashed mid-sequence. After the whole per-table loop finishes, the writer re-writes the sidecar stamping each pin's `confirmed_version` with the exact achieved version, then proceeds to - Phase C. This is the commit point of the recovery WAL: a crash *after* - confirmation rolls forward to those versions; a crash *during* Phase B - (sidecar still unconfirmed) rolls back. Other writers don't confirm — - their drift is derived state (index coverage, compaction) that a partial - roll-forward never corrupts. + Phase C. Schema-v3 Mutation/Load sidecars also confirm: each table must + match the staged Lance transaction's `(read_version, uuid)`, and the + sidecar records the exact `SubTableUpdate` plus original lineage intent. + This is the commit point of the recovery WAL: a crash *after* confirmation + rolls forward only when the captured branch token still matches; a crash + *during* Phase B (sidecar still unconfirmed) rolls back. Remaining legacy + writers don't confirm — their drift is derived state (index coverage, + compaction) that a partial roll-forward never corrupts. 3. **Phase C**: publisher commits the manifest. 4. **Phase D**: writer deletes the sidecar. @@ -226,6 +322,18 @@ recovery sweep in `crates/omnigraph/src/db/manifest/recovery.rs`: `BranchMerge` sidecar, a moved HEAD with no `confirmed_version` classifies as `IncompletePhaseB` (a partial multi-commit publish) and forces roll-back; with a `confirmed_version`, roll-forward targets exactly that version. + Schema-v3 Mutation/Load additionally requires `EffectsConfirmed`, the exact + Lance transaction identity at the confirmed version, the original immutable + manifest delta, and a matching captured authority token. A changed token is + rollback-only; an unknown/foreign effect is refused rather than adopted. + An Armed first-touch intent with no owned transaction is deferred by live + roll-forward-only healing because another handle may still own it. Quiesced + full recovery tolerates an absent target ref (crash before fork), or removes + an exact unchanged unpublished ref after proving no other pending sidecar + claims it. With competing claims, the current no-effect sidecar discards + itself without touching the ref; the final survivor owns cleanup/recovery. + During partial rollback, no-effect refs are removed before the rollback + outcome is published so a retry cannot strand them. - If any table is `InvariantViolation` (Lance HEAD < manifest pinned — should be impossible), **abort** with a loud error and leave the sidecar on disk for operator review. @@ -243,14 +351,14 @@ recovery sweep in `crates/omnigraph/src/db/manifest/recovery.rs`: each iteration. The audit row's `to_version` records the logical rolled-back-to version (`manifest_pinned`); the manifest is published at the restore commit (`manifest_pinned + 1`, same content). -- After a successful roll-forward or roll-back, an audit row is - recorded — the graph commit lineage (the `graph_commit` rows in `__manifest` - since RFC-013 Phase 7) carries a commit tagged - `actor_id = "omnigraph:recovery"`, and a sibling - `_graph_commit_recoveries.lance` row carries `recovery_kind`, - `recovery_for_actor` (the original sidecar's actor), `operation_id`, - per-table outcomes. Operators run `omnigraph commit list --filter - actor=omnigraph:recovery` to find recoveries. +- After a successful roll-forward or roll-back, an internal + `_graph_commit_recoveries.lance` row records `recovery_kind`, + `recovery_for_actor` (the original sidecar's actor), `operation_id`, and + exact per-table outcomes. A v3 roll-forward publishes the interrupted + writer's fixed lineage intent, including its original actor; rollback and + legacy recovery commits use `actor_id = "omnigraph:recovery"`. Ordinary + commit history is therefore not a complete recovery enumeration, and the + CLI currently has no public query for the recovery-audit table. - Sidecar deleted as the final step. Triggers for the residual: transient Lance write errors during finalize @@ -264,13 +372,24 @@ contention exceeding `PUBLISHER_RETRY_BUDGET = 5` retries. Phase B → Phase C residual closes on the next write, without a restart and without an explicit refresh. The heal lists `__recovery/` (one `list_dir`; empty in the steady state) and, per sidecar, acquires -the same per-`(table_key, table_branch)` write queues every sidecar -writer holds from before `write_sidecar` until after `delete_sidecar` — -so it serializes against a live writer instead of rolling its +schema → branch → sorted-table gates that overlap the writer's guarded +sidecar lifetime. RFC-022 mutation/load writers hold the complete order. Branch +merge now holds schema plus source/target branch authority for its whole attempt +and then the all-catalog source/target table envelope; other legacy adapters +serialize through their existing table or schema gate until their own adapter +slices land. The +manager is shared by every +`Omnigraph` handle for one canonical local root identity (relative, absolute, +and symlink aliases converge; object-store/custom schemes stay opaque), so this +also serializes a refresh or separately-opened handle against a live writer instead of rolling its in-flight sidecar forward from under it (a sidecar whose queues can be acquired belongs to a writer that finished or died; an existence re-check after the wait skips the finished case). Lock order is -queues → coordinator, matching every writer's commit→publish path. +schema → branch → sorted tables → coordinator, matching the writer effect path. +Enrolled mutation/load attempts and branch merge perform one additional +`list_dir` after acquiring their authority gates; that final check closes the +pre-gate recovery TOCTOU without moving mutation/load validation or staged-file +construction under the gate. Pinned by the four `tests/failpoints.rs::*_after_finalize_publisher_failure_heals_without_reopen` tests (load, mutation, schema apply, branch merge). The maintenance @@ -280,8 +399,11 @@ Phase-B commit (dropping its rows), and a branch merge publishes the drift as an unattributed side effect — both while the stale sidecar lingers to misclassify later. Sidecars that would require a `Dataset::restore` (mixed / unexpected -state) are deferred to the next `OpenMode::ReadWrite` open: restore is -unsafe under concurrency because Lance's `check_restore_txn` accepts +state) are deferred to the next `OpenMode::ReadWrite` open. Full open-time +recovery uses the same root-scoped ordered gates and post-wait existence check, +so it cannot Restore/delete under a live writer owned by another handle in the +same process. Restore remains unsafe across processes because Lance's +`check_restore_txn` accepts the restore against in-flight Append/Update/Delete commits and silently orphans them (pinned by `tests/staged_writes.rs::lance_restore_loses_to_concurrent_append_via_orphaning`). @@ -289,17 +411,24 @@ When such a deferred sidecar blocks a write, the commit-time drift guard says so explicitly ("a pending recovery sidecar requires rollback — reopen the graph read-write") instead of pointing at `omnigraph repair`, which refuses while a sidecar is pending. +`cleanup` refuses pending sidecars at entry as well, before orphan reconciliation +or version GC: v3 ownership and compensation recovery may need the retained +Lance transaction/version history, so garbage collection cannot outrun the +recovery barrier. Continuous in-process recovery for the rollback path is the goal of a -future background reconciler. `ensure_indices` does not heal at entry -itself — it runs inside the load / schema-apply flows after their -entry heal, and its strict preconditions still fail loudly on drift -when invoked directly. +future background reconciler. `ensure_indices` does not heal at entry itself; +it is an explicit maintenance/reconciliation call, separate from mutation, +load, and schema apply, and its strict preconditions fail loudly on drift. -The publisher-CAS contract is unchanged: a *concurrent writer* that -advances any of our touched tables between snapshot capture and -publisher commit produces exactly one winner. The residual above is -about *our* abandoned commits in the failure path, not about -concurrency races. +For enrolled mutation/load, the publisher rechecks the attempt's exact native +branch identity and `graph_head` as well as the touched-table versions. A +concurrent graph commit anywhere on the target branch therefore invalidates the +prepared authority instead of silently reparenting it. Before effects, an +insert-only mutation or Append/Merge load fully reprepares with a bounded retry; strict +Update/Delete/Overwrite returns `ReadSetChanged`; after any effect, any later +error returns `RecoveryRequired` and leaves the fixed v3 intent durable. Legacy +writers still arbitrate only their explicit touched-table expectations until +their adapters are enrolled. **Sidecar I/O failure semantics** (all sidecar I/O goes through the backend-generic `StorageAdapter`; the contracts below are pinned by the @@ -341,19 +470,25 @@ error, not a silent `false`. ## Conflict shape -Concurrent writers to the same `(table, branch)` produce exactly one -success and one failure. The losing writer's error is -`OmniError::Manifest` with kind `Conflict` and details -`ManifestConflictDetails::ExpectedVersionMismatch { table_key, expected, -actual }`. The HTTP server maps this to **409 Conflict** with body -`{"error": "...", "code": "conflict", "manifest_conflict": { "table_key": -"...", "expected": N, "actual": M }}` — see [docs/user/server.md](../user/operations/server.md). +For mutation/load, a changed authority detected before effects is +`ManifestConflictDetails::ReadSetChanged { member, expected, actual }`. +Retryable Insert/Merge/Append attempts handle this internally by fully +repreparing; strict writes surface **409 Conflict** with structured +`read_set_conflict` details. A changed authority discovered after a physical +effect, or any unresolved overlapping intent found at the synchronous recovery +barrier, is `OmniError::RecoveryRequired { operation_id, … }`, mapped to **503 +Service Unavailable** with structured `recovery_required`; retry only after the +sidecar has been resolved. Legacy, not-yet-enrolled writers may still surface +`ExpectedVersionMismatch` and `manifest_conflict`. -## Audit +## Commit actor history `actor_id` lands in the graph commit lineage — the `graph_commit` rows in `__manifest`, written in the publish CAS (RFC-013 Phase 7; previously -`_graph_commits.lance`). Audit history is queried via `omnigraph commit list`. +`_graph_commits.lance`). Ordinary commit/actor history is queried via +`omnigraph commit list`. Crash-recovery actions additionally live in the internal +`_graph_commit_recoveries.lance` table described above; that exact recovery log +does not yet have a public CLI query. ## Storage versioning (no in-place migration) diff --git a/docs/rfcs/rfc-0018-ingest-wal.md b/docs/rfcs/rfc-0018-ingest-wal.md index 665e4ca5..b7508816 100644 --- a/docs/rfcs/rfc-0018-ingest-wal.md +++ b/docs/rfcs/rfc-0018-ingest-wal.md @@ -2,7 +2,7 @@ type: spec title: "RFC-018 — Streaming-ingest WAL on Lance MemWAL" description: Adds a durability-first streaming ingest path (ack on WAL durability, asynchronous fold into the graph commit chain) built entirely on Lance's MemWAL primitive; reconciled against Lance v8.0.0 and the v9 beta line; analyzed for composition with the upstream multi-table-commit RFCs. -status: draft +status: superseded tags: [eng, rfc, wal, ingest, lance, omnigraph] timestamp: 2026-07-02 owner: @@ -10,12 +10,19 @@ owner: # RFC-018 — Streaming-ingest WAL on Lance MemWAL -**Status:** Draft / for discussion +**Status:** Superseded by [RFC-026](rfc-026-memwal-streaming-ingest.md) **Date:** 2026-07-02 **Surveyed version:** 0.7.2 (branch `dst-extract-crate`); Lance pinned at 7.0.0 **Upstream surveyed:** Lance v8.0.0 (released; RC votes closed 2026-07-01), v9.0.0-beta.10; MemWAL spec (`lance.org/format/table/mem_wal/`, fetched in full 2026-07-02); discussions #7260, #7264, #7222, #7176 **Audience:** OmniGraph maintainers +> **Supersession note (2026-07-10):** RFC-026 carries the streaming-ingest +> design forward under RFC-022's unified graph-write protocol. It also corrects +> this draft's characterization of MemWAL: MemWAL is a strategic Lance +> architecture and a major substrate investment, not an experimental direction. +> The integration risk is its evolving API and format surface across Lance +> releases. + --- ## 0. TL;DR @@ -125,8 +132,10 @@ fencing primitive. ## 3. Substrate inventory — what Lance provides and how we use all of it Per the lance.md protocol the MemWAL spec and adjacent pages were fetched in -full (2026-07-02). The spec is **experimental** upstream — risk register in -§10. Inventory, mapped to consumption: +full (2026-07-02). MemWAL is a strategic Lance architecture with substantial +upstream investment. Its API and format surface are still evolving across +releases; §10 treats that maturity boundary as an integration risk. Inventory, +mapped to consumption: | Lance tooling | Spec/PR | How this RFC uses it | |---|---|---| @@ -323,8 +332,8 @@ this RFC phases on: matures it may fit *small in-place updates* better than WAL-upsert-fold; watch-listed as a possible Phase 4 refinement, not a dependency. - MemWAL fixes keep landing on v9 betas (#7489 cross-generation block-list on - in-memory scan arms) — confirming the experimental-spec churn risk (§10) - and the value of keeping our exposure transient-state-only. + in-memory scan arms) — confirming that its API/format integration surface is + still moving (§10) and the value of keeping our exposure transient-state-only. ## 7. Composition with upcoming Lance multi-table commits @@ -464,8 +473,9 @@ participates in publication authority. ## 10. Risks -- **MemWAL is experimental upstream** (spec banner; live format votes — - #7418 Status field mid-2026-06). Mitigation is structural: WAL state is +- **MemWAL's API and format surface continues to evolve.** This risk concerns beta-era + API churn, not architectural commitment: Lance has made MemWAL a strategic + streaming-write investment. Mitigation is structural: WAL state is *transient* (folded then GC'd), so a format change between Lance versions can be handled by fold-to-quiescent before the bump; no long-lived on-disk state depends on the MemWAL format. This must stay true — resist any diff --git a/docs/rfcs/rfc-0019-heads-and-fences.md b/docs/rfcs/rfc-0019-heads-and-fences.md index 2319706e..c68f5e1e 100644 --- a/docs/rfcs/rfc-0019-heads-and-fences.md +++ b/docs/rfcs/rfc-0019-heads-and-fences.md @@ -2,7 +2,7 @@ type: spec title: "RFC-019 — Heads and Fences: structural O(1) writes without a warm-cache truth fork" description: Replaces the warm-publish/pinned-open machinery of PR #318 with two structural changes — durable per-table head rows ("refs, not replay") and substrate-native key-conflict fencing (Lance's unenforced-PK KeyExistenceFilter) — landing together as internal schema v5; composes with RFC-018 (WAL ingest) and the upstream multi-table-commit direction. -status: draft +status: superseded tags: [eng, rfc, write-path, manifest, lance, omnigraph] timestamp: 2026-07-04 owner: @@ -10,12 +10,21 @@ owner: # RFC-019 — Heads and Fences -**Status:** Draft / for discussion +**Status:** Superseded by [RFC-023](rfc-023-key-conflict-fencing.md) and [RFC-024](rfc-024-durable-table-heads.md) **Date:** 2026-07-04 **Surveyed:** omnigraph `main` @ 98530a0e (0.8.0); Lance pinned 7.0.0 (+ vendored lance-table carrying lance#7480); upstream Lance v8.0.0 (released 2026-07-01), v9.0.0-beta.15; PR #318 at `2aab48ba` (reviewed 2026-07-04, 8 verified findings) **Companion docs:** RFC-018 (streaming-ingest WAL), PR #318's plan doc (`unlimited-history-latency-plan.md`, whose §9 "U2" this RFC promotes from follow-up to prerequisite) **Audience:** OmniGraph maintainers +> **Supersession note (2026-07-10):** key-conflict fencing and durable table +> heads are separate irreversible decisions with different substrate gates and +> rollout requirements. RFC-023 and RFC-024 review them independently under +> RFC-022's shared graph-write protocol. RFC-023 also corrects this draft's +> load-bearing symmetry claim: on the surveyed Lance revision, an unfiltered +> current transaction (including bare `Append`) can rebase after a filtered +> update. Fencing therefore requires both transaction orders plus a fleet +> compatibility barrier; it is not “unblocked for new tables” in a mixed graph. + --- ## 0. TL;DR diff --git a/docs/rfcs/rfc-022-unified-write-path.md b/docs/rfcs/rfc-022-unified-write-path.md new file mode 100644 index 00000000..68059ac1 --- /dev/null +++ b/docs/rfcs/rfc-022-unified-write-path.md @@ -0,0 +1,773 @@ +--- +type: spec +title: "RFC-022 — Unified graph-write protocol" +description: One correctness protocol for graph-visible writes, with synchronous recovery, complete read-set arbitration, writer-specific physical-effect adapters, and explicit control-plane exceptions. +status: draft +tags: [eng, rfc, write-path, manifest, recovery, concurrency, lance, omnigraph] +timestamp: 2026-07-10 +owner: +--- + +# RFC-022: Unified graph-write protocol + +**Status:** Draft / for team review +**Date:** 2026-07-10 +**Surveyed:** OmniGraph 0.8.1 (`main`); Lance 9.0.0-beta.15, git rev `f24e42c1` +**Audience:** engine and storage maintainers +**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md). +Findings marked **BLOCKER** must be dispositioned before acceptance. + +--- + +## 0. Summary + +OmniGraph will have one **correctness protocol** for every operation that changes +manifest-resolved graph state. The protocol does not require every writer to use +the same Lance primitive. A mutation can commit one staged transaction per table, +a branch merge can make several commits to a table, and optimize can use Lance +operations that have no staged API. Writer-specific **effect adapters** describe +those physical operations; one coordinator enforces the common safety rules. + +A graph-visible write follows this state machine: + +```text +recovery barrier + → prepare pinned base + complete ReadSet + Effects + → acquire ordered process-local gates + → revalidate the complete ReadSet, or restart + → durably arm recovery + → apply writer-specific physical effects + → publish exactly one graph-visible __manifest CAS + → finalize derived state +``` + +The protocol has three hard boundaries: + +1. Recovery is a synchronous pre-write safety barrier. It may also run in the + background, but a writer never proceeds merely because recovery was scheduled. +2. Validation and merge classification are valid only for their complete read set. + A changed probed table causes revalidation or a full restart; the writer never + refreshes expected versions underneath a plan computed from an older base. +3. “One `__manifest` CAS” applies only to graph-visible commits. Native graph-branch + ref creation/deletion and physical-only maintenance have explicit, smaller + control protocols and do not manufacture graph commits. + +This RFC deliberately does not combine key fencing, durable table heads, +checkpoint retention, MemWAL ingest, or lineage-based merge-delta discovery into +one format and rollout. They are focused follow-ups: + +- [RFC-023 — Key-conflict fencing](rfc-023-key-conflict-fencing.md) +- [RFC-024 — Durable table heads](rfc-024-durable-table-heads.md) +- [RFC-025 — Checkpoint retention](rfc-025-checkpoint-retention.md) +- [RFC-026 — MemWAL streaming ingest](rfc-026-memwal-streaming-ingest.md) +- [RFC-027 — Lineage merge deltas](rfc-027-lineage-merge-deltas.md) + +## 1. Scope and authority + +### 1.1 Graph-visible writes + +A **graph-visible write** changes state resolved through a graph manifest snapshot: + +- a node or edge table version; +- a registered or tombstoned table; +- accepted schema identity or schema-visible table metadata; +- a graph commit or graph head; +- any future logical marker that changes query or time-travel semantics. + +For such an operation, the only visibility point is one successful `__manifest` +commit containing the entire graph delta. Per-table Lance commits before that point +are physical effects covered by recovery; they are not independently graph-visible. + +### 1.2 Control operations + +Two classes are not graph-visible commits: + +1. Native graph-branch ref create/delete. Lance stores these refs outside the + dataset-version chain and creates no new `__manifest` version for either action. +2. Physical-only maintenance whose result is content-equivalent and is not selected + through a data-table pointer in `__manifest`, such as compacting `__manifest` + itself or reclaiming unreachable files. + +Sections 7 and 8 define their control protocols. Branch **merge** and a data-table +optimize whose new version must be published are graph-visible writes and remain in +the main protocol. + +### 1.3 What “unified” means + +Unified means one set of safety obligations, one coordinator state machine, and one +closed registry of effect adapters. It does **not** mean: + +- one storage trait method for every Lance operation; +- exactly one physical commit per table; +- pretending native refs are manifest rows; +- treating process-local queues as a distributed transaction manager; +- moving commit recovery out of the correctness path. + +## 2. Protocol objects + +The names below are conceptual. Implementations may choose different Rust names, +but they must preserve the represented information and transitions. + +```rust +struct PreparedWrite { + operation_id: OperationId, + writer_kind: WriterKind, + target: BranchTarget, + base: BaseView, + read_set: ReadSet, + effects: Effects, + manifest_delta: ManifestDelta, + lineage_intent: Option, + recovery: RecoveryPlan, +} +``` + +### 2.1 `BaseView` + +`BaseView` is the immutable state against which the operation was computed. It +contains at least: + +- target manifest branch and its incarnation/freshness token; +- pinned manifest version; +- accepted schema identity; +- pinned table entries used by the operation; +- graph head when parentage, merge base, or branch semantics depend on it. + +The base is captured only after the recovery barrier completes. A recovery pass may +advance the manifest or promote schema state, so a base captured before recovery is +not valid write input. + +### 2.2 `ReadSet` + +`ReadSet` is every authority value whose stability is required for the prepared +result to remain correct. It includes, as applicable: + +- target branch incarnation; +- accepted schema identity; +- graph head; +- the table head for every table written; +- the table head for every table probed by uniqueness, referential-integrity, + cardinality, policy-independent structural validation, or merge classification; +- writer-specific authority, such as an enrolled stream's configuration and merge + generation. + +A table belongs in the read set because its value affected the decision, not because +the writer happens to update it. Read-only dependencies are load-bearing. + +The publisher must arbitrate every read-set member atomically with the manifest +commit. A fresh pre-check followed by an unconditional write is not a CAS. The +implementation must ensure that a concurrent change after the check contends on a +stable authority row or equivalent substrate token. If the current representation +cannot arbitrate a read-only dependency, that writer has not completed this RFC and +must not ship the outside-gate validation optimization. + +### 2.3 `Effects` + +`Effects` is an adapter-owned physical plan. Each effect declares: + +- physical dataset and Lance branch/ref it targets; +- expected pre-state; +- whether it is staged, inline, ref-only, or zero-commit; +- the possible post-state shape: exact version, bounded range, or adapter-confirmed + version; +- whether rollback is safe; +- how recovery distinguishes no movement, partial movement, completed movement, + and an already-published result. + +The generic coordinator never assumes `expected_version + 1`. That assumption is +valid for some mutation/load effects and false for branch merge, compaction, index +work, schema metadata changes, and no-op plans. + +### 2.4 `ManifestDelta` + +`ManifestDelta` is the complete logical result to publish. It contains table-version +journal entries, registrations/tombstones, mutable logical heads when present, schema +identity, and lineage rows as required by the operation. Its logical contents are +immutable for one prepared attempt. Physical version fields may be declared output +slots that the adapter binds to an allowed, confirmed effect result; binding such a +slot cannot widen or otherwise change the logical plan. A revalidation mismatch +discards the delta and restarts preparation. + +### 2.5 `RecoveryPlan` + +`RecoveryPlan` supplies the writer-specific classifier and compensation/roll-forward +rules. It must be serializable into a recovery sidecar before the first independently +durable physical effect. It is part of the commit protocol, not a best-effort +maintenance hint. + +## 3. Normative invariants + +1. **Recovery before base capture.** Every graph-visible writer runs and awaits the + recovery barrier before pinning `BaseView`. +2. **No durable effect before durable intent.** On an existing physical ref, + reclaimable uncommitted files may be staged first. A first-touch named-table + transaction is different: Lance writes its uncommitted files into the opened + branch tree, so Prepare retains the logical batch/predicate and pre-mints its + transaction identity; after revalidation the recovery sidecar becomes durable, + then the writer creates the target ref and stages those branch-local files under + the held gates. In every case the sidecar precedes the first independently + persistent physical effect, including a native table ref, data-table HEAD + advance, or native tag/index mutation needed by the graph-visible result. +3. **Complete read-set validation.** Physical effects may start only while the fresh + authority state equals the complete `ReadSet` used to prepare the operation. +4. **Recompute, do not patch.** On a pre-effect mismatch, discard the prepared plan + and rerun validation/classification. Never refresh expected versions beneath an + old plan. +5. **One graph visibility point.** A graph-visible write publishes one manifest CAS; + no subset of its table effects becomes graph-visible independently. +6. **Adapters own effect truth.** Every effect-producing writer uses a registered + adapter. There is no generic fallback that guesses version movement or recovery + safety. +7. **Queues are local optimization.** Ordered process-local gates prevent avoidable + in-process races and deadlocks. Cross-process correctness comes from Lance + conflicts, manifest arbitration, and recovery. +8. **Recovery safety is synchronous.** Background sweeping is permitted, but a later + overlapping writer waits for, performs, or fails on unresolved recovery before it + can advance state. +9. **Derived work follows visibility.** Expensive index reconciliation, cache warming, + and orphan reclaim may run asynchronously only when logical correctness does not + depend on their completion. + +These refine, and do not weaken, invariants 2, 3, 4, 5, 7, 9, 13, and 15 in +[`docs/dev/invariants.md`](../dev/invariants.md). + +## 4. The graph-write state machine + +### 4.1 Stage A — recovery barrier + +Before accepting a base, the coordinator discovers pending recovery intents that can +overlap this operation. It must reach one of three outcomes: + +1. all relevant intents are fully resolved; +2. each remaining intent is proven already satisfied and can be finalized safely; +3. the write fails with a typed recovery-required or live-writer-contention error. + +“Spawn recovery and continue” is not an outcome. Listing sidecars may run concurrently +with non-authoritative I/O, but preparation cannot accept its base until recovery has +completed and the post-recovery schema/manifest state is available. + +Recovery must not destructively act on a sidecar that may belong to a live writer +without ownership/fencing proof. It waits, returns contention, or uses a protocol that +proves the prior owner cannot continue. + +Recovery classifiers use exact effect identity or confirmed post-state. A numeric +test such as `manifest_version >= observed_lance_head` is sufficient only when the +adapter independently proves lineage containment; version ordering alone is not that +proof. Exact identity cuts both ways: a roll-forward that loses its manifest CAS to +a concurrent writer which already published this intent's exact goal state is +**convergence, not failure** — the barrier records the audit outcome and removes the +intent rather than failing the write (preserving the concurrent-advance convergence +behavior fixed in #296). "Exact" forbids adopting an unrelated newer version; it does +not forbid recognizing one's own goal already achieved. + +The barrier is a deliberate availability trade, stated plainly: an unresolvable +overlapping recovery intent — including a live foreign writer's sidecar that cannot +be fenced — fails every overlapping write with a typed error until resolved. Safety +outranks availability here by design; operators observe the condition through the +typed error and recovery telemetry rather than through silently degraded writes. + +### 4.2 Stage B — prepare + +Prepare runs without writer gates. It may: + +- capture `BaseView`; +- evaluate the complete constraint set; +- classify a merge; +- compute embeddings or other deterministic payloads; +- build staged Lance transactions and reclaimable uncommitted objects for targets + whose physical refs already exist; +- for a first-touch named-table target, retain the complete logical stage input and + pre-mint its recovery transaction identity without creating the ref; +- construct the complete `ReadSet`, `Effects`, `ManifestDelta`, and `RecoveryPlan`. + +Prepare must not advance a Lance HEAD or the graph's native branch refs. Staged files +that are not referenced by a committed Lance manifest are permitted and are +reclaimable if the attempt restarts. + +After Stage E arms recovery, a first-touch adapter may create its declared target ref +under the held gates, stage branch-local files on that ref, bind the resulting Lance +transaction to the pre-minted identity, and only then advance HEAD. The ref is itself +an independently durable effect covered by the sidecar; recovery must classify both +sidecar-before-ref and ref-before-HEAD crash states. + +Every validator registers its actual committed-state probes in `ReadSet`. Key fences +are an additional same-key conflict signal; they do not replace read-set arbitration +for non-key uniqueness, RI, cardinality, or merge-target stability. + +### 4.3 Stage C — acquire ordered gates + +The prepared effects declare their gate keys. One attempt acquires the complete set in +this total order: + +1. durable global operation claims (migration, retention, or future claims) by + deterministic claim key; +2. graph/schema-control key, if required; +3. target branch-control key, if required; +4. `(physical_branch, table_key)` keys in deterministic sorted order. + +The gates are held through revalidation, recovery arming, physical effects, and the +manifest visibility decision. They may be released after a successful manifest CAS or +after a failed post-effect attempt has safely left its sidecar for recovery. + +Acquiring a global claim is coordination, not a graph effect for §3's sidecar-order +rule: the claim record must itself contain an owner/fencing token and an explicit +crash-release/takeover contract. No data HEAD, tag, index, or logical authority may +move merely because the claim was acquired. + +The merge-exclusive mutex may protect a coordinator swap, but it is not a semantic +cross-process lock and must not substitute for a target read set. + +### 4.4 Stage D — revalidate or restart + +With all gates held, the coordinator loads fresh authority state and compares every +member of `ReadSet`. + +- If all members match, the attempt may arm recovery. +- If any member differs, no physical effect may run. The attempt releases its gates, + discards staged state, and restarts from Prepare. +- A strict API may return a typed conflict instead of retrying, but it may not publish + the stale plan. + +Retries are bounded and observable. Retrying a merge means recomputing the merge base +and reclassifying against the new target. Retrying a validation-sensitive mutation +means rerunning the validators, including probes of tables the mutation does not +write. + +### 4.5 Stage E — arm recovery + +After successful revalidation, write the recovery sidecar durably before the first +independently durable physical effect. An effect-free or authority-first workflow +described in §4.9 may omit the sidecar. Otherwise the sidecar contains at least: + +- operation id, writer kind, actor, and target branch/incarnation; +- pinned schema identity and complete read set; +- every physical target and expected pre-state; +- adapter recovery strategy; +- intended manifest delta and lineage intent, or a durable reference to them; +- confirmed post-state once the adapter reaches its all-effects-complete boundary. + +A multi-step adapter first records its pre-state plan. After all its physical effects +finish, it durably records the exact confirmed post-state before manifest publish. +Until that confirmation exists, recovery treats the effect set as possibly partial. + +### 4.6 Stage F — apply effects + +The adapter applies its declared physical effects. A failure after recovery is armed +leaves the sidecar intact. The request must not delete it, silently adopt live HEAD, or +start a fresh plan around it. + +The adapter returns exact achieved state to bind the physical output slots declared by +`ManifestDelta`. If achieved state differs from the prepared effect envelope, the +operation fails into recovery rather than widening its plan in place. + +### 4.7 Stage G — manifest CAS + +A graph-visible operation performs exactly one `__manifest` CAS carrying its complete +logical delta and lineage when the plan has `LineageIntent`. Metadata-only plans carry +their explicit authority/operation rows without manufacturing graph lineage. On every +CAS attempt, the commit authority re-reads and arbitrates the complete `ReadSet`. + +The following cases are distinct: + +- **Conflict before any physical effect:** safe bounded restart from Prepare. +- **Conflict after physical effects:** recovery case. Keep the sidecar; do not simply + re-stage or point the manifest at whatever HEAD is now live. +- **CAS success:** the graph commit is visible atomically. + +When durable table heads land under RFC-024, tombstoning a table must update its mutable +head to an explicit deleted state in this same CAS. A stale live head plus an immutable +tombstone history is not a valid O(tables) current-state representation. + +### 4.8 Stage H — finalize + +After CAS success: + +- delete the sidecar best-effort when the workflow has one; +- refresh/invalidate process-local views; +- enqueue derived index reconciliation and orphan reclaim; +- record recovery/audit completion when applicable. + +Sidecar deletion failure does not turn a durable successful graph commit into a user +error. The next recovery barrier proves the exact intent satisfied and removes the +artifact. + +### 4.9 Authority-first control workflows + +Some metadata workflows have no independently durable physical effect before their +manifest CAS. They may use an **authority-first** subtype: + +1. run the recovery barrier, prepare, acquire gates/claims, and revalidate exactly as + above; +2. publish the metadata transition as the first durable effect; +3. perform only idempotent work whose desired target is fully encoded by that + transition and whose interruption cannot expose incorrect graph data. + +No generic sidecar is required before step 2 because there is no pre-authority effect +to recover. The authority row itself is the durable recovery cursor. Checkpoint +deletion, a GC-boundary publish, and `OPEN -> DRAINING` stream intent are candidate +examples; each follow-up RFC must prove its post-CAS work is convergent and safe. + +A long control workflow is not one giant RFC-022 attempt. Each lifecycle transition, +fold, schema/branch operation, and resume transition is a separate prepared write or +native-ref control step with its own read set and visibility point. If a later phase +needs a non-idempotent or independently visible effect not fully described by the +authority row, that phase uses a normal sidecar before the effect. + +## 5. Crash contract + +| Crash point | Required result | +|---|---| +| Before sidecar | No independently durable effect occurred; uncommitted objects are reclaimable. | +| Sidecar durable, no effect | Recovery aborts/finalizes the empty intent. | +| Some effects applied, not confirmed | The adapter rolls back, completes, or refuses safely according to its declared strategy. | +| All effects confirmed, before manifest CAS | Recovery rolls forward the exact confirmed manifest delta, or applies the adapter's explicit all-or-nothing rule. | +| Manifest CAS succeeded, sidecar remains | Recovery proves the exact intent visible, audits it, and removes the sidecar. | + +Rollback is not assumed safe. Lance `Restore`, schema-file promotion, native refs, and +content-replacing operations have different concurrency properties; each adapter must +state which recovery direction is legal and under what fencing. + +## 6. Writer-effect adapters + +The adapter registry is closed by default: adding a graph-visible writer requires a +new adapter or an explicit use of an existing adapter. Code review and tests must be +able to enumerate every adapter and every entry point that invokes it. + +### 6.1 Mutation and load + +- Construct one staged effect per touched table where the Lance API permits it. + Existing-table effects may stage in Prepare; first-touch named-table effects use + the sidecar → target ref → branch-local stage ordering above. +- Put every uniqueness, RI, and cardinality probe in `ReadSet`. +- Revalidate or restart when a probed-but-untouched table changes. +- Preserve strict replacement semantics for overwrite/delete. +- Treat key-conflict fencing and strict keyed Append semantics as RFC-023 concerns; + no fence is credited as protection until that RFC's rollout gates pass. + +### 6.2 Branch merge + +- Compute row classification outside the gates. +- Include the target graph head and every target table used by classification or + validation in `ReadSet`. +- Any target change before effects forces a complete reclassification; publishing a + result computed against an old target and parenting it to a new live head is + forbidden. +- The adapter supports zero, one, or several physical commits per table and records + exact confirmed post-state before manifest publish. +- Lineage-based candidate discovery may replace the classifier only under RFC-027; + this protocol does not assume it is O(delta). + +### 6.3 Schema apply and storage migration + +- Acquire the schema-control gate before effect application. +- Include accepted schema identity and every affected table in `ReadSet`. +- Cover schema staging-file promotion, data-table schema/field-metadata commits, + registrations, tombstones, and final schema identity with one recovery intent. +- Write the sidecar before the first table HEAD advance, including unenforced-PK + metadata backfill or other inline metadata commits. +- A branch-wide or graph-wide migration must enumerate every physical manifest/data + branch it changes; updating main does not implicitly migrate older branch manifests. + +### 6.4 Data-table optimize and index work + +- The adapter may describe zero or multiple inline, content-preserving Lance commits. +- It records the exact achieved version rather than assuming one version of movement. +- If the new data-table version is selected through `__manifest`, publishing that + pointer is a graph-visible commit and uses this protocol. +- Logical operations never fail because a derived index is absent or behind. +- Physical-only internal-table maintenance remains the exception in Section 8. + +### 6.5 MemWAL fold + +RFC-026 owns enrollment, acknowledgement, quiescence, fresh-read semantics, and the +public ingest surface. Any fold that becomes graph-visible is an adapter here: + +- fold-time validation contributes its complete read set; +- Lance `merged_generations` changes atomically with the base-table data commit; +- the sidecar covers the data-commit-to-manifest gap; +- one successful manifest CAS makes the folded graph state visible. + +## 7. Native graph-branch ref control protocol + +Creating or deleting a graph branch mutates a native Lance branch ref for +`__manifest`. Lance specifies that these operations do not generate a dataset version. +There is no target branch on which to publish before create, and no target remains on +which to publish after delete. They therefore cannot truthfully be instances of the +graph-visible manifest-CAS protocol. + +Their control protocol is: + +1. run and await the recovery barrier; +2. quiesce enrolled streams as required by RFC-026; +3. acquire any active global claim, such as RFC-025's retention claim, and then + the graph/branch-control gate in §4.3 order; +4. freshly revalidate source ref, target existence, and branch incarnation; +5. perform one native Lance ref mutation, which is the visibility point; +6. release the gate and reclaim orphaned per-table forks asynchronously. + +Delete has one recovery disposition that create does not: after the complete +schema/target-branch/all-table gate set has waited out any live in-process owner, +an unresolved sidecar scoped to the branch being removed may be rendered +unreachable by the native ref deletion. A later heal records the orphan-discard +audit and retires it. Graph-global schema recovery still blocks the control, and +create/merge may not adopt this exception. + +The native ref operation itself should enforce the freshly checked precondition or +surface concurrent ref mutation as a conflict — but at the pinned Lance revision it +does not: branch-ref creation is an existence check followed by an unconditional +put, not a conditional primitive (the same fact for which RFC-025 §2.3 rejects a +branch ref as a claim mechanism). Until Lance ships a conditional/CAS ref mutation, +graph-branch create/delete therefore inherit the documented single-writer-process +support boundary — the same disposition RFC-023 §10 applies to recovery ownership — +and multi-process branch operations are not advertised. The upstream ask for a +conditional ref primitive is filed alongside this RFC; a process-local branch gate +remains a local optimization, not the missing cross-process guarantee. + +These operations do not emit a synthetic graph commit. If a future product contract +requires a native ref mutation and manifest/audit rows to become atomic together, it +needs a separate multi-authority recovery protocol; this RFC does not claim an +atomicity the substrate does not provide. + +This exception applies only to graph-level create/delete. Branch merge is a +graph-visible write. Lazy per-table forks created while preparing a branch write are +declared physical effects of that writer and remain subject to its recovery/reclaim +contract. + +## 8. Physical-maintenance control protocol + +Physical work that does not change manifest-resolved logical graph state does not +create graph lineage merely to fit the main protocol. Examples include: + +- compacting `__manifest`, which is itself the authority and is read at its Lance HEAD; +- deleting versions/files already proven unreachable under the active retention + contract; +- reclaiming orphaned branch refs or uncommitted objects; +- rebuilding derived physical state when no graph-visible data-table pointer changes. + +Such work must still be idempotent, bounded, observable, and safe under concurrent +native Lance commits. It uses substrate conflict/retry semantics appropriate to the +operation. It must never expose partial logical graph state. + +The exception is from graph-lineage publication, not from recovery safety. Maintenance +must run the recovery barrier or refuse before it can replace or delete an artifact +named by an unresolved sidecar. It must also acquire any relevant process-local gates; +as elsewhere, those gates are an optimization rather than cross-process authority. + +Data-table compaction/index work that advances a version which graph reads must select +through `__manifest` is not exempt; Section 6.4 applies. Checkpoint reachability and +the mapping from graph checkpoint rows to Lance-native GC pins belong to RFC-025. + +## 9. Concurrency and retry semantics + +Process-local gates reduce same-process races and establish one deadlock-free order. +They do not coordinate two servers or CLIs. A conforming implementation remains safe +if every process has its own gate manager. + +Cross-process safety comes from: + +- complete read-set arbitration at the manifest authority; +- Lance transaction conflicts for physical table effects; +- durable recovery before physical HEAD movement; +- refusal to continue past unresolved overlapping recovery. + +Retry rules are phase-specific: + +- before effects, a read-set mismatch discards and recomputes the whole attempt; +- while applying an effect, the adapter may retry only when Lance guarantees the + operation can be safely replanned from fresh physical state; +- after any effect, manifest contention is resolved through the armed recovery intent, + not by silently rebasing the logical plan; +- retry exhaustion returns a typed, observable conflict. + +## 10. Rollout and compatibility + +This RFC authorizes a protocol refactor, not a manifest v5 format moment. RFC-023 +through RFC-027 own their respective format and public-surface changes. + +It also does not require a mutable-tip `GraphState` singleton. Three measured, +local latency fixes can land independently of the adapter conversion: + +1. make the graph's shared Lance `Session` a required parameter of every + manifest open/publisher path, so remote opens do not rebuild clients and cold + metadata state; +2. capture one immutable operation-local manifest/lineage view and pass it down + the call stack instead of reopening the same state repeatedly; and +3. remove the verified-redundant branch-idle refresh and the back-to-back second + `branch_delete_as` refresh once their existing coverage asserts unchanged + behavior. + +These are narrow access-shape fixes, not a second commit-input authority. They +must preserve snapshot pinning and still cross the recovery/read-set barriers +defined above. + +Implementation proceeds in this order: + +1. Introduce `PreparedWrite`, `ReadSet`, effect-adapter, and recovery-plan concepts + while preserving existing behavior. +2. Ship conservative branch-wide arbitration first. Mutation/load captures + `(Lance BranchIdentifier, exact optional graph_head, accepted schema identity)`; + every publisher retry compares that token instead of reparenting. Because every + supported graph-content and schema apply advances `graph_head:` before + schema promotion, the shared head row atomically arbitrates probed-but-untouched + same-branch dependencies. The native branch identifier detects delete/recreate + ABA under the documented single-writer-process branch-control boundary. RFC-024 + later narrows false contention with table heads; it is not a correctness + prerequisite for this coarse step. Existing live committed-state validation + probes remain until the narrowed read set replaces them. + + > **Implementation note (2026-07-11):** mutation/load now use this coarse + > token, schema-v3 exact-effect sidecars, fixed lineage/rollback outcome ids, + > zero transparent Lance commit retries, and bounded full reprepare before + > effects. Branch merge remains on its writer-specific multi-commit path, but + > now holds the root-shared schema plus source/target branch gates from its + > strict recovery barrier and authority capture through publication. It plans + > with an accepted-contract catalog captured under that schema gate, then + > acquires all catalog table gates for source and target, re-lists recovery, + > and compares fresh manifest versions before Phase A. This closes + > same-process delete/recreate ABA and legacy table-only-writer races while + > its full exact-effect adapter remains future work. Schema apply, + > optimize/index, and MemWAL fold remain on their writer-specific paths until + > their adapter slices land. +3. Convert mutation/load, branch merge, schema apply/migration, data-table optimize, + and graph-visible index work one adapter at a time. +4. Add static or runtime enumeration proving no graph-visible entry point bypasses the + coordinator. +5. Delete superseded writer-specific orchestration only after its crash and + concurrency cells pass through the adapter. +6. Optimize background recovery latency only after the synchronous barrier and all + recovery classifications remain intact. + +Mixed writer binaries are not made safe by process-local gates. A deployment may +enable the new protocol only when every writer that can reach the graph obeys the same +sidecar and manifest-arbitration contract, or when a compatibility gate rejects older +writers. + +## 11. Required tests and cost gates + +### 11.1 Protocol conformance + +- Enumerate every graph-visible entry point and its adapter. +- Assert no sidecar-backed adapter can create an independently durable physical + effect before its sidecar is durable. +- Enumerate authority-first workflows and assert their CAS is the first durable + effect and every post-CAS action is idempotently derivable from its authority row. +- Assert one graph-visible operation produces exactly one manifest visibility commit. +- Assert branch create/delete and physical-maintenance exceptions produce no synthetic + graph lineage. + +### 11.2 Read-set races + +- Two distinct ids racing on the same non-key `@unique` value cannot both publish. +- An edge insert racing deletion of an endpoint must revalidate or conflict. +- Cardinality probes of an untouched table participate in arbitration. +- A target advance after merge classification forces complete reclassification. +- Run the same cells with separate `Omnigraph` handles sharing one root-scoped + process-local gate manager, then with separate processes that do not share it. + +### 11.3 Recovery + +- Fail before sidecar, after sidecar, after each physical effect, after confirmation, + after manifest CAS, and during sidecar deletion for every adapter. +- A later overlapping writer blocks, heals, or returns recovery-required; it never + advances around the sidecar. +- A live foreign writer's sidecar is not destructively recovered without fencing. +- Recovery proves exact effect identity; a numerically newer unrelated version is not + accepted as proof of ancestry. + +### 11.4 Adapter-specific truth tables + +- Mutation/load: zero/one table, multi-table, strict, non-strict, and + probed-but-untouched dependency cases. +- Merge: adopt, rewrite, multi-commit, no-op, target advance, and partial-Phase-B crash. +- Schema: staging files, registration-only, metadata HEAD advance, partial multi-table + migration, and branch-local state. +- Optimize/index: zero, one, and several Lance commits, including monotonic publish and + retryable physical contention. +- MemWAL fold, when RFC-026 lands: merged-generation conflict and every fold crash + boundary. + +### 11.5 Cost gates + +The protocol must not move validation, classification, embedding computation, or +staged-file construction under process-local gates. Measure gate hold time separately +from Prepare. Correctness gates precede latency optimization: a cost regression can +delay rollout, but it cannot justify touched-table-only validation or asynchronous +recovery safety. + +## 12. Invariants and deny-list check + +This design reinforces the existing architecture: + +- Lance and `__manifest` remain the sources of truth; `PreparedWrite` is immutable + attempt-local state, not a shadow mutable tip. +- Graph visibility remains manifest-atomic. +- Recovery remains part of the commit protocol. +- Logical constraints fail loudly and are revalidated when their inputs change. +- Derived indexes and reclaim work converge without becoming logical commit points. +- No custom WAL, transaction manager, buffer pool, or distributed lock is introduced. + +The design rejects two tempting deny-list violations: treating process-local queues as +distributed correctness, and treating recovery as derivable background work that a +new writer may outrun. + +Acceptance should also add one clarification to invariant 15: a view of immutable, +version-pinned state may be cached, while an in-memory view of the mutable tip is only +a hint. Every use of mutable-tip state as write input must be re-arbitrated by the +commit authority. Durable heads under RFC-024 are one possible authoritative +representation; this protocol does not require or bless a warm parallel truth path. + +## 13. Drawbacks and rejected alternatives + +### 13.1 One generic staged-storage method + +Rejected. Lance does not expose staged forms for every operation, and existing writers +have materially different version movement and rollback safety. A generic method would +either lie about those differences or accumulate writer-kind conditionals. One +protocol plus explicit adapters has lower long-run liability. + +### 13.2 Asynchronous heal with optimistic continuation + +Rejected. Current recovery classification relies on later writers not advancing past +an unresolved sidecar. Scheduling a sweep does not establish that fact. Recovery may +be proactively asynchronous, but the next writer still crosses a synchronous barrier. + +### 13.3 Touched-table-only CAS + +Rejected. Non-key uniqueness, RI, cardinality, and merge classification read tables +the operation may not write. Ignoring those dependencies admits commits that were never +valid against one serial graph state. + +### 13.4 Treat every state change as a manifest commit + +Rejected. Native branch refs and physical maintenance have different substrate +visibility points. Manufacturing manifest commits would add coordination without +making the native mutation atomic with them. + +## 14. Reversibility + +The in-memory coordinator and adapter refactor is reversible. Recovery-sidecar schema +changes must be versioned and backward-compatible during rollout. This RFC alone does +not authorize a `__manifest` schema-stamp bump, a public wire change, or a new storage +substrate. + +The correctness contract is intentionally difficult to reverse: after writers rely on +complete read-set arbitration and recovery-before-write, weakening either would +reintroduce silent integrity or recovery races. Focused irreversible changes are +reviewed in RFC-023 through RFC-027. + +## 15. Follow-up RFC boundaries + +- **RFC-023** owns PK annotation, fenced merge routing, strict keyed Append behavior, + mixed fenced/unfenced rollout, conflict mapping, and both commit-order tests. +- **RFC-024** owns mutable table-head rows, explicit live/tombstoned head state, + current-state read shape, migration, and cost gates. +- **RFC-025** owns checkpoint rows, Lance-native physical pins, cleanup reachability, + pruned-through semantics, and checkpoint/cleanup crash ordering. +- **RFC-026** owns MemWAL enrollment, durable acknowledgements, fold/dead-letter + behavior, stream quiescence, fresh reads, public surfaces, and upgrade fencing. +- **RFC-027** owns candidate discovery from row lineage, deletion discovery, fallback + semantics, and evidence for any O(delta) merge claim. + +Those RFCs call this protocol when they produce a graph-visible write. None may weaken +the recovery barrier, omit read dependencies from `ReadSet`, or create a second graph +visibility point. diff --git a/docs/rfcs/rfc-023-key-conflict-fencing.md b/docs/rfcs/rfc-023-key-conflict-fencing.md new file mode 100644 index 00000000..c0c3e2ae --- /dev/null +++ b/docs/rfcs/rfc-023-key-conflict-fencing.md @@ -0,0 +1,480 @@ +--- +type: spec +title: "RFC-023 — Substrate-native key-conflict fencing" +description: Make concurrent keyed writes fail or retry through Lance's transaction conflict filters, forbid keyed Append, and activate the contract only behind an all-branch fleet/format barrier. +status: draft +tags: [eng, rfc, write-path, concurrency, lance, primary-key] +timestamp: 2026-07-10 +owner: +--- + +# RFC-023: Substrate-native key-conflict fencing + +**Status:** Draft / for team review +**Date:** 2026-07-10 +**Surveyed:** omnigraph 0.8.1 (`main`); Lance 9.0.0-beta.15 at git rev +`f24e42c1`; full Lance transaction, table-schema, read/write, branching, and +MemWAL specifications; pinned Rust conflict-resolver and merge-insert sources +**Relationship to RFC-022:** this RFC is the fencing decision split from the +earlier monolithic RFC-022 draft. [RFC-022](rfc-022-unified-write-path.md) +defines the shared write/recovery protocol; this RFC owns the substrate, +compatibility stamp, and rollout requirements for key conflicts. It may share a +release with [RFC-024](rfc-024-durable-table-heads.md), but neither RFC depends +on the other's storage decision. +**Audience:** engine, storage, migration, and release maintainers +**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md). +Findings marked **BLOCKER** must be dispositioned before acceptance. + +--- + +## 0. Decision summary + +OmniGraph will use Lance's unenforced-primary-key merge-insert filter as the +key-conflict primitive. It will not emulate the filter in the engine and will +not add a lock table or custom transaction manager. + +The guarantee is deliberately narrow and strong: + +> For a keyed table, two concurrent operations that may insert the same `id` +> MUST NOT both commit silently. They either serialize, retry from a new graph +> base, or fail loudly. + +That guarantee cannot be rolled out by annotating a few new tables in an +otherwise v4 graph. Lance's current mixed filtered/unfiltered conflict handling +is directional, and a bare `Append` can commit after a filtered update. The +contract therefore activates only after all supported writers and every table +state reachable from every graph branch have crossed a fencing-compatible +format barrier. + +Normative decisions: + +1. Node and edge table PK = `id`. +2. Bare Lance `Append` is forbidden for keyed tables. +3. Every keyed insert/upsert path produces the same Lance key filter. +4. Mixed-version serving is forbidden during activation; the fencing-compatible + format stamp is written last and older binaries refuse it. +5. PK metadata is permanent and preserved by every later schema/data rewrite. +6. Existing-table migration covers every graph branch, including lazy-inherited + table states, and is recoverable by rolling forward. +7. Fencing does not replace read-set OCC for `@unique`, RI, or cardinality. + +## 1. Problem + +OmniGraph validates duplicate `@key` values inside one delta, but today two +processes can both read a base where `id = K` is absent, stage disjoint Lance +fragments containing `K`, and let Lance rebase both commits. The graph manifest +CAS orders graph publication; it does not tell Lance that the two data-table +transactions inserted the same logical key. + +The result is worse than an ordinary write conflict: both callers can receive +success while a keyed table contains duplicate IDs. Every subsequent upsert, +edge lookup, uniqueness check, and merge then operates on a broken identity +relation. + +The desired primitive already exists in Lance. The missing work is to use it on +every keyed path and to define a rollout that never admits an unfenced writer. + +## 2. Substrate facts and the directional asymmetry + +This section is load-bearing. Tests pin every statement before implementation. + +### 2.1 Filter attachment + +On the pinned Lance revision, a v2 merge-insert whose ON field IDs exactly equal +the schema's unenforced PK field IDs attaches an `inserted_rows_filter` to its +`Operation::Update`. The filter contains keys for rows classified as inserts; +updates of existing rows continue to use Lance's affected-row / fragment +conflict machinery. + +The legacy indexed merge path does not attach this filter. Therefore a BTREE on +`id` can route an otherwise correct merge onto an unfenced path unless the +caller disables that path or Lance wires the filter into it. + +### 2.2 Conflict compatibility is directional + +Lance transaction compatibility is evaluated from the transaction currently +attempting to commit against transactions that committed after its read +version. It is not implicitly symmetric. + +At `f24e42c1`, `check_update_txn` behaves as follows: + +- current `Some(filter)` vs committed `Some(filter)` — compare field IDs and + filter intersection; overlap or incompatible filter configuration retries; +- current `Some(filter)` vs committed `None` — retry conservatively; +- current `None` vs committed `Some(filter)` — no corresponding conservative + arm; it may rebase; +- current bare `Append` vs committed filtered `Update` — `Append` treats the + `Update` as compatible. + +Consequently, "filtered vs unfiltered conflicts" is not a sufficient rollout +argument. Commit order matters. A filtered writer can win first and a stale +unfiltered writer or bare keyed append can still land second. + +### 2.3 What a PK annotation does not do + +The key is explicitly *unenforced*. Merely setting the metadata: + +- does not validate historical uniqueness; +- does not make bare appends unique; +- does not protect a merge whose ON set differs from the PK; +- does not repair an existing duplicate; +- does not replace OmniGraph's semantic validators. + +## 3. Scope and non-goals + +In scope: + +- node and edge data tables; +- mutation, load, branch merge, recovery replay, and WAL fold paths; +- new-table creation and all-branch existing-table activation; +- schema/overwrite preservation of PK metadata; +- typed retry behavior and coverage gates. + +Out of scope: + +- using a composite edge PK (`src`, `dst`); +- enforcing arbitrary `@unique` groups through the Lance PK; +- keyless streaming-table deduplication; +- a custom OmniGraph WAL, lock table, or transaction manager; +- declaring general multi-process writes supported before foreign-process + recovery-sidecar ownership is solved. + +## 4. Table classes and PK contract + +### 4.1 Keyed graph tables + +Every normal node and edge table has a non-null `id` field. Its Lance schema +MUST mark exactly that field as the unenforced PK. For edges, `src` and `dst` +remain ordinary fields governed by referential-integrity and cardinality +validation. An edge endpoint move is an update of the row identified by `id`. + +The PK field is addressed by stable field ID, not column position or mutable +name. Until rename-stable OmniGraph type/property identity is closed, the +fencing migration cannot claim rename safety. + +### 4.2 Keyless append-only tables + +A table explicitly declared append-only may omit a PK. Such a table may use +Lance `Append`, including MemWAL append-only operation. It receives no +same-logical-key guarantee because it has no logical key. + +Current node and edge types are not in this class: both have graph identity on +`id`. The class is reserved for an explicitly designed internal or future +non-graph table surface. + +The distinction is catalog-derived and first-class. Callers do not choose it +with an ad-hoc flag. + +## 5. Normative write routing + +| Logical operation | Keyed table | Keyless append-only table | +|---|---|---| +| Strict insert / load append | merge-insert ON exactly `id`, `WhenMatched::Fail`, filtered path | `Append` allowed | +| Upsert / load merge | merge-insert ON exactly `id`, `WhenMatched::UpdateAll`, filtered path | workload-specific | +| Fast-forward branch merge of new rows | filtered merge-insert with `WhenMatched::Fail`, even when every row was classified new | `Append` allowed | +| WAL upsert fold | filtered merge-insert with `merged_generations` | append transaction allowed | +| Update existing row | merge-insert/update with affected-row conflict metadata | workload-specific | +| Delete | staged Lance delete; PK filter is not the delete-conflict primitive | staged Lance delete | +| Overwrite | staged overwrite whose output schema preserves the exact PK | staged overwrite | + +### 5.1 No keyed `Append` + +The prohibition includes internal optimization paths. A caller may not infer +"all rows are new" and switch a keyed table to `stage_append`: that inference +was made against a snapshot and is exactly what a concurrent same-key writer can +invalidate. + +Routing through merge-insert does not collapse strict and upsert semantics. +Strict `load --mode append` and other insert-only surfaces use +`WhenMatched::Fail`; a row already present at the pinned base or discovered at +execution remains an error. Only declared upsert surfaces use `UpdateAll`. + +The storage trait and `forbidden_apis` guard MUST make a keyed append difficult +to express accidentally. The fast-forward merge optimization is retained only +for keyless append-only tables unless Lance ships a key-filtered append +transaction. + +This prohibition knowingly retires a measured fix. The fast-forward append +path exists because a whole-delta merge-insert join exhausted the query memory +pool on embedding-bearing tables (#277); routing adopted rows back through a +filtered merge-insert re-exposes that workload to join memory behavior. The +regression class is therefore a named ship gate: the fenced bulk adopt-merge +must pass the §11.4 memory/cost gate on embedding-bearing tables — via bounded +batched fenced merges inside one staged transaction, a pool-bounded execution +mode, or an upstream key-filtered append — before the keyed fast-forward path +is removed. Correctness wins the ordering, but the memory bound is not +optional. + +### 5.2 Routing choice + +There are two acceptable implementations: + +1. use the v2 merge path (`use_index(false)`) and pass its scale gate; or +2. consume a pinned Lance revision whose indexed path emits the identical + filter and passes the same surface guards. + +If the v2 hash-join cost scales unacceptably at the Phase-B workload, fencing +waits for option 2. Correctness is not traded for the old indexed-path speed. + +### 5.3 Symmetric mixed-transaction behavior + +Before activation, the pinned Lance revision MUST conservatively reject both +orders of: + +- filtered Update vs unfiltered Update; +- filtered Update vs bare Append. + +The preferred fix is upstream conflict-resolver symmetry. A workspace-only +fork is not an accepted permanent design. The fleet barrier remains necessary +even after that fix because two old, unfiltered writers still have no filters +to compare. + +## 6. Retry and validation semantics + +A Lance retryable key conflict restarts the entire logical attempt: + +1. gather a new graph snapshot and schema identity; +2. rerun all delta and committed-state validation; +3. restage from that base; +4. commit and publish through the normal recovery-covered pipeline. + +It is incorrect to retry only `commit_staged`: an insert may have become an +update, defaults or checks may now differ, and cross-table validation may have +changed. + +Upsert surfaces may perform the bounded semantic retry. Strict insert surfaces, +including `load --mode append`, do not change meaning under contention: both an +already-present match from `WhenMatched::Fail` and a concurrent same-key commit +normalize to typed `KeyConflict` / HTTP 409 for the whole strict operation. They +do not switch to `UpdateAll`; callers decide whether to resubmit. Other strict +read-modify-write surfaces retain their typed write conflict. Retry exhaustion +on a non-strict upsert remains a retryable 409. + +Fencing covers the PK insertion race only. `@unique` values on different IDs, +edge RI, and cardinality depend on a read set. Their correctness requires the +read-set-in-CAS design or equivalent revalidation before HEAD movement; this RFC +does not claim that fences close those races. + +## 7. Version and fleet barrier + +### 7.1 No partial activation in the old format + +OmniGraph MUST NOT annotate a new data table and advertise fencing while the +graph remains generally writable by older binaries. An older process can select +the legacy merge path or keyed append and bypass the guarantee. + +This RFC owns its activation boundary: + +- operators quiesce every server, CLI writer, and embedded writer for the + graph; +- one migration claimant holds an atomic create-if-absent claim with a random + owner/fencing token; a native Lance branch sentinel is not accepted as CAS; +- only the dedicated migration binary may open the old graph for writes; +- the fencing-compatible stamp is written after every branch/table verification; +- normal serving begins only after the stamp; older binaries then refuse. + +The migration claim uses the storage adapter's `PutMode::Create` contract, +records operation/owner token, and has no time-only takeover. Recovery under the +fleet outage must classify the migration ledger/sidecars before replacing a +stale token. + +The stamp is graph-wide and read from the reserved main manifest before any +named-branch open; selecting a named branch cannot bypass the compatibility +check. + +An in-process mutex is not a fleet barrier. A marker unknown to v4 binaries is +also not a fleet barrier. The operator procedure and cluster control plane must +keep old writers stopped until finalization. + +The exact format number is assigned when this RFC is accepted. If RFC-024 is +also accepted and ready, the two migrations may deliberately share its v5 +release after a combined failure-matrix review. If durable heads fail their +cost gate, fencing still proceeds with its own next compatible stamp; if +fencing is blocked upstream, durable heads need not wait. + +If durable heads are already active when fencing migrates, every PK metadata +version repoint also emits the identity-bearing journal event and matching +`table_head` transition in the same manifest CAS. If fencing lands first, its +format/stamp becomes an explicit predecessor that a later heads migration must +preserve. Acceptance covers heads-first, fencing-first, and co-release orders. + +### 7.2 New graphs + +A graph created directly at the fencing-compatible format creates every keyed +table with the PK metadata already present and enables only the write routing +in §5. There is no post-create annotation window. + +## 8. All-branch PK migration + +Migration operates on graph states, not merely table roots. The unit is every +reachable tuple: + +``` +(graph_branch, table_key, table_path, pinned_table_branch, pinned_table_version) +``` + +This matters for lazy branches: a graph branch may still point at an old main +table version whose schema predates the PK, even after main HEAD is annotated. + +Under the fleet barrier, the migration: + +1. enumerates and incarnation-pins every live graph branch; +2. folds each branch manifest and enumerates its live keyed tables; +3. validates that every pinned table image has non-null, unique `id` values; +4. acquires branch/table gates in RFC-022 order and freshly revalidates the + pinned tuple, schema identity, and migration claim; +5. writes a per-unit RFC-022 sidecar declaring expected branch/table state, the + optional native fork effect, PK metadata effect, and intended manifest delta + before either effect can persist; +6. for an owned table branch, commits a set-if-absent PK metadata update; +7. for a lazy-inherited state, forks an owned table branch from the *exact + pinned version*, applies the PK metadata there, and leaves row contents + unchanged; +8. records exact achieved fork identity and table version in the sidecar; +9. publishes that graph branch's manifest to the annotated physical version + with an audited migration marker but no graph-content commit or graph-head + movement, including a table-head transition when heads are active; +10. records a branch/table completion digest; +11. re-enumerates branches and verifies every live branch before writing the + fencing-compatible stamp. + +PK installation advances a Lance table version before the graph manifest can +publish it, and a lazy fork creates native ref state first. The sidecar covers +both gaps and lets recovery reclaim or adopt the exact fork rather than infer +from a branch name. Because Lance forbids clearing/changing a set PK, +migration is roll-forward-only: + +- an already-correct PK is an idempotent success and is not rewritten; +- an absent PK resumes installation; +- a different PK is a loud, operator-visible refusal; +- recovery never attempts to "undo" PK metadata. + +Branch create/delete, schema apply, and normal data writes remain blocked for +the whole enumeration/install/verify interval. The migration ledger makes a +crash resumable without treating partial annotation as a served graph. + +## 9. Preservation after activation + +Once set, the following are storage invariants: + +- a table overwrite carries the same PK field IDs and positions; +- schema apply cannot remove, replace, reorder semantically, or make nullable a + PK field; +- rename preserves the PK field ID and metadata; +- branch fork/clone preserves it; +- import/rebuild creates it before accepting data; +- recovery restore may select an older data image only if that image is already + fencing/PK-compatible; +- a table recreation uses a new table incarnation but installs the same + catalog-derived PK contract at creation; +- `__manifest`'s existing legacy PK key form is preserved exactly as stored; + the migration never rewrites or "normalizes" it. Lance forbids changing a + set PK, and the native-namespace decoupling documented in the Lance + alignment audit depends on that legacy form remaining in place. + +Every open-for-write path verifies the physical schema matches the catalog PK +contract. The check is against the pinned physical schema and is not a +maintained parallel registry. + +## 10. Recovery and multi-process scope + +All data writes retain the existing Phase A-D sidecar protocol. The key filter +does not close the table-HEAD-before-graph-manifest window. + +The fenced data-table transaction is cross-process safe in its failure-free +commit path. OmniGraph's current recovery sweep, however, serializes with live +writers only in-process; a foreign recovery process can still inspect a live +sidecar, and destructive `Restore` cannot be made convergence-idempotent. + +Therefore this RFC MUST use one of two honest dispositions: + +1. retain the documented single-writer-process support boundary and describe + fences as closing the silent key-race primitive only; or +2. land a cross-process sidecar claim/lease before advertising general + multi-process writes. + +Fences alone are not evidence for disposition 2. + +## 11. Tests and acceptance gates + +### 11.1 Lance surface guards + +- exact PK ON set + v2 path produces a non-empty filter for inserts; +- `WhenMatched::Fail` preserves that filter and reports an existing match + without writing; +- mismatched ON set produces no filter; +- legacy/indexed path behavior is pinned until replaced; +- filtered/filtered overlapping keys retry; +- filtered/filtered disjoint keys may rebase; +- filtered/unfiltered retries in **both** commit orders; +- filtered Update/bare Append retries in **both** commit orders; +- PK metadata cannot be changed or removed once installed. + +### 11.2 Engine concurrency tests + +- the same-key DST cell becomes a hard assertion with N concurrent writers; +- different keys remain concurrently writable; +- every keyed load/mutation/merge/fold path is observed to use the filtered + primitive; +- strict append of an existing `id` still fails and never updates the row; +- strict pre-existing-match and concurrent-insert cases normalize to the same + external `KeyConflict` while preserving `WhenMatched::Fail`; +- a source-walk guard rejects keyed `stage_append`, including the former + fast-forward path; +- a retry reruns validation rather than committing the stale staged batch. + +### 11.3 Migration and recovery tests + +- main plus owned and lazy-inherited graph branches all emerge PK-annotated; +- duplicate historical IDs abort before the fencing-compatible stamp; +- crash after each table annotation and before each manifest repoint resumes + without data change; +- crash before/after lazy fork and PK metadata commit recovers the exact + sidecar-recorded ref/version; +- branch enumeration is incarnation-safe; +- old binary/new graph and new binary/partially migrated graph refuse loudly; +- heads-first, fencing-first, and co-release upgrades preserve every active + format capability and produce identical logical rows; +- overwrite, schema apply, branch fork, restore, and import preserve PK metadata. + +### 11.4 Cost gate + +Measure a small upsert into 10K, 100K, and 1M-row indexed tables using the +shared cost harness. If `use_index(false)` makes work scale with table size +beyond the accepted budget, the indexed-path upstream work is a ship blocker. + +Additionally measure the bulk adopt-merge shape that motivated the keyed +fast-forward path (#277): a many-row, all-new-rows fenced merge into an +embedding-bearing table, asserting peak memory bounded by batch size rather +than table or delta width. If the fenced path cannot meet that bound, the +keyed fast-forward removal waits for the mitigation named in §5.1. + +> 💬 **Instrument required (tightening 5 in the +> [review ledger](../dev/rfc-022-027-architecture-review.md)):** +> `helpers::cost` measures I/O, not peak RSS, so this memory bound is +> unenforceable as written. Use the subprocess `scenarios.rs` harness or an +> equivalent `wait4`/`ru_maxrss` instrument, and name dataset sizes, baseline, +> cap, and pass threshold. + +## 12. Decisions and open gates + +### Decided + +- `id`, not `src`+`dst`, is the edge PK. +- No keyed append, including optimization-only append. +- No mixed-fleet or new-table-only v4 rollout. +- PK migration is all-branch, offline, idempotent, and roll-forward-only. +- A retryable upsert conflict retries the logical operation; strict insert maps + both existing and concurrent matches to `KeyConflict` without changing mode. +- Read-set validation remains a separate required concurrency design. + +### Open ship gates + +1. Upstream symmetric filtered/unfiltered and filtered/Append conflict behavior. +2. v2-path scale result versus indexed-path filter availability. +3. Operator repair procedure for pre-existing duplicate IDs. +4. Rename-stable field/type identity. +5. Cross-process recovery ownership before any broadened topology claim. +6. Final format-number/release sequencing and the all-branch fleet stamp. +7. The fenced bulk adopt-merge memory/cost gate on embedding-bearing tables + (the #277 regression class) — see §5.1 and §11.4. diff --git a/docs/rfcs/rfc-024-durable-table-heads.md b/docs/rfcs/rfc-024-durable-table-heads.md new file mode 100644 index 00000000..13625975 --- /dev/null +++ b/docs/rfcs/rfc-024-durable-table-heads.md @@ -0,0 +1,558 @@ +--- +type: spec +title: "RFC-024 — Durable table heads and the v5 manifest" +description: Materialize one live-or-tombstoned current-state row per table inside each manifest branch, prove bounded physical lookup, and migrate every branch atomically before stamping v5. +status: draft +tags: [eng, rfc, manifest, write-path, versioning, migration, lance] +timestamp: 2026-07-10 +owner: +--- + +# RFC-024: Durable table heads and the v5 manifest + +**Status:** Draft / for team review +**Date:** 2026-07-10 +**Surveyed:** omnigraph 0.8.1 (`main`); Lance 9.0.0-beta.15 at git rev +`f24e42c1`; full Lance table layout, transaction, branching, indexing, +compaction, cleanup, and object-store specifications +**Relationship to RFC-022:** this RFC is the durable-heads decision split from +the earlier monolithic RFC-022 draft. [RFC-022](rfc-022-unified-write-path.md) +defines the shared publisher/recovery protocol; this RFC owns the v5 format and +migration boundary. It deliberately excludes checkpoint retention, which +[RFC-025](rfc-025-checkpoint-retention.md) reviews separately. Key fencing in +[RFC-023](rfc-023-key-conflict-fencing.md) is also independently reviewable; +the two may share a release but do not block one another's evidence gates. +**Audience:** engine, manifest, migration, branch, and release maintainers +**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md). +Findings marked **BLOCKER** must be dispositioned before acceptance. + +--- + +Throughout this draft, **v5** means the next internal schema after today's v4. +If another independently accepted format change lands first, the release number +is reassigned; none of the head-row semantics depend on the numeral. + +## 0. Decision summary + +The current manifest is both an immutable graph journal and the place writers +ask "what is current?" Current-state resolution folds history, so its physical +cost grows with commit count even though the answer contains only one value per +table. + +v5 adds one mutable, durable `table_head` row per stable table identity inside +each native `__manifest` branch. The publisher updates the head in the same +Lance merge-insert transaction as immutable table-version rows, tombstone +events, graph lineage, and `graph_head`. The journal remains the history source; +heads become the current-state source. + +The format does **not** ship merely because the logical result has O(tables) +rows. A filtered scan over a history-sized Lance table is still O(history) +physical work. v5 is gated on a Lance-native indexed lookup whose measured I/O +is flat at history depth and whose uncovered tail is bounded and observable. + +Normative decisions: + +1. Heads live in `__manifest`; there is no second heads dataset and no warm + mutable-tip authority. +2. Head state is explicitly `live` or `tombstoned` and carries an incarnation. +3. Every publish and recovery outcome updates journal and head atomically. +4. Current-state reads use heads; history reads use the journal. +5. Missing or duplicate heads in v5 are corruption, not a reason to silently + return to the history fold. +6. Migration covers every live manifest branch, then stamps main v5 last. +7. If RFC-023 is co-released, the combined migration also verifies PK fencing; + durable heads do not depend on that decision. +8. Checkpoint/retention markers are deferred to a separate RFC. + +## 1. Problem + +`__manifest` currently stores immutable table-version and tombstone rows. To +resolve a branch tip, readers scan those rows, select the greatest version per +table, and apply tombstones. This makes a write's coordination cost depend on +the total graph history. Compaction reduces fragment count but cannot remove +the semantic journal rows, so even a compacted manifest retains a row-volume +slope. + +Caching that fold as mutable in-process state is the wrong authority shape for +writes: invalidation becomes a correctness condition across processes and +branch incarnations. Storing the folded answer durably in the same transaction +as the journal removes both liabilities: + +- no parallel authority can drift; and +- current-state work is proportional to catalog width, provided physical + lookup is also bounded. + +## 2. Scope and non-goals + +In scope: + +- v5 table-head schema and state transitions; +- atomic publisher, recovery, and current-read semantics; +- bounded physical access proof; +- all-branch predecessor→heads migration and compatibility refusal; +- optional co-release integration with RFC-023's all-branch PK activation. + +Out of scope: + +- a `GraphState` singleton or warm publish input; +- a separate `__heads` Lance dataset; +- deleting immutable journal rows; +- commit-graph ancestry acceleration; +- checkpoint retention and arbitrary-version GC; +- using a physical index as a correctness precondition. + +Checkpoint retention is excluded because a row in `__manifest` does not by +itself pin a version in another Lance dataset. Lance cleanup recognizes its own +versions, tags, and branch references, not foreign references. A later RFC must +define substrate-native per-table pins and their crash-safe lifecycle. + +## 3. v5 table-head schema + +Each native manifest branch contains exactly one current head row for each +stable table identity known to that branch. + +### 3.1 Identity and object key + +``` +object_id = "table_head:" +object_type = "table_head" +``` + +`stable_table_id` survives a type rename. It is not the display name and is not +derived anew from `kind:name`. Closing the current rename-stable identity gap is +a v5 ship gate, and **this RFC owns it**: the stable-ID encoding, the compiler +contract that mints and preserves it, and the incarnation baseline are defined +here. RFC-023 §4.1 (rename-safe PK claims) and RFC-025 §2.1 (checkpoint table +rows) consume this identity and must not ship identity-dependent claims before +it lands. No other RFC in the family defines a competing identity scheme. + +> 💬 **Superseded by review (BLOCKER-07 in the +> [review ledger](../dev/rfc-022-027-architecture-review.md)):** ownership +> *inside* this RFC contradicts siblings that consume the identity while +> calling this RFC optional (RFC-025) or without declaring the dependency +> (RFC-026). Disposition: extract identity/incarnation into its own +> prerequisite RFC, or accept this RFC in full before the identity-dependent +> siblings. Replace this ownership note accordingly. + +The head row is mutable under `WhenMatched::UpdateAll`, just like +`graph_head:`. There is one object ID per stable identity; recreation +does not leave multiple candidate heads. + +### 3.2 Payload + +v5 reuses the manifest's typed columns where they already fit and stores the +remaining versioned payload in a typed JSON structure: + +```text +TableHeadMetadata { + state: "live" | "tombstoned", + stable_table_id: String, + incarnation_id: ULID, + schema_hash: String, + head_graph_commit_id: Option, +} +``` + +The row columns have these meanings: + +| Column | `live` | `tombstoned` | +|---|---|---| +| `table_key` | current public table key/name | last public key/name | +| `location` | physical table location | last physical location, diagnostic only | +| `table_version` | current visible Lance version | last live Lance version | +| `table_branch` | physical owner branch, nullable for main | last owner branch | +| `row_count` | current row count | null | +| `metadata` | `TableHeadMetadata` | `TableHeadMetadata`; `head_graph_commit_id` names the tombstoning graph commit | + +The `state` field is authoritative. A tombstoned row MUST NOT become live merely +because an older journal version has a greater data-table version than some +other row. + +### 3.3 Incarnations + +`incarnation_id` distinguishes drop/recreate ABA: + +- rename preserves stable ID and incarnation; +- ordinary writes preserve both; +- physical owner handoff preserves both; +- dropping the type transitions the one head to `tombstoned`; +- recreating a logically new table mints a new incarnation and updates the same + stable-identity head to `live` only through an explicit schema operation. + +If recreation is assigned a new stable table identity by schema semantics, it +gets a new head object and the old identity remains tombstoned. The schema +planner, not a name comparison, chooses this outcome. + +### 3.4 Journal identity + +The mutable head is not the only place that records identity. Every v5 +table-version, registration, rename, and tombstone journal event carries +`stable_table_id` and `incarnation_id`; otherwise drop/recreate followed by a +new physical dataset whose Lance versions restart cannot be replayed +unambiguously. + +Migration writes one immutable v5 incarnation-baseline row per known identity, +bound to the source-tip digest. New registrations/recreations write an immutable +incarnation transition. Head repair starts from that baseline and replays only +identity-bearing v5 events. It does not infer identity from mutable names or +compare unrelated Lance version numbers. If pre-v5 history cannot be mapped to +one baseline identity deterministically, migration refuses before the stamp. + +## 4. State-transition rules + +| Event | Required head transition | +|---|---| +| Register table | absent → live, new incarnation | +| Data write / optimize / index publish | live version N → live version M | +| Owner-branch handoff | live owner A → live owner B, even if version is equal | +| Rename | live key/name A → live key/name B; identity/incarnation unchanged | +| Drop table | live → tombstoned in the same graph publish as the tombstone journal row | +| Recreate | tombstoned → live with the schema-planned identity/incarnation outcome | +| Recovery roll-forward | apply the failed writer's intended live/tombstone transition | +| Recovery rollback | publish a head matching the restored physical version and logical pre-write state | + +No path may append a table-version or tombstone journal row without including +its corresponding head mutation in the same publisher source batch. + +## 5. Atomic publisher contract + +The v5 publisher constructs one merge-insert source containing: + +- immutable, identity-bearing table-version rows; +- immutable, identity-bearing table-tombstone/transition rows; +- mutable table-head rows; +- when the RFC-022 plan carries `LineageIntent`, the immutable `graph_commit` + and mutable `graph_head:` rows; +- for metadata-only plans, their specific CAS authority/operation rows without + manufacturing graph lineage. + +One Lance manifest commit makes the entire set visible. The publisher still +resolves the graph parent and re-reads commit authority inside every CAS retry. +Expected table versions are compared against table heads, not reconstructed by +folding the journal. + +Two graph-content writers touching disjoint data tables still contend on +`graph_head:`, form one linear graph history, and re-parent on retry. +Writers touching the same table also contend on its one `table_head` object. +Metadata-only CASes contend on the stable authority rows named by their complete +`ReadSet`; they do not update `graph_head` merely to create contention. + +The immutable journal remains necessary for snapshots, diffs, audit, migration +verification, and head repair. Head rows do not replace or truncate it. + +## 6. Read contract + +### 6.1 Current state + +A v5 current-state read: + +1. derives the expected live stable table IDs from the pinned catalog and fixed + system-table registry; +2. issues a structured lookup for the exact head object IDs; +3. requires exactly one valid row per expected identity; +4. includes only rows whose authoritative state is `live`; +5. validates schema identity from the head payload; +6. returns one immutable `Snapshot` used for the operation's lifetime. + +Missing, duplicate, unknown-state, or schema-mismatched **live** heads fail +loudly. The hot path does not enumerate every identity ever dropped merely to +prove all tombstone heads exist; the branch completion digest plus explicit +`heads verify`/repair owns bounded tombstone-set validation. A missing or +duplicate tombstone is still corruption, but normal reads do not regain an +O(history) scan to discover it. + +### 6.2 History + +`snapshot_at_version`, commit resolution, change feeds, and audit continue to +read immutable journal/lineage state at the requested manifest version. A v5 +manifest version contains the heads as they stood at that version, but the +journal remains the normative explanation of how the state arose. + +### 6.3 Diagnostic repair + +An explicit offline repair loads the branch's v5 incarnation baseline, replays +identity-bearing journal transitions from that baseline, compares the result to +its heads/marker digest, and publishes corrected heads with an audited system +actor. Repair is not part of the read hot path and never silently runs from a +query. + +## 7. Bounded physical lookup is a ship gate + +Logical O(tables) output does not prove physical O(tables) work. Without an +index, `object_id IN (...)` still scans the journal-bearing manifest fragments; +compaction reduces files but not semantic row count. + +### 7.1 Required property + +At fixed catalog width, a reconciled v5 current-state lookup MUST have zero +positive slope in: + +- manifest object-store reads; +- bytes read; +- fragments/pages scanned; and +- rows decoded + +as commit history grows. + +The bound must hold on a real object store as well as local FS and must be shown +for compacted and uncompacted histories. The test uses the shared IO-tracking +harness and installs the tracker before the manifest handle opens. + +### 7.2 Candidate access shape + +The primary candidate is a structured exact lookup on `object_id` backed by a +Lance scalar index. It is acceptable only if measurement proves: + +- indexed head lookup avoids journal-fragment scans; +- newly committed head rows leave at most a bounded uncovered tail; +- reconciliation restores coverage without synchronous expensive work in the + logical write path; +- index absence or partial coverage remains logically correct and is surfaced + as an observable degraded-cost mode. + +The index is derived state. Queries MUST remain correct if it is missing, and a +missing index cannot block a logical write. The performance promise applies to +the reconciled serving state and includes an explicit bound on uncovered work; +it is not inferred merely from the existence of an index declaration. + +### 7.3 Rejected access shape + +A separate heads dataset is rejected. Lance commits are per dataset, so it +would reintroduce a journal→heads crash gap and require another sidecar protocol +for the very pointer whose purpose is to remove drift. + +If no in-manifest Lance-native access shape passes the gate, v5 does not ship +with head rows. The fallback is to retain v4 plus the local session/view-passing +improvements, not to waive the cost claim. + +## 8. Recovery protocol + +Data-table writers still use the existing four phases: + +1. write sidecar before a Lance HEAD advance; +2. commit staged/inline table work; +3. publish `__manifest`; +4. delete sidecar. + +The sidecar's logical intent in v5 includes the expected and desired table-head +payload. Recovery behavior is therefore complete: + +- roll-forward publishes the data version, journal row, table head, lineage, + and graph head together; +- rollback restores the physical version, then publishes journal/audit state + and a table head matching the restored logical state; +- a stale sidecar whose goal is already represented by the exact head + incarnation/version converges idempotently; +- a partially matching head is not treated as success. + +Recovery remains a synchronous barrier before any later writer advances a +touched table. Index reconciliation may be asynchronous; unresolved commit +protocol state may not. + +## 9. v5 boundary and compatibility + +v5 comprises: + +1. table-head rows and their publish/read semantics; +2. branch-local v5 completion markers; and +3. the graph-level internal-schema stamp written after verification. + +It does **not** comprise checkpoint/retention markers. + +If RFC-023 is independently accepted and ready for the same release, v5 may +also carry its PK annotation and fencing-compatible marker after a combined +migration review. That is release coordination, not a prerequisite of heads. + +Format sequencing is explicit: + +| Order | Result | +|---|---| +| Heads first | proposed v5 contains heads; later fencing migration preserves and atomically updates heads for every PK-version repoint | +| Fencing first | fencing takes the next format number; this heads migration takes the following number, accepts that exact predecessor, and preserves its PK/stamp invariant | +| Co-release | one format maps to both independently accepted capabilities and runs the combined failure matrix | + +Migration code dispatches from an exact supported predecessor feature set; it +never assumes that “pre-heads” means pristine v4 or drops a capability it does +not own. + +After upgrade, serving is strict-single-version: + +- v5 binaries refuse unstamped or partially migrated graphs in normal mode; +- older binaries refuse v5 with the existing upgrade message; +- every open reads the graph-wide main stamp before selecting a named branch; +- only the dedicated offline migration command may open the exact supported + predecessor for conversion; +- there is no mixed v4/v5 serving period. + +## 10. All-branch predecessor→heads migration + +### 10.1 Preconditions + +The operator stops every graph writer and acquires an atomic +create-if-absent migration claim (with exact owner/fencing token; not a Lance +branch sentinel). The barrier covers server, CLI, embedded, maintenance, +branch, and schema writes. Because predecessor binaries do not understand the +new in-graph migration marker, the offline fleet barrier remains mandatory +until the final stamp. + +The claim uses `PutMode::Create`, records the migration operation and owner +token, and permits no time-only takeover. Recovery classifies the durable +ledger/sidecars under the fleet outage before replacing a stale token. + +After acquiring the claim, migration completes RFC-022's recovery barrier +before pinning any branch source tip. + +If fencing is co-released, the migration first executes RFC-023's all-branch +table-PK plan. Otherwise head migration neither adds nor changes PK metadata. + +### 10.2 Per-branch conversion + +For each live native `__manifest` branch: + +1. capture branch name, ref incarnation, manifest version, and e_tag/timestamp; +2. run the predecessor journal+tombstone fold once at that pinned tip; +3. construct exactly one live or tombstoned head plus one immutable incarnation + baseline per stable table identity; +4. validate table schema hashes and, only in a combined release, RFC-023 PK state; +5. publish all heads/baselines, an audited migration record + (`actor = omnigraph:migration/v5`), and a branch-local completion marker in + one manifest CAS that revalidates the captured ref incarnation/source tip; + this physical representation migration does not create a graph-content + commit or move `graph_head`; +6. store in the marker the source tip, head count, identity-baseline digest, + head-set digest, and heads-format version; the marker is content-scoped and + deliberately does not embed the source branch incarnation; source tip is + provenance, while digest/format determine inherited-marker validity; +7. verify by reopening the produced branch version through the v5 head reader. + +The completion marker has a deterministic object ID within each manifest +branch. A retry that finds a matching source/digest is complete; a mismatching +marker is a loud migration conflict. + +### 10.3 Finalization + +After all branches report completion, migration re-enumerates native manifest +refs and verifies the same incarnation set. Branch create/delete is blocked, so +any difference indicates out-of-band modification and aborts finalization. + +Only then does it stamp main as internal schema v5. The stamp is the fleet +commit point: before it, no serving process may start; after it, only v5 code may +open the graph. + +New branches created after v5 fork a source manifest that already contains +complete heads and the content-scoped v5 marker. The inherited marker remains +valid for the identical snapshot; branch open separately binds that content to +the new native ref incarnation and validates the graph-wide stamp. No +post-create marker rewrite is required. + +## 11. Migration recovery + +The migration keeps a durable ledger outside the ordinary read path with one +record per branch and, in a combined release, per RFC-023 table conversion. It +records expected source incarnations, achieved physical versions, produced +manifest versions, and digests. + +Recovery is idempotent and roll-forward-only: + +- completed, digest-matching branch conversions are skipped; +- a table HEAD advance not yet represented in its graph branch is recovered by + the normal sidecar and then branch conversion resumes; +- an uncommitted head-row source leaves no visible state and is rebuilt; +- a committed branch marker is authoritative for that branch conversion; +- the main v5 stamp is never written while any ledger unit is incomplete; +- co-released PK metadata is never cleared to simulate rollback. + +If migration crashes, the graph remains offline. Restarting an old serving +binary is not a recovery procedure; the operator resumes the v5 migration. + +## 12. Tests and acceptance gates + +### 12.1 Head semantics + +- current state from heads is byte-equivalent to the predecessor fold across realistic + histories; +- live→tombstoned never resurrects an older live version; +- drop/recreate distinguishes incarnations; +- identity-bearing journal replay remains unambiguous when a recreated physical + dataset restarts Lance version numbering; +- rename preserves identity/incarnation and changes the public key only; +- owner-branch handoff at an equal table version updates the head; +- missing, duplicate, malformed, and schema-mismatched heads fail loudly. +- `heads verify` detects a missing/duplicate tombstone against the baseline and + completion digest without adding that enumeration to every current read. + +### 12.2 Publisher and recovery + +- concurrent disjoint writers produce one linear graph chain and correct heads; +- same-table writers contend on one head row; +- failpoints after every table commit but before manifest publish recover to + matching physical version, journal, table head, and graph head; +- rollback and roll-forward assertions include head payloads, not only table + versions; +- a stale sidecar converges exactly once with one audit record. + +### 12.3 All-branch migration + +- main, owned named branches, and lazy-inherited branches all convert; +- tombstoned types are represented on every relevant branch; +- crashes before/after every per-branch CAS resume from the ledger; +- branch ref deletion/recreation is caught by incarnation verification; +- final stamp is impossible while any table or branch marker is absent; +- each branch's graph head and content lineage are unchanged by head migration; +- old/new binary refusal matrix covers every supported predecessor capability + set, partial heads migration, and complete heads format; +- a post-v5 branch inherits and validates complete heads. +- the inherited completion marker is content-scoped while ref-incarnation + validation is performed separately. + +### 12.4 Cost gates + +At fixed table count and increasing commit depth, assert flat curves for +manifest reads, bytes, fragments/pages, and decoded rows: + +- local FS, compacted and uncompacted; +- S3/RustFS with real e_tags; +- warm repeated read; +- cold operation-local open with shared Session; +- one uncovered-head update before reconciliation; +- reconciled steady state. + +The test must fail if the lookup silently falls back to scanning journal +history in the claimed steady state. + +The decode term is part of the gate: parsing head rows — including the typed +JSON `TableHeadMetadata` payload — must be bounded by catalog width. A +per-read parse cost that grows with anything other than table count fails the +gate even when I/O is flat. + +### 12.5 Format guards + +- exact v5 head metadata schema and object IDs; +- one head row per stable identity; +- incarnation-baseline and identity-bearing journal event schemas; +- content-scoped branch completion marker schema/digest; +- RFC-023 PK metadata on node and edge tables when the release combines them; +- v5 publisher source always pairs a journal/tombstone event with a head row. + +## 13. Decisions and open gates + +### Decided + +- Heads and journal share one `__manifest` transaction. +- Current reads use heads; historical reads keep the journal. +- Heads represent `live | tombstoned` plus incarnation explicitly. +- A separate heads dataset and a mutable in-process tip authority are rejected. +- Migration is offline, all-branch, resumable, and stamps v5 last. +- RFC-023 PK activation is verified by v5 only when deliberately co-released. +- Checkpoint retention is deferred. + +### Open ship gates + +1. Rename-stable table/type identity and final stable-ID encoding — owned by + this RFC; consumed by RFC-023 and RFC-025 (§3.1). +2. The in-manifest indexed lookup implementation and bounded uncovered-tail + proof. +3. Passing local and object-store depth-slope cost gates. +4. Atomic cross-process migration-claim implementation and operator runbook. +5. Final v5 metadata JSON/typed-column compatibility review. +6. Full all-branch migration/failpoint matrix. diff --git a/docs/rfcs/rfc-025-checkpoint-retention.md b/docs/rfcs/rfc-025-checkpoint-retention.md new file mode 100644 index 00000000..1e891b68 --- /dev/null +++ b/docs/rfcs/rfc-025-checkpoint-retention.md @@ -0,0 +1,444 @@ +--- +type: spec +title: "RFC-025 — Checkpoint-pinned retention" +description: Makes named graph checkpoints authoritative retention roots, materializes them as Lance-native manifest and data-table tags, and defines crash-safe reconciliation and cleanup ordering on the RFC-022 unified write path. +status: draft +tags: [eng, rfc, retention, checkpoint, cleanup, manifest, lance, omnigraph] +timestamp: 2026-07-10 +owner: +--- + +# RFC-025 — Checkpoint-pinned retention + +**Status:** Draft / for team review +**Date:** 2026-07-10 +**Depends on:** [RFC-022](rfc-022-unified-write-path.md)'s publisher and +recovery-sidecar protocol. It composes with +[RFC-024](rfc-024-durable-table-heads.md), but is not part of v5 by implication. +**Surveyed:** omnigraph 0.8.1; Lance 9.0.0-beta.15 (`f24e42c1`) +**Audience:** engine, storage, CLI, and operations maintainers +**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md). +Findings marked **BLOCKER** must be dispositioned before acceptance. + +--- + +## 0. Decision + +OmniGraph adopts checkpoint-pinned retention. A checkpoint is a durable, named +graph snapshot: one reference set in the reserved main-manifest registry +containing every physical manifest/table lineage and version needed to +reconstruct that graph state. + +The checkpoint rows are the **logical authority**. Lance tags are the +**physical pins** that make that authority effective. This distinction is +load-bearing: Lance `cleanup_old_versions` protects Lance tags and branches; it +does not inspect OmniGraph rows in `__manifest`. A checkpoint row without the +corresponding tags would therefore promise time travel to versions that Lance +is free to delete. + +The safe ordering is asymmetric: + +- create physical tags first, publish checkpoint authority last; +- tombstone checkpoint authority first, reclaim physical tags last. + +Every crash window consequently over-retains. None under-retains. + +## 1. Scope and non-goals + +This RFC specifies: + +1. checkpoint and GC-boundary manifest rows plus their own format activation; +2. deterministic Lance tags for the pinned manifest and every pinned table version; +3. create, delete, and reconciliation protocols; +4. how `cleanup` computes and protects its root set; +5. CLI, policy, audit, observability, migration, and acceptance contracts. + +It does not change snapshot isolation, invent a second transaction log, or +replace Lance cleanup. It does not make every historical graph commit a +checkpoint. History outside branch heads, named checkpoints, and the explicit +retention window remains eligible for collection after operator confirmation. + +## 2. Sources of truth + +### 2.1 Logical authority: manifest rows + +A checkpoint is immutable after creation. Retargeting a name means deleting the +old checkpoint and creating a new checkpoint ID; a name never silently moves. +The stable name-reservation row carries `state = live | tombstoned` and a +monotonic `generation`. Reuse is a CAS transition from the exact tombstoned +generation to `live` with `generation + 1` and a fresh checkpoint ID. Thus a +deleted name may be deliberately reused, but two concurrent re-creations cannot +both win. + +All checkpoint authority rows live on the reserved **main** branch of +`__manifest`, regardless of the logical branch being checkpointed. This is a +global registry, not branch-local graph state: deleting the source branch must +not delete the authority that protects its snapshot. One main-manifest publish +writes: + +- `checkpoint_name:` — unique name reservation pointing at the + immutable checkpoint ID; +- `checkpoint:` — header containing `name`, source logical + branch, `graph_commit_id`, source physical manifest branch and version, + manifest schema stamp, `created_at`, and `actor`; +- `checkpoint_table:::` — one + row per pinned table containing stable table identity, table key, + `table_path`, `table_branch`, `table_version`, and physical incarnation. + +The name reservation, header, and every table row land in one RFC-022 publisher +CAS on main. First creation is insert-only; reuse requires the exact tombstoned +reservation generation in the `ReadSet`. Two concurrent attempts for the same +normalized name therefore conflict even when they captured different source +branches. A missing or duplicate table row is an invalid checkpoint, not a +partial checkpoint. + +Stable table identity/incarnation is a ship gate even when RFC-024 has not +landed; checkpoint rows cannot key retention to mutable names. A deployment may +reuse RFC-024's identity baseline, but durable heads themselves are not a +dependency. + +Deletion changes the name reservation from `live` to `tombstoned` and writes +manifest tombstones for the checkpoint header and all table rows in one CAS. +The reservation and every tombstone carry the same fresh +`delete_operation_id`; that CAS is the durable delete/recovery marker. The +retention claim may repeat the operation ID while the writer is live, but it is +only a fence, never recovery authority. Checkpoint IDs are never reused. + +Checkpoint create/delete are manifest metadata transactions. They do not create +a graph-content commit or move `graph_head`: taking a checkpoint must not change +the graph state it names. They still use RFC-022's read-set arbitration, +manifest CAS, actor, and audit contracts. Creation uses a recovery sidecar for +its pre-CAS tag effects; deletion is authority-first and embeds its idempotent +operation marker in the tombstone CAS instead of writing a generic sidecar. + +The named snapshot is the source branch version and graph commit pinned in step +2 of creation, not whatever is current when the later main-registry CAS lands. +That immutable version may remain a valid checkpoint if the source branch +advances after capture. The response always returns the exact captured commit +and manifest version; it never implies that the checkpoint tracks a moving tip. + +### 2.2 Physical protection: Lance tags + +Every checkpoint owns deterministic Lance tags of two kinds: + +```text +ogcp__manifest_ +ogcp__ +``` + +The manifest tag targets the exact `(manifest_branch, manifest_version)` named +by the header. Each table tag targets the exact `(table_branch, table_version)` +recorded by its table row. The spelling stays within Lance tag-name constraints +and includes enough physical-lineage identity to avoid collisions across lazy +forks. Pinning `__manifest` itself is mandatory: data-table tags alone cannot +preserve the schema, registrations, and lineage needed to reconstruct the graph +snapshot. + +Internal `ogcp_` tags are reserved. User-supplied tooling must not create, +move, or delete them. If an existing deterministic tag points anywhere else, +checkpoint creation and cleanup fail loudly with a typed corruption error. + +Tags are derived state. A live checkpoint row with a missing tag is repaired +from the row. An internal tag with neither a live checkpoint nor an in-flight +checkpoint sidecar is orphaned and may be removed. + +Tags protect versions from `cleanup_old_versions`; they do **not** protect a +named branch from Lance branch deletion, which removes `tree/`. The +initial contract therefore refuses physical deletion of any manifest or data +branch named by a live checkpoint row. Graph branch deletion and orphan-branch +reclamation read the main checkpoint registry under the retention claim and +return `CheckpointPinsBranch` with the blocking checkpoint IDs. They never call +`force_delete_branch` on a pinned lineage. Operators delete the checkpoints +first; hidden checkpoint branches or full snapshot copies require a separate +design. + +### 2.3 Cross-process retention claim + +Checkpoint create/delete, pin reconciliation, destructive cleanup, and graph +branch deletion serialize through one graph-wide, substrate-backed retention +claim. Acquisition uses the storage adapter's atomic create-if-absent primitive +(`PutMode::Create`) on `__leases/retention.json`; Lance branch creation is +explicitly rejected because its branch-metadata step is an existence check +followed by an unconditional put. + +The claim payload records operation ID, action, actor, creation time, and a +random fencing/owner token. Every protected phase re-reads and verifies that +token. Normal release verifies the exact token before deleting the claim; no +other owner can replace it while create-if-absent observes the object. There is +no time-based lease stealing. Crash takeover requires an explicit fleet write +outage, classification of the sidecar/manifest operation marker, removal of the +stale claim, and acquisition with a new token. A resumed stale process must +fail its next token check. + +A process-local mutex is only a contention optimization. The retention claim is +held across tag creation or physical cleanup, preventing this race: cleanup +computes roots, another process captures an untagged old version, then cleanup +deletes it before the tag lands. A crash leaves the claim and therefore +over-retains. Takeover first classifies the prior operation from its recovery +sidecar and/or exact manifest operation marker, then may release the claim +only after that operation is completed or safely aborted. It never steals on +elapsed wall time alone. + +The claim is not presented as a distributed reader/writer lock for arbitrary +graph writes. Initial destructive cleanup is an **offline** operation: operators +quiesce every server, CLI, embedded writer, and MemWAL stream before acquiring +the claim. New binaries also refuse to start a write while the claim exists, +but that check is defense in depth rather than proof that an already-running +foreign writer drained. Online cleanup requires a separate fleet writer-epoch +or substrate lease design and is out of scope. + +## 3. Checkpoint create protocol + +Checkpoint creation is a writer on the RFC-022 pipeline: + +1. Run and await RFC-022's synchronous recovery barrier. +2. Authorize, capture one immutable source-branch manifest snapshot and graph + commit, and prepare the complete header/table/tag set plus a `ReadSet` that + includes source branch incarnation, source manifest version, format/schema + identity, applicable GC boundaries, and the main-registry name reservation; + a source at or behind a pruned-through boundary is refused even if its files + happen to remain. +3. Acquire the retention claim, checkpoint/cleanup process-local serialization, + source branch-control gate, and adapter-declared metadata gates in RFC-022's + canonical order. Ordinary data queues are not held merely because immutable + versions are being tagged. +4. Freshly revalidate the complete `ReadSet`. A changed branch incarnation, + missing tag target, conflicting name, or format change releases the gates and + restarts before any tag exists. +5. Write a generic recovery sidecar containing the intended rows and tags. +6. Create the manifest tag and every table tag idempotently. Existing correct + tags are no-ops; conflicting tags fail. +7. Verify every tag, then release source branch control. A source-head advance + is harmless because the tags already pin the exact captured version; + source-branch deletion takes the same retention claim, classifies overlapping + create sidecars, and then refuses if the published checkpoint pins the ref. +8. Publish the name/header/table rows in one CAS on the main manifest registry. +9. Delete the recovery sidecar. Sidecar deletion remains best-effort after the + authority publish, as in the existing write protocol. +10. Release the retention claim after the outcome is durably classifiable. + +A crash before step 8 leaves tags but no checkpoint. Recovery may complete the +publish when every precondition still holds or remove the orphan tags. A crash +after step 8 leaves a valid checkpoint even if sidecar cleanup did not finish. + +## 4. Checkpoint delete protocol + +Deletion deliberately reverses the create order: + +1. run the recovery barrier, authorize, and prepare the complete checkpoint plus + name/format/registry `ReadSet`; +2. acquire the retention claim and local gates in canonical order, then freshly + revalidate the complete checkpoint; +3. publish the tombstoned name reservation plus header/table tombstones, all + carrying one fresh `delete_operation_id`, in one manifest CAS; +4. release the claim and acknowledge once the authority change is durable; +5. delete the deterministic Lance tags asynchronously and idempotently. + +A failed tag deletion only retains extra data. The checkpoint is already gone +from the user-visible namespace. The reconciler and the next `cleanup` retry +the physical reclaim. + +Delete is an RFC-022 authority-first metadata workflow: no independently durable +effect precedes the atomic tombstone CAS, and later tag deletion is derived, +over-retaining cleanup. It therefore needs no generic write sidecar. The +`delete_operation_id` persisted by the tombstone CAS lets crash recovery +distinguish pre-CAS from post-CAS state exactly; the retention claim does not. + +## 5. Pin reconciler + +Read-write open and every destructive `cleanup` run reconcile checkpoint pins +from the main-manifest checkpoint registry. +The reconciler: + +1. reads the live checkpoint rows once; +2. classifies checkpoint sidecars before touching apparently orphaned tags; +3. creates or verifies every required tag; +4. deletes internal tags proven to have no live or in-flight authority; +5. emits a typed result for repaired pins, reclaimed pins, and failures. + +`cleanup` is fail-closed: any missing, conflicting, unreadable, or ambiguous +pin aborts deletion for the affected graph. It never treats reconciliation +failure as permission to collect data. + +The reconciler runs under the retention claim and is serialized with +checkpoint creation and table maintenance. +It may run concurrently across independent graphs, but it must not delete a +tag belonging to a foreign process's live create sidecar. + +## 6. Cleanup protocol and pruned-through boundary + +The cleanup root set is: + +- every live graph branch head; +- every live checkpoint manifest and table row; +- every version inside the operator-selected time/count retention window; +- versions Lance itself protects through non-OmniGraph tags or branch refs. + +Read-only preview may run without a claim, but it is explicitly provisional. +Confirmed execution uses this order: + +1. establish the operator write outage, persistently seal/fold streams through + RFC-026 **before** taking the retention claim, and complete the RFC-022 + recovery barrier; +2. acquire the retention claim, then revalidate the fleet outage, every stream's + `SEALED` cut, and absence of recovery sidecars; +3. run pin reconciliation and recompute the exact root set and per-dataset + cutoffs under the claim; +4. prepare and revalidate a complete RFC-022 `ReadSet` containing format/schema + identity, branch incarnations, current manifest table entries/heads, prior GC + boundaries, and the root digest; +5. publish mutable `gc_boundary:` rows containing cutoff, + root digest, cleanup operation ID, and timestamp in one reserved-main + manifest CAS; +6. invoke Lance `cleanup_old_versions`, relying on the verified tags to retain + sparse checkpoint versions; +7. record per-table removal statistics/failures and release the claim only when + the operation is durably resumable or complete. + +Step 5 is an RFC-022 authority-first metadata transaction: it changes which +versions recovery may select, carries no graph-content lineage, and needs no +sidecar because no physical effect precedes its CAS. Step 6 is RFC-022 §8 +physical maintenance under the already-published boundary. The two are not one +indistinguishable “cleanup commit.” + +The boundary advances before delete. Any later recovery, restore, or publisher path +that could make an older physical version current must include the boundary in +its RFC-022 `ReadSet` and revalidate it before the physical effect. A position +at or behind the boundary is retried from current state or refused. Cleanup +never starts with an armed recovery that may still need an older version. + +Per-table cleanup remains fault-isolated, but partial operational success is +reported explicitly. A successful table does not hide another table's error. +The claim metadata plus `gc_boundary` operation ID is the cleanup recovery +marker: takeover resumes or reports the remaining table units under the same +root digest before releasing the claim; it does not silently recompute a +broader destructive plan after a partial run. + +### 6.1 Operational cadence — the cost of never running this + +Offline-only destructive cleanup has a predictable failure mode: operators +defer it indefinitely, and the graph silently reacquires the unbounded-history +cost class this program measured (per-version chains that grow one entry per +commit; latest-version listings whose page size grows with history; on hosted +deployments, seconds of added latency per operation). The docs and `cleanup` +preview therefore state the deferral cost explicitly, and deployments are +expected to schedule a periodic maintenance window — for continuously written +graphs, the same cadence discipline as `optimize` — rather than treating +cleanup as exceptional. Read-only preview and pin reconciliation remain online +so drift is visible between windows. + +Online destructive cleanup — concurrent with live writers under a fleet +writer-epoch or substrate lease — is the named successor design, deliberately +out of scope here (§2.3). This RFC's offline barrier is the correct first +delivery, not the end state; a deployment for which write outages are +unacceptable should treat the successor design as the gating requirement for +adopting aggressive retention policies. + +## 7. Public and policy surface + +Initial delivery is engine plus direct-storage CLI, matching `cleanup`: + +```text +omnigraph checkpoint create [--branch ] +omnigraph checkpoint list +omnigraph checkpoint show +omnigraph checkpoint delete +``` + +JSON output includes checkpoint ID, name, actor, branch, graph commit, creation +time, table count, and pin-health status. `cleanup` preview reports which branch, +checkpoint, or window protects each retained version and states that execution +requires a graph-wide write outage. + +Checkpoint creation and deletion use dedicated Cedar actions. Embedded callers +hit the same engine gate as the CLI. HTTP management endpoints are out of scope +until server-side maintenance has a general design; no transport-only bypass is +introduced here. + +## 8. Migration and compatibility + +This RFC owns its own internal-format activation. RFC-022 authorizes no format +bump, and RFC-024 explicitly excludes checkpoint rows from its heads format. +The retention format may receive the next schema version after heads, or the +two may share one release only if both RFCs are independently accepted and +RFC-024 is amended before implementation. A fresh activated graph starts with +no named checkpoints and a zero pruned-through boundary. No older commit is +silently promoted into a checkpoint. + +The upgrade mechanism must be chosen before implementation: + +- an in-place upgrade preserves branches and history and must define quiescence, + crash recovery, stamp ordering, and rollback; +- export/import creates a fresh graph at the new format but loses the old commit + DAG, branches, snapshots, and time-travel history. Those losses must be shown + in the plan and confirmation output; there is nothing left to backfill as a + checkpoint. + +Mixed-version writers are unsupported. An old binary must not write after the +retention format stamp or tag protocol becomes authoritative. + +## 9. Observability and bounds + +Expose: + +- live checkpoint count and pinned table-version count; +- missing, conflicting, repaired, and orphan-tag counts; +- oldest checkpoint age and retained bytes when Lance reports them; +- current GC boundary and root digest per dataset lineage; +- cleanup versions/bytes removed and per-table failures; +- checkpoint create/delete/reconcile latency and retry counts. + +Checkpoint enumeration and cleanup planning must be bounded by live checkpoints +times catalog width, not total commit history. Operators can limit checkpoint +count by policy; the engine refuses names or payloads beyond configured bounds +before creating tags. + +That is a physical-I/O claim, not just a logical row-count claim. Registry +lookup must use a structured Lance access path with a measured bound on +uncovered fragments, reusing RFC-024's in-manifest scalar-index work when it is +available or proving an equivalent access shape here. A filtered scan that +still reads history-sized manifest fragments does not pass this RFC's cost gate; +a separate checkpoint dataset is rejected because its authority rows could not +share the main-manifest CAS. + +## 10. Acceptance gates + +- A checkpoint on an old sparse version survives cleanup while adjacent + unpinned versions are removed. +- Branch delete and orphan reclamation refuse a non-main manifest/data lineage + while a live checkpoint references it; after checkpoint deletion and tag + reconciliation, branch deletion succeeds. `force_delete_branch` is never used + to bypass the pin. +- The source `__manifest` version survives cleanup; data-table tags without the + manifest tag fail the checkpoint-validity test. +- Concurrent creation of the same normalized name yields exactly one authority + record; the losing attempt leaves only safely reclaimable tags. +- After deletion, concurrent attempts to reuse the tombstoned name generation + yield exactly one fresh checkpoint ID; the old checkpoint ID never revives. +- Create failpoints cover sidecar, tag, and manifest boundaries; delete + failpoints cover authority CAS, claim release, and tag reclaim. Every outcome + converges to valid pins or safe over-retention. +- Missing live tags are repaired before cleanup; conflicting tags block cleanup. +- The retention claim blocks checkpoint create/delete/reconcile and branch delete + during cleanup on local FS and S3; destructive cleanup refuses active streams + or recovery sidecars. +- Two-process races prove the atomic retention claim, not a process-local gate, + closes checkpoint-vs-branch-delete windows; crash takeover never relies on + wall-clock expiry alone. +- Local and S3 claim tests prove `PutMode::Create` admits exactly one owner and + that mismatched owner tokens cannot release another operation's claim. +- A documented fleet write-outage test runs writers before and after cleanup; + online writer-vs-cleanup concurrency is explicitly not advertised. +- Genuine cross-version tests cover the selected upgrade path. +- Cost tests use `helpers::cost` at realistic commit depth and prove that a + steady checkpoint list/lookup and cleanup plan do not grow with commit count + once physical layout is held constant, including a bounded uncovered tail. + +## 11. Phasing + +| Phase | Content | Gate | +|---|---|---| +| A | row schemas, engine DTOs, format activation, deterministic tag encoding | schema and tag surface guards | +| B | create sidecar, delete-operation fields in the authority CAS, and reconciler | crash matrix; sparse-pin correctness | +| C | offline cleanup integration and GC boundary enforcement | quiescence/refusal tests; cost budgets | +| D | CLI, policy, audit, docs, and selected migration path | CLI outputs; genuine upgrade test | diff --git a/docs/rfcs/rfc-026-memwal-streaming-ingest.md b/docs/rfcs/rfc-026-memwal-streaming-ingest.md new file mode 100644 index 00000000..24761fc8 --- /dev/null +++ b/docs/rfcs/rfc-026-memwal-streaming-ingest.md @@ -0,0 +1,415 @@ +--- +type: spec +title: "RFC-026 — MemWAL streaming ingest" +description: Adopts Lance MemWAL as OmniGraph's strategic streaming-write architecture, with durable per-row acknowledgement, graph-atomic folds, epoch-fenced quiescence, and explicit fresh-read cuts on the RFC-022 unified write path. +status: draft +tags: [eng, rfc, streaming, ingest, wal, memwal, lance, omnigraph] +timestamp: 2026-07-10 +owner: +--- + +# RFC-026 — MemWAL streaming ingest + +**Status:** Draft / for team review +**Date:** 2026-07-10 +**Depends on:** [RFC-022](rfc-022-unified-write-path.md)'s unified write and +generic recovery-sidecar protocol, plus +[RFC-023](rfc-023-key-conflict-fencing.md) for the initial keyed graph-stream +mode. Durable heads from +[RFC-024](rfc-024-durable-table-heads.md) are compatible but not required. +**Surveyed:** omnigraph 0.8.1; Lance 9.0.0-beta.15 (`f24e42c1`); complete MemWAL format specification +**Audience:** engine, server, CLI, policy, and operations maintainers +**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md). +Findings marked **BLOCKER** must be dispositioned before acceptance. + +--- + +## 0. Decision and risk posture + +OmniGraph adopts Lance MemWAL as its strategic streaming-write architecture. +MemWAL is a major Lance architectural bet: a sharded LSM write path with durable +WAL entries, flushed Lance generations, merge progress committed with base-table +data, maintained indexes, and epoch-fenced writers. OmniGraph consumes that +architecture rather than building a WAL, shard protocol, or LSM reader. + +This RFC does **not** characterize the architecture as experimental. The risk is +narrower: Rust API names, some format details, and operational helpers are still +maturing across Lance releases. We manage that API/format-maturity risk with a +small adapter, compile/runtime surface guards, a quiescence requirement before +Lance upgrades, and a fresh full-spec alignment audit on every bump. It is not a +reason to fork or reimplement MemWAL. + +The contract is: + +- stream acknowledgement means the row's WAL entry is durable; +- acknowledgement does not mean graph visibility; +- default queries see only the manifest-committed graph; +- a fold is an ordinary RFC-022 graph writer and is the sole visibility point; +- fresh reads are explicit and never claim cross-table atomicity. + +## 1. Scope and non-goals + +This RFC specifies enrollment, the public stream API, acknowledgement semantics, +folding, fold-time integrity, dead-letter atomicity, branch/schema quiescence, +fresh-read cuts, resource bounds, observability, testing, and upgrade posture. + +It does not replace `load` or `mutate`, provide cross-query transactions, store +manifest mutations in MemWAL, create a second metadata authority, or weaken +default snapshot isolation. Stream-mode +deletes remain out of the first delivery and require the Lance tombstone surface +plus a separate acceptance pass. + +## 2. Stream mode and key semantics + +Initial delivery exposes +`@stream(mode="upsert", on_reject="strict")` on a node or edge type; +`on_reject` accepts `strict` or `dead_letter`. +It requires the table's immutable unenforced primary key to equal OmniGraph's +merge key: `id` for nodes and edges. All occurrences of one key map to one shard +and MemWAL applies last-write-wins ordering. + +Public append mode is deliberately out of scope. Nodes and edges always have +logical identity; allowing a retry to append the same `id` twice would violate +that contract. A future explicitly keyless, non-graph append-only table class +may consume MemWAL append semantics under its own schema/API decision. + +Stream ordering intentionally differs from the interactive fence: + +- concurrent interactive same-key writes serialize or fail/retry loudly; +- same-key stream entries resolve by MemWAL generation/position order; +- duplicate keys inside one bulk-load input retain the existing load error. + +The schema and user docs state all three together. + +## 3. Enrollment is a recoverable inline commit + +Schema apply records `@stream` intent only. First stream use enrolls the physical +table by creating the singleton `__lance_mem_wal` system index and its sharding +configuration. + +Enrollment advances Lance HEAD inline. It therefore uses the RFC-022 generic +recovery protocol, not an ad-hoc state machine: + +1. run and await RFC-022's synchronous recovery barrier; +2. authorize, pin the manifest/schema/table state, and prepare a complete + `ReadSet` containing schema identity, table entry/head, PK metadata, stream + intent/configuration, and lifecycle-row absence; +3. acquire any global claims and then the `(table, branch)` write queue in + RFC-022 order, then freshly revalidate the complete `ReadSet`; a mismatch restarts + before any inline effect; +4. verify RFC-023's already-installed PK; enrollment never performs a first-use + PK migration; validate the sharding configuration; +5. write a generic sidecar with writer kind `stream_enrollment` and pre-commit + table pin; +6. create the MemWAL index, advancing Lance HEAD; +7. publish the new table version plus `stream_state = OPEN` in one manifest CAS, + including the table-head row when RFC-024 is active; +8. delete the sidecar best-effort after publication. + +Recovery rolls the enrollment forward or back under the same classification and +audit machinery as other inline residuals. Repeating enrollment with identical +metadata is a no-op. A different PK, sharding spec, maintained-index set, or +writer-default configuration is a typed conflict. + +No row is acknowledged until enrollment is manifest-committed. + +Initial delivery supports one unsharded shard per `(table, main)`. Non-main +branches remain refused until the Lance branch-scoping question is proven by a +surface guard and end-to-end test. Later `bucket(id, N)` sharding must preserve +the one-key-to-one-shard rule. + +## 4. New public API + +The shipped `POST /graphs/{id}/ingest` path remains the deprecated, compatible +alias of `/load`. Streaming receives a new, non-conflicting surface: + +```text +POST /graphs/{graph_id}/streams/{type_name}/ingest?branch=main +GET /graphs/{graph_id}/streams +GET /graphs/{graph_id}/streams/{type_name} +POST /graphs/{graph_id}/streams/{type_name}/fold +POST /graphs/{graph_id}/streams/{type_name}/quiesce +POST /graphs/{graph_id}/streams/{type_name}/resume +``` + +The ingest request and response use `Content-Type: application/x-ndjson` and +`Accept: application/x-ndjson`. + +Each input line is one row payload. Each output line corresponds to the same +input ordinal: + +```json +{"ordinal":17,"status":"durable","shard_id":"...","writer_epoch":8,"wal_position":42} +``` + +Synchronous validation failures return a per-row error before a WAL append. +Previously acknowledged rows in the same request remain durable; the response +is a stream, not an all-request transaction. Ordering, cancellation, and retry +rules are explicit: + +- acknowledgements are emitted in input order for one HTTP stream; +- disconnecting does not cancel entries whose durability waiter resolved; +- a missing response is ambiguous; retrying the same `id` and payload may add + another WAL entry but produces the same last-write-wins graph state; +- server shutdown stops admission, drains durability waiters up to a bound, and + reports any unacknowledged tail as unknown to the client. + +CLI commands mirror the new namespace rather than overloading deprecated +`omnigraph ingest`: + +```text +omnigraph stream ingest --data ... +omnigraph stream status [] ... +omnigraph stream fold [] ... +omnigraph stream quiesce [] ... +omnigraph stream resume [] ... +``` + +Every endpoint has a dedicated OpenAPI operation and handler tests. Ingest +passes the engine `stream_ingest` Cedar action and per-actor admission +accounting before acquiring a shard writer; fold/quiesce/resume use a separate +`stream_manage` action. Status is authorized like other graph operational +metadata. The same engine gates apply to embedded and remote CLI use. + +## 5. Ack-path validation and writer lifecycle + +Before append, OmniGraph applies checks that need no base-table read: Arrow +shape/type, required/default fields, enum/range/check constraints, reserved +columns, and stream mode. RI, cardinality, cross-version uniqueness, and +external embedding computation remain fold-time work. + +One warm `ShardWriter` is held per active shard behind a bounded registry. The +registry has idle eviction and hard limits for resident writers, MemTable bytes, +unflushed WAL bytes, pending generations, and per-actor inflight bytes. Exceeding +a bound backpressures with a typed retryable response; it never drops a row. + +Initial topology has one active ingest owner for each `(graph, table, main)` +shard. MemWAL's epoch fence makes restart/failover safe; it is not a load +balancer. A deployment with multiple server replicas must route a shard to its +current owner (or return a typed retry/redirect) instead of letting replicas +reclaim the epoch per request. General multi-owner routing waits for the +multi-shard phase and its ownership protocol. + +## 6. Fold protocol + +The fold consumes flushed generations in ascending order. Embeddings and +base-dependent validation run outside the table queue and register every probed +table plus stream configuration/generation in RFC-022's `ReadSet`. The commit +phase then: + +1. stages accepted rows with Lance merge-insert and includes + `merged_generations` in that transaction; +2. stages any rejection/audit rows required by §7; +3. acquires every affected queue in canonical sorted order and revalidates the + complete `ReadSet`; any mismatch discards and replans the whole fold; +4. writes one generic RFC-022 recovery sidecar before the first + `commit_staged` call; +5. commits every staged Lance transaction; +6. publishes all data/internal table versions and lineage in one `__manifest` + CAS, including table heads when RFC-024 is active; +7. deletes the sidecar after successful publication. + +The sidecar is mandatory even though merge-insert is staged. After +`commit_staged`, Lance HEAD and `merged_generations` have moved while the graph +manifest has not. A failure in that window is the ordinary multi-table recovery +gap, not invisible staged state. + +MemWAL generation GC starts only after the exact fold is graph-visible, its +sidecar is resolved, index catchup permits reclamation, and no `FreshReadCut` +retention guard references the generation. Data-HEAD merge progress alone is +never permission to delete the only fresh-tier copy. + +Concurrent folders reload `merged_generations`: a generation already committed +is skipped; otherwise the fold is replanned from current state. A forced fold +stops new flush creation for its cut, waits for in-flight durability waiters, +and never aborts an acknowledged row. + +Fold lineage uses `omnigraph:ingest` as the mechanism actor and persists the +authenticated contributor actor for every folded WAL range in the same internal +audit participant. Commit and status output can therefore answer both who ran +the fold and who supplied the data after WAL GC. + +## 7. Fold-time rejection is atomic + +`strict` is the default. A permanent RI, cardinality, uniqueness, or embedding +failure stops the fold at the offending generation, marks the shard blocked, +and backpressures new ingestion once configured lag bounds are reached. + +`dead_letter` is explicit. `_ingest_rejects` is then a versioned internal Lance +table and a participant in the same fold pipeline, not best-effort state outside +the commit protocol. Every reject has deterministic identity: + +```text +(table_key, shard_id, generation, wal_position) +``` + +The fold stages reject rows, accepted rows, and merge progress before committing +any of them. The generic sidecar covers every participant; the single manifest +publish records both the base-table and reject-table versions. Replay is +idempotent by reject identity. There is no ordering in which progress can become +visible while the corresponding rejection is lost. + +`stream status` reports blocked generations and typed reject details. Reject +retention is explicit and cannot be shorter than the WAL/fold audit retention +needed to explain a durable acknowledgement. + +## 8. Epoch-fenced quiescence barrier + +Branch operations, schema changes, stream teardown, and Lance upgrades require a +real barrier, not an empty check. + +Each enrolled table has a durable +`stream_state::` row in its manifest branch with +`OPEN | DRAINING | SEALED`, configuration hash, and epoch floor. The row is the +logical lifecycle authority and is updated by an RFC-022 CAS; MemWAL shard +epochs are the physical writer fence. Neither an in-memory registry nor an +empty-generation observation can substitute for both. +Lifecycle-only transitions are audited manifest metadata transactions; they do +not create graph-content commits or move `graph_head`. + +The shared drain sequence is: + +1. publish stream intent `OPEN -> DRAINING` for the target table/branch; +2. increment and persist each shard writer epoch and seal claims, fencing stale + writers across processes; +3. reject or backpressure new appends; +4. wait for in-flight durability waiters, flush active MemTables, and fold every + generation to empty; +5. verify shard manifests and base `merged_generations` agree on emptiness; +6. publish `DRAINING -> SEALED` with the verified generation/epoch cut; +7. for an operation-scoped drain, perform the guarded operation; persistent + public quiesce stops after step 6. + +Each lifecycle CAS is an RFC-022 authority-first metadata write; each fold is a +separate normal RFC-022 graph write. `DRAINING` fully encodes the target epoch +floor, so an interrupted epoch mutation is idempotently resumed from that row. +The sequence is not one giant sidecar spanning multiple commits. + +There are two dispositions after the drain reaches `SEALED`: + +- **operation-scoped drain** — branch/schema maintenance automatically publishes + `SEALED -> OPEN` with a newer epoch only after the guarded operation succeeds + and the stream contract remains compatible; +- **persistent quiesce** — the public `quiesce` command leaves the stream + `SEALED`. It never auto-reopens. `stream resume` explicitly revalidates schema, + PK, configuration, MemWAL format, and epoch, then publishes a newer `OPEN` + state. Stream teardown deletes intent only from `SEALED`. + +The barrier never holds the table write queue while waiting for a fold that +needs that queue. State transition and epoch fencing happen first; fold commit +then acquires the normal queue. Crash recovery resumes from the durable state +and epoch. + +Schema apply must drain every affected enrolled type before changing fields, +constraints, PK, embeddings, or `@stream` and resumes only when compatible. + +A Lance version upgrade requires persistent `stream quiesce --all`, but empty +generations alone are insufficient: the MemWAL system index, shard manifests, +epoch records, and generation directories may still use the old format. Before +the bump, the implementation must prove one of: (a) upstream guarantees and +cross-version tests cover every retained MemWAL artifact, (b) a public Lance +metadata migration converts them, or (c) OmniGraph tears down the enrolled +MemWAL metadata under recovery and re-enrolls after the bump. Without one of +those gates the upgrade refuses; `resume` never opens unverified old metadata. + +## 9. Fresh-read cuts + +Freshness is a first-class engine/IR enum: + +```text +Committed +Fresh +``` + +At query planning, `Fresh` captures one `FreshReadCut` containing: + +- the ordinary manifest snapshot; +- each selected shard-manifest version and writer epoch; +- included flushed-generation paths and maximum generation; +- the active same-process MemTable row-position watermark, when available; +- the base table's `merged_generations` and index-catchup state read from the + exact table version selected by the manifest snapshot, never from live HEAD. + +Capture uses a retrying handshake: + +1. read the selected shard manifests/epochs, acquire Lance generation retention + guards for the flushed files in the tentative cut, and under one + same-process writer snapshot capture/pin any active-MemTable watermark; +2. pin the graph manifest snapshot and read `merged_generations` from each exact + base-table version it selects; +3. re-read the shard manifest versions/epochs; any epoch/configuration change + restarts the whole capture; +4. if a generation from step 1 disappeared, accept that only when the pinned + base's `merged_generations` proves it is included; otherwise release guards, + discard the whole graph snapshot, and retry from step 1; +5. exclude generations that appeared after step 1 and hold the generation and + MemTable read guards captured in step 1 until query completion. + +If Lance exposes no guard that prevents generation GC for the query lifetime, +cross-process `Fresh` does not ship. A missing generation is never interpreted +as “probably folded” against an older pinned base. + +Execution never refreshes that cut mid-query. It excludes every flushed +generation `<= merged_generations[shard]`; otherwise old WAL data could outrank +or duplicate its newer base-table image. + +Fresh reads have no cross-table atomicity. Same-process active MemTables provide +read-your-writes; other processes can promise only the latest flushed state +captured by their shard-manifest reads. The HTTP request and query docs state +those limits wherever the tier is exposed. + +## 10. Observability and resource contracts + +Per shard expose durable WAL position, replay position, active epoch, current +generation, flushed and merged generation, index catchup, pending rows/bytes, +oldest acknowledged age, last fold error, blocked reject, and quiescence state. + +Metrics cover ack latency, durability-wait batching, fenced writers, replayed +entries, fold rows/bytes/generations, fold retries, lag, reject counts, and +sidecar recovery. Defaults for every byte/count/time bound are documented and +configuration changes are observable behavior. + +`stream status` resolves the exact lifecycle rows and MemWAL metadata through a +structured, bounded access path; it may reuse RFC-024's scalar-index machinery +but cannot claim history-flat cost while scanning manifest history. + +## 11. Acceptance gates + +- Surface guards pin claim, append, durability waiter, flush, epoch fencing, + staged merge with `merged_generations`, index catchup, and seal/reopen APIs. +- A WAL append failure emits no durable acknowledgement. Every acknowledged row + survives crash, replay, fold, and recovery. +- Failpoints cover enrollment's inline-commit gap and every fold participant + around sidecar, `commit_staged`, reject persistence, and manifest publish. +- Enrollment restarts without an inline effect when schema, PK, table head, or + stream configuration changes between prepare and gated revalidation. +- Two folders converge exactly once; a fenced stale writer can never produce a + false durable acknowledgement after WAL GC. +- Two server replicas do not epoch-ping-pong one shard; owner failover fences + the old process before the new owner acknowledges. +- Quiescence tests race appends with branch and schema operations across two + coordinators and prove no post-drain tail appears; failpoints cover every + lifecycle-row, epoch-fence, fold, and `SEALED` boundary. +- Persistent quiesce never auto-reopens; explicit resume validates a newer + epoch. Upgrade tests cover every retained MemWAL artifact through declared + compatibility, migration, or teardown/re-enrollment. +- Fresh reads race capture with fold/GC, retry on an unexplained disappearing + generation, hold generation/MemTable guards through execution, exclude merged + generations, and document cross-table inconsistency explicitly. +- Server/OpenAPI tests preserve the old `/ingest` alias and cover the new route; + CLI parity covers embedded and remote stream commands. +- Ack-path object-store operations are O(1) and flat in graph history and WAL + depth. Fold cost is bounded by generations/rows folded, not graph history. +- S3 correctness runs against RustFS; API/format guards are rerun before every + Lance bump. + +## 12. Phasing + +| Phase | Content | Gate | +|---|---|---| +| A | MemWAL adapter, surface guards, enrollment sidecar | inline-commit crash matrix | +| B | new ingest route/CLI, durable ack, strict fold | ack durability; API compatibility; cost budget | +| C | atomic dead letter, audit provenance, status/bounds | reject crash matrix; backpressure tests | +| D | epoch-fenced drain, persistent quiesce/resume, schema/branch/upgrade integration | two-coordinator race and format-transition suite | +| E | fresh cuts and maintained-index reads; cross-process `Fresh` ships only if the substrate generation-retention guard exists (§9), otherwise same-process only | cut consistency; merged-generation exclusion | +| F | multi-shard upsert and stream deletes | one-key-one-shard proof; Lance re-audit | diff --git a/docs/rfcs/rfc-027-lineage-merge-deltas.md b/docs/rfcs/rfc-027-lineage-merge-deltas.md new file mode 100644 index 00000000..a6dbfab0 --- /dev/null +++ b/docs/rfcs/rfc-027-lineage-merge-deltas.md @@ -0,0 +1,254 @@ +--- +type: spec +title: "RFC-027 — Lineage-based merge deltas" +description: Research specification for replacing full-width branch-merge classification with Lance row-version lineage, explicitly blocked on a sublinear deletion-delta source and enforceable I/O cost gates. +status: research-blocked +tags: [eng, rfc, merge, lineage, change-feed, performance, lance, omnigraph] +timestamp: 2026-07-10 +owner: +--- + +# RFC-027 — Lineage-based merge deltas + +**Status:** Research / blocked on deletion-delta discovery +**Date:** 2026-07-10 +**Depends on:** [RFC-022](rfc-022-unified-write-path.md)'s unified branch-merge +pipeline and capture-once write view +**Surveyed:** omnigraph 0.8.1; Lance 9.0.0-beta.15 (`f24e42c1`) +**Audience:** merge, storage, and performance maintainers +**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md). +Findings marked **BLOCKER** must be dispositioned before acceptance. + +--- + +## 0. Status and decision boundary + +The direction is recommended: branch merge should discover changed row IDs from +storage lineage, then read wide values only for those candidates. The proposed +replacement is **not implementation-ready** and this RFC does not authorize +removing `OrderedTableCursor`. + +Two facts block the O(delta) claim: + +1. filtering `_row_last_updated_at_version` is still a physical O(rows) column + scan unless a substrate index or change-log makes it selective; +2. a deleted row is absent from the target snapshot, so its version columns + cannot identify it. The current implementation finds deletions by scanning + and differencing both complete ID sets. + +The RFC advances only when both candidate discovery and deletion discovery have +measured costs bounded by the changed working set. Until then the existing merge +classifier remains the correctness fallback. + +## 1. Problem + +Current three-way merge classification streams base, source, and target tables +and compares row signatures. A one-row change can therefore read every row and +every wide property, including embeddings and blobs, while holding merge-wide +coordination longer than necessary. + +`changes/mod.rs` is narrower but not yet asymptotically different: + +- changed live rows are filtered by `_row_last_updated_at_version`; +- insert/update classification builds the full base ID set; +- deletion classification scans both ID sets; +- cross-branch fallback scans and compares complete ordered rows. + +Reusing that code unchanged is O(rows), not O(delta). This RFC replaces those +specific scans rather than relabeling them. + +## 2. Target contract + +For each table touched since the merge base: + +1. discover the source and target candidate row IDs from Lance metadata; +2. classify insert, update, and delete without loading unrelated user columns; +3. join the two candidate sets into the existing merge truth table; +4. fetch complete rows only for candidates whose disposition needs values; +5. stage the resulting delta through RFC-022 and publish once. + +The semantic oracle remains `merge_truth_table`. This RFC changes candidate +discovery and I/O, not conflict kinds, delete/update precedence, constraint +validation, or manifest atomicity. + +## 3. Live-row candidates + +On a stable-row-ID table in one physical lineage, Lance exposes: + +- `_row_created_at_version` — first creation of the logical row; +- `_row_last_updated_at_version` — latest modification of the logical row. + +For merge-base table version `Vb` and side version `Vs`, a live row is: + +- an insert when `Vb < created_at <= Vs`; +- an update when `created_at <= Vb` and + `Vb < last_updated_at <= Vs`. + +This classification removes the current full base-ID membership set. Candidate +scans project only `id`, edge endpoints when applicable, stable row ID, and the +two version columns. + +It does **not** by itself make the scan O(delta). Phase R1 must prove one of: + +1. Lance can build and maintain a scalar index over the version columns with + correct partial-coverage fallback; +2. transaction/fragment metadata exposes an equivalent bounded changed-row + iterator; +3. an upstream Lance change-feed primitive supplies these IDs directly. + +An index is derived state. Missing or partial coverage falls back to a correct +narrow scan and is reported; it never makes merge fail or omit candidates. + +## 4. Deletion delta is the blocker + +Deleted logical rows have no live record whose version columns can be filtered. +The current `deleted_ids_by_set_diff` scans all IDs at base and side. That term +alone prevents an O(delta) merge, even if inserts and updates become selective. + +Research must disposition these substrate-shaped options: + +### 4.1 Deletion-vector and stable-row-ID lineage + +Walk only transactions/fragments changed between `Vb` and `Vs`, compare their +deletion vectors, and translate newly deleted offsets to stable row IDs. This is +acceptable only if it handles merge-insert rewrites, updates that move rows, +compaction, fragment reuse, and deletion-vector materialization without scanning +unaffected fragments. + +### 4.2 Upstream Lance change log + +Consume a public Lance API that yields durable inserted/updated/deleted stable +row IDs by version range. A source-level prototype or private API is evidence, +not a dependency; the production surface must be public and pinned by +`lance_surface_guards.rs`. + +### 4.3 Atomic OmniGraph deletion deltas + +Write immutable per-commit deleted-ID rows in the same manifest CAS as the graph +commit. This is first-class commit metadata, not a side channel. It adds storage +and format liability and therefore needs a separate format decision before use; +it is not silently folded into v5. + +Until one option passes the cost and correctness gates, delete-bearing histories +use the existing classifier. There is no "usually O(delta)" claim that excludes +deletes without saying so in plan and metrics. + +## 5. Branch lineage and unsupported operations + +Lance version numbers are branch-local and can overlap. Candidate discovery must +use each manifest entry's physical `(table_path, table_branch, table_version)`; +it must not subtract graph commit numbers or compare equal numeric versions from +different branch lineages. + +Research must specify behavior for: + +- a lazy fork that still physically reads its parent table branch; +- the first branch-owned write after a fork; +- compaction and index-only versions between base and side; +- overwrite, restore, schema rewrite, and hard-drop operations; +- tables without stable row IDs or version metadata; +- a table dropped or introduced on only one side. + +Any shape not proven lineage-compatible takes the correctness fallback and +records a typed reason. Physical metadata gaps never weaken the logical merge. + +## 6. Proposed planner shape + +```text +ResolveMergeBase + -> DiscoverSideDelta(source) + -> DiscoverSideDelta(target) + -> JoinCandidateIds + -> FetchCandidateRows + -> ExistingTruthTableAndValidation + -> RFC022StageAndPublish +``` + +`DiscoverSideDelta` returns ordered, typed operations keyed by table and logical +row ID. Candidate order is deterministic. Payload fetches use structured key +lookups or SIP; they do not synthesize string `IN` filters or read embedding/blob +columns for candidates whose disposition needs only identity. + +The merge base, physical entries, and candidate cuts are captured once before +heavy work. After acquiring publish queues, RFC-022's OCC/read-set rule either +confirms the cut or restarts discovery; it never publishes a delta classified +against a moved side. + +## 7. Fallback and rollout safety + +`OrderedTableCursor` remains the universal fallback through the research and +shadow phases. Fallback reasons are closed enum values, including: + +- `DeletionDeltaUnavailable`; +- `VersionIndexUncovered`; +- `CrossLineageUnsupported`; +- `StableRowIdsUnavailable`; +- `OverwriteOrRestoreInRange`; +- `LineageMetadataInconsistent`. + +Before enabling the new path, shadow mode runs both classifiers on the same +captured snapshots and compares ordered operation sets and merge outcomes. A +mismatch fails the test or records a production diagnostic; it never silently +chooses the new answer. + +## 8. Cost contract + +Use `helpers::cost` with fixed `delta = 1` and table sizes of at least 10k, +100k, and 1M rows, both scalar-only and embedding-bearing. Measure candidate +discovery separately from candidate payload fetch. + +The acceptance target for a true lineage path is: + +- insert and update discovery I/O is flat within named slack as rows grow; +- delete discovery I/O is flat within the same discipline; +- bytes read scale with candidate identity/version columns plus fetched delta + payload, not total row width; +- peak RSS is bounded by candidate batch size, not table size; +- graph-history depth and unrelated catalog width do not change the curve. + +Local tests gate scan/fragment terms. S3 tests gate object-store RPC and opener +terms that local FS cannot expose. A benchmark without an asserted I/O slope is +supporting evidence, not the acceptance gate. + +If only live-row indexing passes, this RFC may prototype or shadow an +insert/update-only path, but it does not authorize production shipping. A +partial fast path requires a separately accepted RFC that names its O(rows) +delete fallback and operational value. This RFC and the default-classifier +O(delta) claim remain blocked until R2 passes. + +## 9. Correctness gates + +- `merge_truth_table` remains byte-for-byte the semantic disposition oracle. +- Property tests compare lineage and cursor classifiers over inserts, updates, + deletes, endpoint moves, cycles, and conflicting constraints. +- Dedicated cases cover compaction, index maintenance, restore, overwrite, + lazy forks, and same-number/different-branch versions. +- A one-row delete in a 1M-row table is the blocker test: the RFC cannot leave + research status until it is both correct and flat in table size. +- Surface guards pin every Lance version-column, stable-row-ID, deletion-vector, + transaction, and index behavior used by the chosen implementation. +- Fault tests move a side after discovery and prove OCC restarts instead of + publishing stale classification. + +## 10. Observability + +For every merge table report classifier path, fallback reason, source/target +candidate counts, deleted-ID discovery mechanism, fragments and bytes read, +payload columns fetched, discovery latency, and whether shadow results matched. + +Metrics distinguish logical delta size from physical discovery work. This is +required to catch an apparently correct path silently regressing to O(rows). + +## 11. Research plan + +| Phase | Content | Exit criterion | +|---|---|---| +| R0 | Instrument current cursor and `changes` paths; build shadow comparison harness | semantic and I/O baselines | +| R1 | Prove selective live-row discovery and branch-version mapping | insert/update flat-cost gate | +| R2 | Prototype all deletion-delta options against pinned Lance | one option passes delete correctness + flat-cost gate | +| R3 | Shadow new classifier across the full merge truth table and histories | zero mismatches; explicit fallback ledger | +| R4 | Enable lineage path behind a scoped feature/config gate | production diagnostics within budgets | +| R5 | Make lineage default and consider cursor retirement | all fallbacks dispositioned; no physical gap can break merge | + +The RFC remains **research / blocked** through R2. Choosing an OmniGraph commit +delta format in §4.3 requires its own format amendment before R3. diff --git a/docs/user/branching/index.md b/docs/user/branching/index.md index 20ea1252..951323f2 100644 --- a/docs/user/branching/index.md +++ b/docs/user/branching/index.md @@ -33,8 +33,10 @@ conflict kinds are on the [merge](merge.md) page. ## L2 — Recovery audit trail -Interrupted multi-table writes are recovered automatically the next time the graph is opened read-write. Recovery commits are recorded in the audit trail under the actor `omnigraph:recovery`, so you can find them with: - -```bash -omnigraph commit list --filter actor=omnigraph:recovery -``` +Interrupted multi-table writes are recovered automatically the next time the +graph is opened read-write. Each completed recovery is recorded internally in +`_graph_commit_recoveries.lance`. A roll-forward keeps the interrupted +writer's original commit id and actor; rollback and legacy recovery commits use +the reserved actor `omnigraph:recovery`. Consequently, `commit list` is not a +complete recovery log, and the CLI does not currently expose a query for the +internal recovery-audit table. diff --git a/docs/user/branching/transactions.md b/docs/user/branching/transactions.md index 6e6b1c42..bd559010 100644 --- a/docs/user/branching/transactions.md +++ b/docs/user/branching/transactions.md @@ -19,7 +19,7 @@ Two primitives, two scopes: | **One `.gq` query** (any number of statements inside) | The query itself — handled by the publisher's atomic manifest commit | Yes — all statements land together or none of them do | The publisher never publishes; target unchanged | | **Many queries that must succeed together** | Branches: `branch_create` → run N queries on the branch → `branch_merge` | Yes — the merge is a single atomic publish | Drop the branch (`branch_delete`); main is unaffected | -Snapshot isolation is per-query — every read inside one query sees one consistent manifest version. Two concurrent queries on the same branch see independent snapshots; the publisher's CAS catches racing writes. +Snapshot isolation is per-query — every read inside one query sees one consistent manifest version. Two concurrent queries on the same branch see independent snapshots. Mutation/load capture the branch head as coarse OCC authority, so a prepared plan is never silently reparented after another graph commit. ## Comparison with `BEGIN` / `COMMIT` @@ -136,7 +136,9 @@ This is the workflow agentic loops are designed around: **branches are the unit | Scenario | What happens | Caller action | |---|---|---| | Single query fails mid-flight | Publisher never publishes; target unchanged | Read the error, decide whether to retry | -| Concurrent writers race the same `(table, branch)` | Publisher CAS rejects the loser with a version-mismatch conflict | Refresh handle, retry the query | +| Branch authority changes before physical effects | Retryable inserts/loads fully reprepare; strict writes return `read_set_conflict` | For a surfaced strict conflict, refresh and retry deliberately | +| Authority changes after a physical effect | The write returns `recovery_required` and leaves its durable sidecar | Resolve recovery by read-write reopen/server restart before retrying | +| An overlapping recovery intent remains unresolved before effects | The write returns `recovery_required` with that intent's operation id and does not advance a table | Resolve recovery by read-write reopen/server restart before retrying | | Branch with N successful mutations, then merge fails (three-way conflict) | Each individual mutation already committed on the branch; merge surfaces `MergeConflicts` | Inspect, decide whether to keep working on the branch, abandon it (`branch_delete`), or resolve and re-merge | | Process crashes mid-branch-workflow | Each completed mutation on the branch is durable | Re-open the graph, continue where you left off | diff --git a/docs/user/cli/reference.md b/docs/user/cli/reference.md index 3b97800c..83b59236 100644 --- a/docs/user/cli/reference.md +++ b/docs/user/cli/reference.md @@ -21,7 +21,7 @@ Top-level command families and subcommands. Graph-targeting commands accept a po | `schema plan \| apply \| show (alias: get)` | migrations. `apply` refuses a cluster-managed graph (one whose storage is inside a cluster) and points at `cluster apply` — those graphs evolve through the cluster ledger, not a direct apply | | `lint` (alias: `check`) | offline / graph-backed query validation. Replaces `query lint` / `query check`, which are kept as deprecated argv-level shims that print a one-line warning and rewrite to `omnigraph lint` | | `cluster validate \| plan \| apply \| approve \| status \| refresh \| import \| force-unlock` | declarative cluster control plane. `validate` checks a local `cluster.yaml` folder and referenced schema/query/policy files; `plan` diffs it against local JSON state at `__cluster/state.json`, annotates dispositions, and embeds real schema-migration previews; `apply` converges the cluster — stored-query/policy catalog writes (content-addressed under `__cluster/resources/`), graph creates, schema updates (soft drops only; `--as` records the actor), and graph deletes behind a digest-bound approval from `cluster approve --as ` (`apply`/`approve` default the actor from `~/.omnigraph/config.yaml`'s `operator.actor` when `--as` is omitted); what apply converges is what an `omnigraph-server --cluster ` deployment serves on its next restart (`--cluster` is the server's only boot source — cluster-only); `status` reads the state ledger; `refresh`/`import` explicitly update local JSON state from read-only graph observations; `force-unlock ` manually removes a held local state lock by exact id | -| `optimize` | non-destructive Lance compaction (skips tables with `Blob` columns or uncovered drift; `--json` reports `skipped`) | +| `optimize` | non-destructive Lance compaction + index reconciliation (blob-bearing tables use the normal path; tables with uncovered drift are skipped and `--json` reports `skipped`) | | `repair [--confirm] [--force]` | preview or explicitly publish uncovered manifest/head drift. `--confirm` heals verified maintenance drift and exits non-zero if suspicious/unverifiable drift is refused; `--force --confirm` publishes suspicious/unverifiable drift after operator review | | `cleanup --keep N --older-than 7d --confirm` | destructive version GC (`--confirm` to execute; also needs `--yes` against a non-local `s3://` target — see *Write diagnostics & destructive confirmation*) | | `embed` | offline JSONL embedding pipeline | diff --git a/docs/user/concepts/storage.md b/docs/user/concepts/storage.md index bad7726c..e5329827 100644 --- a/docs/user/concepts/storage.md +++ b/docs/user/concepts/storage.md @@ -7,7 +7,7 @@ Every node type and every edge type is its own Lance dataset: - **Columnar Arrow storage**: each property is a column; nullable per Arrow schema. - **Fragments**: data is partitioned into fragments; new writes create new fragments. - **Manifest versioning**: every commit produces a new dataset version; old versions remain readable. -- **Stable row IDs**: stable row IDs are enabled on every Lance dataset OmniGraph creates — node and edge data tables, `__manifest`, the commit-graph datasets, and any future system tables. This is an architectural invariant: the flag is one-way at dataset create, so a future change that introduces a Lance dataset must preserve it. Consequences: `_row_created_at_version` and `_row_last_updated_at_version` are available on every dataset (load-bearing for change-feed validators); indices survive `omnigraph optimize`. Pre-0.4.x graphs created before this code path settled may have datasets without the flag and cannot be retrofitted in place — the supported path is dump-and-reload. The rewrite path used by `schema_apply` preserves the flag. +- **Stable row IDs**: stable row IDs are enabled on every Lance dataset OmniGraph creates — node and edge data tables, `__manifest`, `_graph_commit_recoveries.lance`, and any future system tables. This is an architectural invariant: the flag is one-way at dataset create, so a future change that introduces a Lance dataset must preserve it. Consequences: `_row_created_at_version` and `_row_last_updated_at_version` are available on every dataset (load-bearing for change-feed validators); indices survive `omnigraph optimize`. Pre-0.4.x graphs created before this code path settled may have datasets without the flag and cannot be retrofitted in place — the supported path is dump-and-reload. The rewrite path used by `schema_apply` preserves the flag. - **Append / delete / `merge_insert`**: native Lance write modes. - **Per-dataset branches** (Lance native): copy-on-write at the dataset level. - **Object-store agnostic**: file://, s3://, gs://, az://, http (read-only via Lance) — OmniGraph wires file:// and s3://. @@ -30,7 +30,7 @@ OmniGraph is **not** a single Lance dataset; it is a *graph* of datasets coordin - **Snapshot reconstruction**: latest visible `table_version` per `(table_key, table_branch)` minus tombstones — rows where `object_type = table_tombstone`, whose own `table_version` (acting as the tombstone version) is `>= the entry's table_version`. - **Atomic publish**: multi-dataset commits publish so that a single write to `__manifest` flips all the new sub-table versions visible at once. - **Row-level CAS on the merge-insert join key**: `object_id` carries an unenforced-primary-key annotation so Lance's bloom-filter conflict resolver rejects two concurrent commits that land the same `object_id` row. Without this annotation, Lance's transparent rebase would admit silent duplicates from racing publishers. -- **Optimistic concurrency control on publish**: a publish asserts the manifest's current latest non-tombstoned version for each touched table is exactly what the caller observed; mismatches surface as an `ExpectedVersionMismatch` manifest conflict naming the table and the expected/actual versions. Concurrent advances surface as a conflict rather than being silently rebased through. +- **Optimistic concurrency control on publish**: legacy writers assert the manifest's current latest non-tombstoned version for each touched table; a mismatch surfaces as `ExpectedVersionMismatch`. RFC-022-enrolled mutation/load attempts use a stronger, branch-wide contract: preparation captures the Lance-native branch identity, the exact `graph_head` (including absence), the accepted schema identity/catalog, and one base table snapshot. Under root-shared schema → branch → sorted-table gates, the engine revalidates that complete authority before any physical effect, then the publisher rechecks the exact native branch identity/head plus the touched-table versions. An insert-only mutation or Append/Merge load whose authority changed before effects discards and fully reprepares the bounded attempt; Update/Delete/Overwrite returns `ReadSetChanged`. Once any Lance effect is durable, any later failure leaves the recovery sidecar authoritative and returns `RecoveryRequired` instead of silently rebasing or replaying the prepared plan. ### Internal schema versioning @@ -95,7 +95,7 @@ flowchart TB - **`__manifest/`** is a Lance dataset whose rows describe which sub-table version is published at which graph-branch. Reading a snapshot starts here. - **`nodes/`** and **`edges/`** are sibling directories holding one Lance dataset per declared type. Names are `fnv1a64-hex` of the type name to keep paths fixed-length and case-safe. - The graph commit DAG lives in **`__manifest`** as `graph_commit` / `graph_head` rows written in the publish CAS (RFC-013 Phase 7). The former `_graph_commits.lance` / `_graph_commit_actors.lance` lineage tables are retired — a graph this binary creates has neither. -- **`_graph_commit_recoveries.lance`** — one row per crash-recovery action. Joined by `graph_commit_id` to the graph commit lineage (the `graph_commit` rows in `__manifest` since RFC-013 Phase 7); the linked commit carries `actor_id=omnigraph:recovery`. Operators correlate recoveries with the original mutations they rolled forward / back via this join. +- **`_graph_commit_recoveries.lance`** — one internal row per completed crash-recovery action, including its exact per-table outcomes and the original actor. It joins by `graph_commit_id` to the graph commit lineage in `__manifest`. A v3 roll-forward keeps the interrupted writer's original actor; rollback and legacy recovery commits use `omnigraph:recovery`. The CLI does not currently expose this internal table. - **`__recovery/{ulid}.json`** — transient sidecar files written by a writer before it advances the underlying dataset, deleted once the matching manifest publish succeeds. A sidecar persisting after process exit means the writer crashed mid-commit; the next read-write open processes it. Steady-state directory is empty. - **`_refs/branches/{name}.json`** is graph-level branch metadata — pointers from a branch name to the manifest version it heads. - **Inside each Lance dataset** (orange): the standard Lance directory layout. `_versions/{n}.manifest` records every commit; `data/` holds the actual Arrow fragments; `_indices/{uuid}/` holds index segments with their own `fragment_bitmap` for partial coverage; `_refs/` holds Lance-native per-dataset branches and tags. diff --git a/docs/user/mutations/index.md b/docs/user/mutations/index.md index 736d3449..afc02c35 100644 --- a/docs/user/mutations/index.md +++ b/docs/user/mutations/index.md @@ -21,6 +21,12 @@ properties in the assignment block (`insert WorksAt { person: $p, org: $o }`). `` is a literal, `$param`, or `now()`. +On a blob-bearing type, an update materializes and rewrites blob payloads only +for the rows matched by its predicate, including blobs the update does not +change. This keeps correctness independent of physical index state, but adds +read/write I/O proportional to the matched blob bytes; use selective update +predicates for large blobs. + ## Atomicity A change query publishes **one commit** at the end of the query. Multiple @@ -29,6 +35,19 @@ failure leaves the graph untouched. See [transactions](../branching/transactions for the per-query atomicity contract and [branches](../branching/index.md) for multi-query workflows. +Concurrent changes use optimistic concurrency over the whole target branch. +Insert/Merge/Append operations whose branch changed before physical effects are +discarded and fully revalidated with a bounded internal retry. Strict +Update/Delete/Overwrite operations instead return a structured conflict. This +branch-wide token is deliberately conservative: a change to a different table +can invalidate a prepared strict write because constraints may have read it. + +If the synchronous barrier finds an unresolved overlapping recovery intent, or +if a conflict is discovered after a Lance table effect is durable, the request +returns `recovery_required` with an operation id. Do not immediately retry that +request; reopen the graph read-write (or restart the server) so the durable +recovery intent is resolved first. + ## Inserts/updates and deletes cannot mix in one query A single change query must be **either insert/update-only or delete-only**. diff --git a/docs/user/operations/audit.md b/docs/user/operations/audit.md index 7e8b24de..c5cd6f90 100644 --- a/docs/user/operations/audit.md +++ b/docs/user/operations/audit.md @@ -31,13 +31,12 @@ List commits to see who made each change: omnigraph commit list graph.omni ``` -System-initiated writes use reserved actor ids — for example, automatic recovery -of an interrupted write records `omnigraph:recovery`, so operator changes and -machine repairs are distinguishable in the history: - -```bash -omnigraph commit list --filter actor=omnigraph:recovery graph.omni -``` +System-initiated writes use reserved actor ids. Rollback and legacy recovery +commits use `omnigraph:recovery`, while a v3 roll-forward preserves the +interrupted writer's original commit id and actor. Exact recovery actions and +per-table outcomes are stored in the internal +`_graph_commit_recoveries.lance` audit table; the CLI does not currently expose +that table, so `commit list` alone is not a complete recovery log. ## What is tracked diff --git a/docs/user/operations/errors.md b/docs/user/operations/errors.md index 85b4fdec..20058eb3 100644 --- a/docs/user/operations/errors.md +++ b/docs/user/operations/errors.md @@ -8,9 +8,11 @@ - `Io(io::Error)` - `Manifest(ManifestError { kind: BadRequest|NotFound|Conflict|Internal, details: Option, … })` - `ManifestConflictDetails::ExpectedVersionMismatch { table_key, expected, actual }` — caller's `expected_table_versions` did not match the manifest's current latest non-tombstoned version (set by `OmniError::manifest_expected_version_mismatch`). + - `ManifestConflictDetails::ReadSetChanged { member, expected, actual }` — an RFC-022 prepared write's branch/head/table authority changed before physical effects. HTTP returns **409** with `read_set_conflict`. A retry must start from preparation; strict writes leave that choice to the caller. - `ManifestConflictDetails::RowLevelCasContention` — Lance row-level CAS rejected the publish because a concurrent writer landed the same `object_id`. Retried internally by the publisher; only surfaces if the retry budget exhausts. - **D₂ parse-time rejection**: a single mutation query that mixes inserts/updates with deletes errors out *before any I/O* with kind `BadRequest`. Message: `mutation '' on the same query mixes inserts/updates and deletes; split into separate mutations: (1) inserts and updates, then (2) deletes`. See [query-language.md](../queries/index.md) for the rule. - `MergeConflicts(Vec)` +- `RecoveryRequired { operation_id, reason }` — an overlapping durable recovery intent remains unresolved. Its physical effects may already have landed, or it may still be armed before the first effect. HTTP returns **503** with `recovery_required.operation_id`. Resolve the sidecar through a read-write reopen/server restart before retrying; this is intentionally not an ordinary OCC retry. Compiler-side `CompilerError` covers parse / catalog / type / storage / plan / execution / arrow / lance / IO / manifest / unique-constraint, each with structured spans (`SourceSpan { start, end }`) for ariadne-style diagnostics. The legacy `NanoError` name remains as a deprecated compatibility alias. diff --git a/docs/user/operations/maintenance.md b/docs/user/operations/maintenance.md index 4f901ff5..a731af8e 100644 --- a/docs/user/operations/maintenance.md +++ b/docs/user/operations/maintenance.md @@ -9,13 +9,13 @@ - **Also compacts the internal `__manifest` table** (RFC-013 step 2), which accumulates one fragment per commit — it now carries the graph lineage and actor rows inline (RFC-013 Phase 7: `graph_commit` / `graph_head` rows), so on the authenticated write path every commit's actor lands here too — and otherwise makes every write's metadata scan grow with history. (The `_graph_commits.lance` / `_graph_commit_actors.lance` tables are retired, so there is no separate lineage table to compact.) It takes a simpler path than data tables: `__manifest` is read at its latest version, so compaction just advances its version in place — **no manifest publish and no recovery sidecar**. (The sidecar-free property is not because it is one commit — `compact_files` can emit a `ReserveFragments` commit before the `Rewrite`, and the auto-cleanup strip below is a further commit — but because every one of those commits is content-preserving and the table is read at its latest version, so a crash at any point leaves it readable and content-identical and the next `optimize` re-plans.) It appears in the returned stats under `table_key` `"__manifest"`. It is **not yet covered by `cleanup`**, so its version chain still grows until the cleanup half lands (it requires a cleanup-resurrection safeguard first); run `optimize` on a cadence to keep per-write metadata scans flat. - **`optimize` is non-destructive by construction — it never garbage-collects versions, on any table (data or internal).** Compaction rewrites fragments and advances the version; old versions stay reachable until you run `cleanup`. This holds even for a graph created by an older binary that stored an on-by-default Lance `auto_cleanup` hook: `compact_files` / `optimize_indices` commit with the hook enabled and expose no skip override, so before compacting **any** table `optimize` strips its stale `lance.auto_cleanup.*` config first, so Lance's commit-time GC hook cannot fire and silently prune `__manifest`-pinned versions. (Graphs created by current binaries store no such config; the strip is the upgrade-path safety net.) The internal-table path additionally tolerates a concurrent live writer: it runs a **bounded** rebase-and-retry, so transient contention does not fail the operator's `optimize` or the live write — but sustained contention past the retry budget surfaces a loud conflict error rather than looping forever (bounded and observable, not a silent give-up). The data-table path holds the per-table write queue while it compacts, so it does not contend with mutations on that table in the first place. - **Reindex (index coverage maintenance).** A scalar/FTS/vector index only covers the fragments it was built over. Rows appended after the index was built (e.g. by `load --mode merge`, whose commit does not rebuild an already-existing index) are scanned unindexed, and compaction itself rewrites fragments out of an index's coverage. `optimize` runs Lance's incremental `optimize_indices` after compaction to fold those fragments back in (a delta merge, not a full retrain), restoring full coverage so equality/range/traversal predicates stay index-accelerated. This is why a table with **no compaction work but stale index coverage still commits** a new version under `optimize`. Run `optimize` on a cadence at least as frequent as your freshness window so recently-loaded rows do not linger in the unindexed flat-scan tail. -- **Create declared-but-missing indexes (the index reconciler).** `@index`/`@key` declares intent; `schema apply` records it but builds nothing, and `load`/`mutate` defer a column that cannot be built yet (a `Vector` column with no trainable vectors). `optimize` materializes any such declared-but-unbuilt index over the compacted layout — so it is the convergence path for an `@index` added after data exists, or a vector index whose embeddings arrived via a later `embed`. A column still not buildable (no vectors yet) is reported on the table's stat as `pending_indexes` (visible in `--json`), not treated as a failure; the next `optimize` retries. So `optimize` is the single operator-facing index reconciler: it compacts, restores coverage, **and** builds declared-but-missing indexes. +- **Create declared-but-missing indexes (the index reconciler).** `@index`/`@key` declares intent; `schema apply`, `load`, and `mutate` build no physical indexes inline. They record or publish only their exact logical/data effects and leave all index materialization to `ensure_indices`/`optimize`. `optimize` materializes every buildable declared-but-missing index over the compacted layout — so it is the convergence path for an `@index` added after data exists, or a vector index whose embeddings arrived via a later `embed`. A column still not buildable (no vectors yet) is reported on the table's stat as `pending_indexes` (visible in `--json`), not treated as a failure; the next `optimize` retries. So `optimize` is the single operator-facing index reconciler: it compacts, restores coverage, **and** builds declared-but-missing indexes. - Each table's compact→reindex→publish serializes with concurrent mutations on the same table. A crash mid-operation is recovered automatically on the next open (both compaction and reindex are content-preserving, so roll-forward is always safe). - **Requires a recovered graph.** `optimize` refuses (errors) when a pending crash-recovery operation is present — operating on an unrecovered graph could publish a partial write that recovery would roll back. Reopen the graph to run recovery, then re-run `optimize`. - **Uncovered drift is skipped, not interpreted.** If a table's underlying version is ahead of the version recorded in `__manifest` and no crash-recovery record covers that movement, `optimize` reports `skipped: DriftNeedsRepair` with the manifest/head versions and leaves the table untouched. Run `omnigraph repair` to classify and explicitly publish that drift. - Bounded by `OMNIGRAPH_MAINTENANCE_CONCURRENCY` (default 8). - Returns per-table stats: `table_key, fragments_removed, fragments_added, committed, skipped, manifest_version, lance_head_version, pending_indexes` (the last lists any declared `@index` column the reconciler could not build this run, with the reason — e.g. a vector column with no trainable vectors yet). -- **Blob tables are skipped.** A table that declares any `Blob` property is not compacted: it is reported with `skipped: BlobColumnsUnsupportedByLance` (and logged) instead of compacted, and the rest of the sweep proceeds normally. **Reads and writes are unaffected** — only compaction is. Consequence: fragment count and deleted-row space on blob tables are not reclaimed; query results are never affected. A skipped blob table is also **not reindexed** in the same sweep (the skip happens before the reindex step), so its index coverage on appended rows is not refreshed by `optimize` today. +- **Blob tables use the normal compaction and reindex path.** Lance 8.0.0+ supports blob-v2 compaction, so OmniGraph no longer has a blob-specific skip or capability gate. Fragment reclamation and index-coverage repair therefore apply to blob-bearing tables like every other table. ## `repair` — explicit @@ -39,7 +39,12 @@ - CLI guards with `--confirm`; without it, prints a preview line. - **Non-local consent.** Against a non-local target (an `s3://` store/cluster), `cleanup` additionally requires `--yes` on top of `--confirm`: a TTY is prompted, and a non-interactive run (no TTY, or `--json`) refuses rather than destroying. A local (`file://`) target needs only `--confirm`. The same `--yes` gate applies to overwrite `load` and `branch delete`; every maintenance run echoes its resolved target to stderr (suppress with `--quiet`). - **Recovery floor:** `--keep < 3` may garbage-collect versions that crash recovery needs as a rollback target. Default `--keep 10` is safe. -- **Orphaned-branch reconciliation:** before the version GC, cleanup reclaims any per-table or commit-graph branch absent from the manifest branch list. These orphans arise when a `branch_delete` flips the manifest authority but a downstream best-effort reclaim does not complete (see [branches-commits.md](../branching/index.md)). The reconciler is idempotent (it no-ops once nothing is orphaned), runs regardless of the `keep_versions` / `older_than` values (those gate version GC only), and never reclaims `main` or system-branch forks. Reclaimed forks are logged. +- **Requires clean recovery state.** If any durable recovery intent is pending, + cleanup refuses before orphan reconciliation or version GC. Reopen the graph + read-write (or restart the server) to resolve recovery, then rerun cleanup; + deleting transaction/version history while an intent is pending would make + exact effect ownership unverifiable. +- **Orphaned-branch reconciliation:** before the version GC, cleanup reclaims any per-table Lance branch absent from the manifest branch list. These orphans arise when a `branch_delete` flips the manifest authority but a downstream best-effort reclaim does not complete (see [branches-commits.md](../branching/index.md)). The reconciler is idempotent (it no-ops once nothing is orphaned), runs regardless of the `keep_versions` / `older_than` values (those gate version GC only), and never reclaims `main` or system-branch forks. Reclaimed forks are logged. Graph lineage has no separate branch dataset: it lives in `__manifest`. ## Tombstones diff --git a/docs/user/operations/server.md b/docs/user/operations/server.md index 3f6bcd0d..b047904b 100644 --- a/docs/user/operations/server.md +++ b/docs/user/operations/server.md @@ -162,25 +162,51 @@ Only `/export` streams (`application/x-ndjson`, MPSC channel + `Body::from_strea ## Error model -Uniform `ErrorOutput { error, code?, merge_conflicts[], manifest_conflict? }` with `code ∈ unauthorized | forbidden | bad_request | not_found | conflict | too_many_requests | internal`. Merge conflicts attach structured `MergeConflictOutput { table_key, row_id?, kind, message }`. +Uniform +`ErrorOutput { error, code?, merge_conflicts[], manifest_conflict?, read_set_conflict?, recovery_required? }` +with +`code ∈ unauthorized | forbidden | bad_request | not_found | method_not_allowed | conflict | too_many_requests | internal`. +Merge conflicts attach structured +`MergeConflictOutput { table_key, row_id?, kind, message }`. -`manifest_conflict` is set on **concurrent-write rejections** (HTTP 409): the -caller's pre-write view of one table's manifest version was stale. -`ManifestConflictOutput { table_key, expected, actual }` tells the client -which table to refresh and retry. This is the conflict shape produced by -concurrent `/mutate` (or its `/change` alias), `/load` (or its deprecated -`/ingest` alias) calls landing the same `(table, branch)` race. +`manifest_conflict` is set on legacy per-table manifest-version rejections +(HTTP 409). `ManifestConflictOutput { table_key, expected, actual }` tells the +client which table was stale. Mutation and load use the unified coarse-OCC +adapter described next; other writers retain this older conflict shape until +they are enrolled. -HTTP status codes used: 200, 400, 401, 403, 404, 409, 429, 500. +`read_set_conflict` is set when a prepared write is rejected before any table +effect because its branch authority changed. The HTTP status is 409 and +`ReadSetConflictOutput { member, expected, actual }` identifies the stale +authority member. The engine already performs a bounded full-attempt retry for +mutation inserts and load `append`/`merge`. Strict mutation updates/deletes and +load `overwrite` return the 409 to the caller instead of being replayed. + +`recovery_required` is set when an overlapping durable recovery intent remains +unresolved; its table effects may or may not have started. The HTTP status is 503 and +`RecoveryRequiredOutput { operation_id }` names the durable recovery intent. +The optional `code` field is omitted for this response: adding a new value to +the closed error-code enum would break older clients, while the optional +structured field is additive and rolling-safe. +Do not blindly resubmit the write: let a read-write open or the recovery sweep +resolve that operation first, then retry from a fresh snapshot. + +HTTP status codes used: 200, 400, 401, 403, 404, 405, 409, 429, 500, 503. ## Per-actor admission control -Disjoint -`(table, branch)` writes from different actors now run concurrently, -guarded only by the engine's per-(table, branch) write queue. To keep -one heavy actor from exhausting shared capacity (Lance I/O, manifest -churn, network), the server gates mutating handlers through per-process -admission limits configured from environment variables: +RFC-022-enrolled mutation/load preparation runs outside the effect gates, so +parsing, validation, and reclaimable fragment staging can overlap across branches. +Readers acquire none of these gates. Before the first durable effect, however, an +attempt acquires the exclusive root schema gate, then its branch-effect gate and +sorted table queues, and holds all of them through manifest publication. The root +schema gate means enrolled effect windows on one graph currently serialize +in-process even across different branches; the branch gate preserves one atomic +graph-head validation authority, while table queues protect each concrete Lance +effect and legacy writer. These are process-local ordering gates, not a +cross-process lock. To keep one heavy actor from exhausting shared capacity +(Lance I/O, manifest churn, network), the server gates mutating handlers through +per-process admission limits configured from environment variables: | Env var | Default | Purpose | |---|---|---| diff --git a/docs/user/reference/constants.md b/docs/user/reference/constants.md index e952a377..f3324912 100644 --- a/docs/user/reference/constants.md +++ b/docs/user/reference/constants.md @@ -4,14 +4,13 @@ |---|---|---| | `MANIFEST_DIR` | `__manifest` | manifest layout | | Commit graph dirs (retired) | `_graph_commits.lance` / `_graph_commit_actors.lance` | retired in Phase B; lineage lives in `__manifest` (`graph_commit` / `graph_head` rows) since RFC-013 Phase 7. A graph this binary creates has neither. | -| Recovery audit dir | `_graph_commit_recoveries.lance` | one row per crash-recovery action (`omnigraph commit list --filter actor=omnigraph:recovery`) | +| Recovery audit dir | `_graph_commit_recoveries.lance` | internal exact record of completed crash-recovery actions; no public CLI query yet | | Run branch prefix (legacy, removed) | `__run__` | pre-v0.4.0 Run state machine; no longer a reserved name. A graph still carrying `__run__*` branches is sub-v4 and refused on open (rebuild via export/import). | | Schema apply lock | `__schema_apply_lock__` | schema apply | | Manifest publisher retry budget | `PUBLISHER_RETRY_BUDGET = 5` | manifest publish | | Internal manifest schema version | `INTERNAL_MANIFEST_SCHEMA_VERSION = 4` | manifest migrations (v4 = graph lineage in `__manifest`, RFC-013 Phase 7) | | Merge stage batch | `MERGE_STAGE_BATCH_ROWS = 8192` | merge execution | | Maintenance concurrency | `OMNIGRAPH_MAINTENANCE_CONCURRENCY=8` | optimize/cleanup | -| Lance blob compaction support | `LANCE_SUPPORTS_BLOB_COMPACTION = false` | optimize | | Graph index cache size | `8` (LRU) | runtime cache | | Expand indexed-path frontier ceiling | `OMNIGRAPH_EXPAND_INDEXED_MAX_FRONTIER=1024` | traversal | | Expand indexed-path hop ceiling | `OMNIGRAPH_EXPAND_INDEXED_MAX_HOPS=6` | traversal | diff --git a/docs/user/search/indexes.md b/docs/user/search/indexes.md index af8c1287..c861ec22 100644 --- a/docs/user/search/indexes.md +++ b/docs/user/search/indexes.md @@ -27,7 +27,7 @@ list/`Blob` columns → none. ## L2 — OmniGraph orchestration -- **`@index`/`@key` declares intent; the physical index is derived state.** A migration records the declaration in the catalog/IR and never fails on it — `schema apply` builds **no** indexes (adding an `@index` to an existing column is a pure metadata change that touches no table data). `load`/`mutate` build declared indexes inline as part of the write, but a column that can't be built yet (a `Vector` column with no trainable vectors — IVF k-means needs ≥1 vector, e.g. rows loaded before `embed` runs) is left **pending**, not fatal. Reads stay correct meanwhile: a missing/partial index degrades to a scan (vector search to brute-force). A later `ensure_indices`/`optimize` materializes the pending index once it is buildable. This mirrors how LanceDB builds indexes asynchronously and serves unindexed rows by brute-force. +- **`@index`/`@key` declares intent; the physical index is derived state.** A migration records the declaration in the catalog/IR and never fails on it — `schema apply` builds **no** indexes. Mutation/load likewise publish only their exact data effects; they do not widen the recovery plan with index commits. Reads stay correct while an index is missing or partially covered by falling back to scans (vector search to brute-force). A later `ensure_indices`/`optimize` materializes every buildable declaration; an untrainable Vector column remains pending rather than failing the logical write. - `ensure_indices()` / `ensure_indices_on(branch)` — idempotent build of BTREE + inverted + vector indexes for the current head; safe to re-run; returns the columns it had to defer as pending. `optimize` runs it after compaction, so the maintenance cron is the convergence path for deferred indexes. - Indexes are built on the *branch head* (not on a snapshot), so reads always see the current index state. - **Lazy branch forking for indexes**: a branch that hasn't mutated a sub-table doesn't need its own index — the main lineage's index is reused until the first write triggers a copy-on-write fork. diff --git a/openapi.json b/openapi.json index df532f35..93b8969c 100644 --- a/openapi.json +++ b/openapi.json @@ -212,6 +212,16 @@ } } } + }, + "503": { + "description": "An overlapping durable recovery intent must be resolved before retry", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } } }, "security": [ @@ -310,6 +320,16 @@ } } } + }, + "503": { + "description": "An overlapping durable recovery intent must be resolved before retry", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } } }, "security": [ @@ -397,6 +417,16 @@ } } } + }, + "503": { + "description": "An overlapping durable recovery intent must be resolved before retry", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } } }, "security": [ @@ -477,7 +507,7 @@ } }, "409": { - "description": "Merge conflict", + "description": "Write-authority conflict", "content": { "application/json": { "schema": { @@ -495,6 +525,16 @@ } } } + }, + "503": { + "description": "An overlapping durable recovery intent must be resolved before retry", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } } }, "deprecated": true, @@ -795,6 +835,16 @@ } } }, + "409": { + "description": "Prepared load authority changed before effects", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, "429": { "description": "Per-actor admission cap exceeded; honor `Retry-After` header", "content": { @@ -804,6 +854,16 @@ } } } + }, + "503": { + "description": "An overlapping durable recovery intent must be resolved before retry", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } } }, "deprecated": true, @@ -884,6 +944,16 @@ } } }, + "409": { + "description": "Prepared load authority changed before effects", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, "429": { "description": "Per-actor admission cap exceeded; honor `Retry-After` header", "content": { @@ -893,6 +963,16 @@ } } } + }, + "503": { + "description": "An overlapping durable recovery intent must be resolved before retry", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } } }, "security": [ @@ -908,7 +988,7 @@ "mutations" ], "summary": "Apply a GQ mutation to a branch (canonical mutation endpoint).", - "description": "Writes to the named `branch` (defaults to `main`). Mutations are atomic\nper call and produce a new commit. Returns counts of nodes and edges\naffected. **Destructive**: on success the branch is updated; rejected\nmutations may still acquire locks briefly. Returns 409 on merge conflict.\n\nPairs with `POST /query` (read-only). The legacy `POST /change` route\nhas identical semantics and is kept as a deprecated alias.", + "description": "Writes to the named `branch` (defaults to `main`). Mutations are atomic\nper call and produce a new commit. Returns counts of nodes and edges\naffected. **Destructive**: on success the branch is updated; rejected\nmutations may still acquire locks briefly. Returns 409 when the prepared\nwrite authority changes before effects.\n\nPairs with `POST /query` (read-only). The legacy `POST /change` route\nhas identical semantics and is kept as a deprecated alias.", "operationId": "cluster_mutate", "parameters": [ { @@ -973,7 +1053,7 @@ } }, "409": { - "description": "Merge conflict", + "description": "Write-authority conflict", "content": { "application/json": { "schema": { @@ -991,6 +1071,16 @@ } } } + }, + "503": { + "description": "An overlapping durable recovery intent must be resolved before retry", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } } }, "security": [ @@ -1154,7 +1244,7 @@ } }, "409": { - "description": "Merge conflict", + "description": "Stored mutation write-authority conflict", "content": { "application/json": { "schema": { @@ -1182,6 +1272,16 @@ } } } + }, + "503": { + "description": "A stored mutation is blocked by a durable recovery intent", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } } }, "security": [ @@ -1900,6 +2000,28 @@ "items": { "$ref": "#/components/schemas/MergeConflictOutput" } + }, + "read_set_conflict": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/ReadSetConflictOutput", + "description": "Set when a prepared write's logical authority changed before effects." + } + ] + }, + "recovery_required": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/RecoveryRequiredOutput", + "description": "Set when an overlapping durable recovery intent must be resolved before\nretry. Its table effects may or may not have started." + } + ] } } }, @@ -2413,6 +2535,30 @@ } } }, + "ReadSetConflictOutput": { + "type": "object", + "description": "Structured authority mismatch for an RFC-022 prepared write. Values are\nstrings because members include optional graph commit ids and future\nauthority tokens, not only numeric table versions.", + "required": [ + "member" + ], + "properties": { + "actual": { + "type": [ + "string", + "null" + ] + }, + "expected": { + "type": [ + "string", + "null" + ] + }, + "member": { + "type": "string" + } + } + }, "ReadTargetOutput": { "type": "object", "properties": { @@ -2430,6 +2576,17 @@ } } }, + "RecoveryRequiredOutput": { + "type": "object", + "required": [ + "operation_id" + ], + "properties": { + "operation_id": { + "type": "string" + } + } + }, "SchemaApplyOutput": { "type": "object", "required": [