apunkt/omnigraph
1
0
Fork 0
mirror of https://github.com/ModernRelay/omnigraph.git synced 2026-07-12 03:12:11 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Implement RFC-022 unified graph write protocol (#343)

Browse source 
* Implement unified graph write protocol

* Preserve recovery error wire compatibility
This commit is contained in:
Andrew Altshuler 2026-07-11 14:02:54 +03:00 • committed by GitHub
parent 0c8d769501
commit f758ff0d17
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
80 changed files with 13393 additions and 2050 deletions
Download patch file Download diff file Expand all files Collapse all files

10
AGENTS.md
View file

@ -32,7 +32,7 @@ OmniGraph is a typed property-graph engine built as a coordination layer over ma
- **Languages**: a `.pg` schema language and a `.gq` query language, both Pest-based, with a typed IR.
- **Multi-modal querying**: vector ANN (`nearest`), full-text (`search`/`fuzzy`/`match_text`/`bm25`), Reciprocal Rank Fusion (`rrf`), and graph traversal (`Expand`, anti-join `not { … }`) in one runtime.
- **Branches and commits across the whole graph**: Git-style — every successful publish appends to a commit DAG; merges are three-way at the row level.
- **Atomic per-query writes**: `mutate_as` and `load` accumulate insert/update batches into an in-memory `MutationStaging.pending` per touched table; one `stage_*` + `commit_staged` per table runs at end-of-query, then `ManifestBatchPublisher::publish` commits the manifest atomically with per-table `expected_table_versions` CAS. A mid-query failure leaves Lance HEAD untouched on staged tables — no drift, no run state machine, no staging branches. Deletes stage through the same path (MR-A: `stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`), so they no longer advance Lance HEAD inline. D₂ at parse time is a deliberate boundary — one mutation query is constructive (insert/update) XOR destructive (delete) — so read-your-writes within a query stays unambiguous and each table commits at most one version; compose mixed operations via separate mutations, or a branch for single-commit atomicity.
- **Atomic per-query writes**: `mutate_as` and `load` accumulate insert/update batches into an in-memory `MutationStaging.pending` per touched table. Their RFC-022 adapter resolves or rejects relevant recovery intents before base capture, captures `(native branch id, exact graph_head, schema identity)`, then rechecks recovery and authority under schema → branch → table gates. It arms a schema-v3 sidecar containing exact Lance transaction identities + pre-minted lineage, commits each table with zero transparent conflict retries, confirms the achieved effects, then publishes under the same token. Retryable pre-effect authority conflicts fully reprepare; strict conflicts return `ReadSetChanged`; an unresolved intent or any post-effect publish error returns `RecoveryRequired`. Deletes stage through the same path. D₂ at parse time remains the constructive (insert/update) XOR destructive (delete) boundary.
- **HTTP server**: Axum + utoipa OpenAPI, bearer auth (SHA-256 hashed, optional AWS Secrets Manager). Cedar policy enforcement is engine-wide — every `_as` writer calls `Omnigraph::enforce(action, scope, actor)`, so HTTP, CLI, and embedded SDK consumers all hit the same gate. **Cluster-only boot** (RFC-011): the server always boots from a cluster directory (`--cluster <dir | s3://…>`, RFC-005) and serves N graphs (N ≥ 1) under multi-graph routes (`/graphs/{graph_id}/...` + read-only `GET /graphs` enumeration); there are no single-graph flat routes and no positional-URI boot. Per-graph + server-level Cedar policies. Runtime add/remove (`POST /graphs`, `DELETE /graphs/{id}`) is not exposed — operators run `cluster apply` and restart.
- **CLI** with two-surface config (RFC-007/008): the team-owned cluster directory (`cluster.yaml`) plus the per-operator `~/.omnigraph/config.yaml` (servers, clusters, credentials, actor, profiles, aliases, defaults). Graphs are addressed via `--store`/`--server`/`--cluster`/`--profile`/operator defaults (RFC-011). Multi-format output (json/jsonl/csv/kv/table).
@ -53,7 +53,7 @@ CLI (omnigraph) HTTP Server (omnigraph-server, Axum)
omnigraph (engine) ── ManifestCoordinator, CommitGraph, RunRegistry, GraphIndex (CSR/CSC), exec
│
▼
Lance 7.x ── columnar Arrow, fragments, per-dataset versions/branches, indexes
Lance 9.x ── columnar Arrow, fragments, per-dataset versions/branches, indexes
│
▼
Object store (file / s3 / RustFS / MinIO / S3-compat)
@ -253,11 +253,11 @@ omnigraph policy explain --cluster ./company-brain --graph knowledge --actor act
| Columnar storage on object store | ✅ Arrow/Lance | URI normalization, S3 env-var plumbing |
| Per-dataset versioning + time travel | ✅ | `snapshot_at_version`, `entity_at`, snapshot-pinned reads across many tables |
| Per-dataset branches | ✅ | **Graph-level** branches (atomic across all sub-tables), lazy fork, system branch filtering |
| Atomic single-dataset commits | ✅ | **Multi-table publish via three layers**, NOT a single Lance primitive: (1) per-table Lance `commit_staged` for the data write, (2) `__manifest` row-level CAS via `ManifestBatchPublisher` for cross-table ordering, (3) the open-time recovery sweep for the residual gap between (1) and (2). All three layers ship; the five migrated writers (`MutationStaging::finalize`, `schema_apply`, `branch_merge`, `ensure_indices`, `optimize_all_tables`) write a `__recovery/{ulid}.json` sidecar before Phase B and delete it after Phase C. The next `Omnigraph::open` (gated on `OpenMode::ReadWrite`) runs the sweep in `db/manifest/recovery.rs`: classify, decide all-or-nothing per sidecar, roll forward via single `ManifestBatchPublisher::publish` or roll back via `Dataset::restore` followed by a manifest publish of the restored version (so both directions converge to `manifest == HEAD` — no residual drift), and record an audit row in `_graph_commit_recoveries.lance` (queryable via `omnigraph commit list --filter actor=omnigraph:recovery`). The write entry points (`load_as`, `mutate_as`, `apply_schema_as`, `branch_merge_as`) and `refresh` additionally run an in-process roll-forward-only heal (serialized against live writers via the per-table write queues), so a long-lived server converges on its next write without restart; only rollback-eligible sidecars still defer to the next read-write open (a future background reconciler's goal). Engine writes route through a sealed `TableStorage` trait (`db.storage()`) exposing only `stage_*` + `commit_staged` + reads; the sole inline-commit residual (`create_vector_index`) is split onto a separate sealed `InlineCommitResidual` trait reached via `db.storage_inline_residual()` (MR-854), so the default surface cannot couple a write with a HEAD advance — §1 holds by construction. `delete` migrated to the staged path in MR-A (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, [#6658](https://github.com/lance-format/lance/issues/6658)); `create_vector_index` stays inline until upstream Lance ships a public two-phase API ([#6666](https://github.com/lance-format/lance/issues/6666)); `LoadMode::Overwrite` uses Lance `Overwrite` staged transactions. |
| Atomic single-dataset commits | ✅ | **Multi-table publish via three layers**, NOT a single Lance primitive: (1) per-table Lance `commit_staged` for the data write, (2) `__manifest` row-level CAS via `ManifestBatchPublisher` for cross-table ordering, (3) the open-time recovery sweep for the residual gap between (1) and (2). All three layers ship; the five migrated writers (`MutationStaging::commit_all`, `schema_apply`, `branch_merge`, `ensure_indices`, `optimize_all_tables`) write a `__recovery/{ulid}.json` sidecar before Phase B and delete it after Phase C. The next `Omnigraph::open` (gated on `OpenMode::ReadWrite`) runs the sweep in `db/manifest/recovery.rs`: classify, decide all-or-nothing per sidecar, roll forward via single `ManifestBatchPublisher::publish` or roll back via `Dataset::restore` followed by a manifest publish of the restored version (so both directions converge to `manifest == HEAD` — no residual drift), and record an internal audit row in `_graph_commit_recoveries.lance`. A v3 roll-forward preserves the interrupted writer's fixed commit lineage and actor; rollback and legacy recovery commits use `omnigraph:recovery`. There is currently no public CLI query for the recovery-audit table, and ordinary commit history is not a complete recovery enumeration. The write entry points (`load_as`, `mutate_as`, `apply_schema_as`, `branch_merge_as`) and `refresh` additionally run an in-process roll-forward-only heal, serialized against same-process live writers through the shared root-scoped gate manager, so a long-lived server converges on its next write without restart; only rollback-eligible sidecars still defer to the next read-write open (a future background reconciler's goal). Engine writes route through a sealed `TableStorage` trait (`db.storage()`) exposing only `stage_*` + `commit_staged` + reads; the sole inline-commit residual (`create_vector_index`) is split onto a separate sealed `InlineCommitResidual` trait reached via `db.storage_inline_residual()` (MR-854), so the default surface cannot couple a write with a HEAD advance — §1 holds by construction. `delete` migrated to the staged path in MR-A (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, [#6658](https://github.com/lance-format/lance/issues/6658)); `create_vector_index` stays inline until upstream Lance ships a public two-phase API ([#6666](https://github.com/lance-format/lance/issues/6666)); `LoadMode::Overwrite` uses Lance `Overwrite` staged transactions. |
| Compaction (`compact_files`) + reindex (`optimize_indices`) | ✅ | `omnigraph optimize` orchestrates over all node/edge tables, bounded concurrency; per table runs `compact_files` **then Lance `optimize_indices`** (folds appended/rewritten fragments back into existing indexes — incremental merge, not retrain) and **publishes the resulting version to `__manifest`** (so the manifest tracks the Lance HEAD — required for reads to observe the work and for schema apply / strict writes to pass their HEAD-vs-manifest precondition), under the per-`(table, main)` write queue with `SidecarKind::Optimize` recovery coverage spanning both ops; **commits even with no compaction work if index coverage is stale**; **refuses on an unrecovered graph**; **skips uncovered HEAD > manifest drift** with `DriftNeedsRepair`; **compacts blob-bearing tables** (the pre-9 `LANCE_SUPPORTS_BLOB_COMPACTION` skip was removed once Lance 8.0.0+ shipped blob-v2 compaction — see [docs/dev/invariants.md](docs/dev/invariants.md) Known Gaps) |
| Repair uncovered drift | — | `omnigraph repair` explicitly classifies uncovered table `HEAD > manifest` drift: verified maintenance drift (`ReserveFragments`/`Rewrite`) can be published with `--confirm`; suspicious or unverifiable drift requires `--force --confirm`. Sidecar-covered crash residuals still recover automatically on open. |
| Cleanup (`cleanup_old_versions`) | ✅ | `omnigraph cleanup` with `--keep` / `--older-than` policy |
| BTREE / inverted (FTS) / vector indexes | ✅ | `@index`/`@key` declares intent; the physical index is derived state that never fails a logical op. Built per column through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector); idempotent; lazy across branches. **Schema apply builds nothing** (records intent only); `load`/`mutate` build inline but **defer an untrainable Vector column** (no trainable vectors yet) as *pending* rather than aborting. `ensure_indices`/`optimize` is the reconciler that materializes declared-but-missing indexes and restores coverage of appended/rewritten fragments (`optimize_indices`), reporting still-pending columns (see Compaction row). |
| BTREE / inverted (FTS) / vector indexes | ✅ | `@index`/`@key` declares intent; the physical index is derived state that never fails a logical op. Built per column through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector); idempotent; lazy across branches. **Schema apply and mutation/load build no indexes inline**: the latter publish only their exact data effects, leaving physical intent pending. `ensure_indices`/`optimize` materializes declared-but-missing indexes, restores fragment coverage, and continues to report untrainable Vector columns as pending. |
| `merge_insert` upsert | ✅ | `LoadMode::Merge`, mutation `update`/`insert`/`delete` lowering |
| Vector search | ✅ | `nearest()` query op; embedding pipeline (Gemini / OpenAI clients); `@embed` in schema |
| Full-text search | ✅ | `search/fuzzy/match_text/bm25` query ops |
@ -267,7 +267,7 @@ omnigraph policy explain --cluster ./company-brain --graph knowledge --actor act
| Query language | — | `.gq` + Pest grammar + IR + lowering + linter |
| Schema migration planning | — | `plan_schema_migration` + `apply_schema` step types + `__schema_apply_lock__` |
| Commit graph (DAG) across whole graph | — | Lineage (linear + merge parents, ULID ids, actor) stored as `graph_commit`/`graph_head` rows in `__manifest`, written in the same publish CAS as the table-version rows (RFC-013 Phase 7 — atomic with the graph commit). The in-memory commit graph is a pure projection of those rows; the legacy `_graph_commits.lance` / `_graph_commit_actors.lance` tables are **retired** (a fresh graph creates neither) |
| Per-query atomic writes | — | In-memory `MutationStaging.pending` accumulator + `stage_*` / `commit_staged` per touched table at end-of-query + publisher CAS via `commit_with_expected` (single manifest commit per `mutate_as` / `load`); D₂ parse-time rule keeps inserts/updates and deletes from mixing |
| Per-query atomic writes | — | In-memory `MutationStaging.pending` accumulator + one exact staged Lance transaction per touched table + schema-v3 recovery confirmation + exact branch-head publisher CAS (single manifest commit per `mutate_as` / `load`); D₂ parse-time rule keeps inserts/updates and deletes from mixing |
| Three-way row-level merge | — | `OrderedTableCursor` + `StagedTableWriter`, structured `MergeConflictKind` |
| Change feeds | — | `diff_between` / `diff_commits` with manifest fast path + ID streaming |
| Cedar policy | — | Per-graph actions plus server-scoped actions (see [docs/user/operations/policy.md](docs/user/operations/policy.md) for the current list), branch / target_branch / protected scopes, validate/test/explain CLI. **Engine-wide enforcement** (MR-722): every `_as` writer (`apply_schema_as`, `mutate_as`, `load_as` — the deprecated `ingest_as` shims route through it — `branch_create_as` / `branch_create_from_as`, `branch_delete_as`, `branch_merge_as`) calls `Omnigraph::enforce(action, scope, actor)` — HTTP, CLI, embedded SDK all hit the same gate. |

22
crates/omnigraph-api-types/src/lib.rs
View file

@ -577,6 +577,21 @@ pub struct ManifestConflictOutput {
pub actual: u64,
}
/// Structured authority mismatch for an RFC-022 prepared write. Values are
/// strings because members include optional graph commit ids and future
/// authority tokens, not only numeric table versions.
#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
pub struct ReadSetConflictOutput {
pub member: String,
pub expected: Option<String>,
pub actual: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
pub struct RecoveryRequiredOutput {
pub operation_id: String,
}
#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
pub struct ErrorOutput {
pub error: String,
@ -590,6 +605,13 @@ pub struct ErrorOutput {
/// manifest is now at `actual`. Refresh and retry.
#[serde(skip_serializing_if = "Option::is_none")]
pub manifest_conflict: Option<ManifestConflictOutput>,
/// Set when a prepared write's logical authority changed before effects.
#[serde(skip_serializing_if = "Option::is_none")]
pub read_set_conflict: Option<ReadSetConflictOutput>,
/// Set when an overlapping durable recovery intent must be resolved before
/// retry. Its table effects may or may not have started.
#[serde(skip_serializing_if = "Option::is_none")]
pub recovery_required: Option<RecoveryRequiredOutput>,
}
pub fn snapshot_payload(

43
crates/omnigraph-server/src/handlers.rs
View file

@ -788,8 +788,9 @@ pub(crate) async fn run_query(
(status = 400, description = "Bad request", body = ErrorOutput),
(status = 401, description = "Unauthorized", body = ErrorOutput),
(status = 403, description = "Forbidden", body = ErrorOutput),
(status = 409, description = "Merge conflict", body = ErrorOutput),
(status = 409, description = "Write-authority conflict", body = ErrorOutput),
(status = 429, description = "Per-actor admission cap exceeded; honor `Retry-After` header", body = ErrorOutput),
(status = 503, description = "An overlapping durable recovery intent must be resolved before retry", body = ErrorOutput),
),
security(("bearer_token" = [])),
)]
@ -837,8 +838,9 @@ pub(crate) async fn server_change(
(status = 400, description = "Bad request", body = ErrorOutput),
(status = 401, description = "Unauthorized", body = ErrorOutput),
(status = 403, description = "Forbidden", body = ErrorOutput),
(status = 409, description = "Merge conflict", body = ErrorOutput),
(status = 409, description = "Write-authority conflict", body = ErrorOutput),
(status = 429, description = "Per-actor admission cap exceeded; honor `Retry-After` header", body = ErrorOutput),
(status = 503, description = "An overlapping durable recovery intent must be resolved before retry", body = ErrorOutput),
),
security(("bearer_token" = [])),
)]
@ -847,7 +849,8 @@ pub(crate) async fn server_change(
/// Writes to the named `branch` (defaults to `main`). Mutations are atomic
/// per call and produce a new commit. Returns counts of nodes and edges
/// affected. **Destructive**: on success the branch is updated; rejected
/// mutations may still acquire locks briefly. Returns 409 on merge conflict.
/// mutations may still acquire locks briefly. Returns 409 when the prepared
/// write authority changes before effects.
///
/// Pairs with `POST /query` (read-only). The legacy `POST /change` route
/// has identical semantics and is kept as a deprecated alias.
@ -904,9 +907,10 @@ pub(crate) fn parse_optional_invoke_body(
(status = 401, description = "Unauthorized", body = ErrorOutput),
(status = 403, description = "Forbidden (the inner `change` gate for a stored mutation)", body = ErrorOutput),
(status = 404, description = "Unknown stored query, or `invoke_query` denied — indistinguishable to a caller without the grant", body = ErrorOutput),
(status = 409, description = "Merge conflict", body = ErrorOutput),
(status = 409, description = "Stored mutation write-authority conflict", body = ErrorOutput),
(status = 429, description = "Per-actor admission cap exceeded; honor `Retry-After` header", body = ErrorOutput),
(status = 500, description = "Policy evaluation error (a denial is reported as 404, not 500)", body = ErrorOutput),
(status = 503, description = "A stored mutation is blocked by a durable recovery intent", body = ErrorOutput),
),
security(("bearer_token" = [])),
)]
@ -1203,25 +1207,11 @@ pub(crate) async fn server_schema_apply(
.await
.map_err(ApiError::from_omni)?
};
// Prompt index convergence (iss-848): schema apply records `@index` intent
// but defers the physical build. On a long-lived server, materialize it
// promptly rather than waiting for the next `optimize` cron — spawned
// detached so it never blocks or fails the apply response. Best-effort: a
// failure is logged and the index still converges on the next optimize.
// The CLI is one-shot, so it has no equivalent; its convergence path is the
// operator's optimize cadence.
if result.applied {
let engine = Arc::clone(&handle.engine);
tokio::spawn(async move {
if let Err(err) = engine.ensure_indices().await {
tracing::warn!(
target: "omnigraph::server",
error = %err,
"post-apply ensure_indices failed; indexes will converge on the next optimize",
);
}
});
}
// Physical indexes are derived state. Schema apply records intent only;
// explicit `ensure_indices` / `optimize` maintenance owns convergence on
// every surface, including a long-lived server. Keeping the handler free
// of detached physical writes also makes a successful response describe
// the complete effect envelope of this request.
Ok(Json(schema_apply_output(handle.uri.as_str(), result)))
}
@ -1313,7 +1303,9 @@ async fn run_ingest(
(status = 400, description = "Bad request", body = ErrorOutput),
(status = 401, description = "Unauthorized", body = ErrorOutput),
(status = 403, description = "Forbidden", body = ErrorOutput),
(status = 409, description = "Prepared load authority changed before effects", body = ErrorOutput),
(status = 429, description = "Per-actor admission cap exceeded; honor `Retry-After` header", body = ErrorOutput),
(status = 503, description = "An overlapping durable recovery intent must be resolved before retry", body = ErrorOutput),
),
security(("bearer_token" = [])),
)]
@ -1357,7 +1349,9 @@ pub(crate) async fn server_load(
(status = 400, description = "Bad request", body = ErrorOutput),
(status = 401, description = "Unauthorized", body = ErrorOutput),
(status = 403, description = "Forbidden", body = ErrorOutput),
(status = 409, description = "Prepared load authority changed before effects", body = ErrorOutput),
(status = 429, description = "Per-actor admission cap exceeded; honor `Retry-After` header", body = ErrorOutput),
(status = 503, description = "An overlapping durable recovery intent must be resolved before retry", body = ErrorOutput),
),
security(("bearer_token" = [])),
)]
@ -1437,6 +1431,7 @@ pub(crate) async fn server_branch_list(
(status = 403, description = "Forbidden", body = ErrorOutput),
(status = 409, description = "Branch already exists", body = ErrorOutput),
(status = 429, description = "Per-actor admission cap exceeded; honor `Retry-After` header", body = ErrorOutput),
(status = 503, description = "An overlapping durable recovery intent must be resolved before retry", body = ErrorOutput),
),
security(("bearer_token" = [])),
)]
@ -1518,6 +1513,7 @@ pub(crate) struct BranchPath {
(status = 403, description = "Forbidden", body = ErrorOutput),
(status = 404, description = "Branch not found", body = ErrorOutput),
(status = 429, description = "Per-actor admission cap exceeded; honor `Retry-After` header", body = ErrorOutput),
(status = 503, description = "An overlapping durable recovery intent must be resolved before retry", body = ErrorOutput),
),
security(("bearer_token" = [])),
)]
@ -1579,6 +1575,7 @@ pub(crate) async fn server_branch_delete(
(status = 403, description = "Forbidden", body = ErrorOutput),
(status = 409, description = "Merge conflict", body = ErrorOutput),
(status = 429, description = "Per-actor admission cap exceeded; honor `Retry-After` header", body = ErrorOutput),
(status = 503, description = "An overlapping durable recovery intent must be resolved before retry", body = ErrorOutput),
),
security(("bearer_token" = [])),
)]

121
crates/omnigraph-server/src/lib.rs
View file

@ -286,10 +286,12 @@ impl Write for ExportStreamWriter {
#[derive(Debug)]
pub struct ApiError {
status: StatusCode,
code: ErrorCode,
code: Option<ErrorCode>,
message: String,
merge_conflicts: Vec<api::MergeConflictOutput>,
manifest_conflict: Option<api::ManifestConflictOutput>,
read_set_conflict: Option<api::ReadSetConflictOutput>,
recovery_required: Option<api::RecoveryRequiredOutput>,
}
impl AppState {
@ -616,40 +618,48 @@ impl ApiError {
pub fn unauthorized(message: impl Into<String>) -> Self {
Self {
status: StatusCode::UNAUTHORIZED,
code: ErrorCode::Unauthorized,
code: Some(ErrorCode::Unauthorized),
message: message.into(),
merge_conflicts: Vec::new(),
manifest_conflict: None,
read_set_conflict: None,
recovery_required: None,
}
}
pub fn forbidden(message: impl Into<String>) -> Self {
Self {
status: StatusCode::FORBIDDEN,
code: ErrorCode::Forbidden,
code: Some(ErrorCode::Forbidden),
message: message.into(),
merge_conflicts: Vec::new(),
manifest_conflict: None,
read_set_conflict: None,
recovery_required: None,
}
}
pub fn bad_request(message: impl Into<String>) -> Self {
Self {
status: StatusCode::BAD_REQUEST,
code: ErrorCode::BadRequest,
code: Some(ErrorCode::BadRequest),
message: message.into(),
merge_conflicts: Vec::new(),
manifest_conflict: None,
read_set_conflict: None,
recovery_required: None,
}
}
pub fn not_found(message: impl Into<String>) -> Self {
Self {
status: StatusCode::NOT_FOUND,
code: ErrorCode::NotFound,
code: Some(ErrorCode::NotFound),
message: message.into(),
merge_conflicts: Vec::new(),
manifest_conflict: None,
read_set_conflict: None,
recovery_required: None,
}
}
@ -660,30 +670,36 @@ impl ApiError {
pub fn method_not_allowed(message: impl Into<String>) -> Self {
Self {
status: StatusCode::METHOD_NOT_ALLOWED,
code: ErrorCode::MethodNotAllowed,
code: Some(ErrorCode::MethodNotAllowed),
message: message.into(),
merge_conflicts: Vec::new(),
manifest_conflict: None,
read_set_conflict: None,
recovery_required: None,
}
}
pub fn conflict(message: impl Into<String>) -> Self {
Self {
status: StatusCode::CONFLICT,
code: ErrorCode::Conflict,
code: Some(ErrorCode::Conflict),
message: message.into(),
merge_conflicts: Vec::new(),
manifest_conflict: None,
read_set_conflict: None,
recovery_required: None,
}
}
pub fn internal(message: impl Into<String>) -> Self {
Self {
status: StatusCode::INTERNAL_SERVER_ERROR,
code: ErrorCode::Internal,
code: Some(ErrorCode::Internal),
message: message.into(),
merge_conflicts: Vec::new(),
manifest_conflict: None,
read_set_conflict: None,
recovery_required: None,
}
}
@ -694,10 +710,12 @@ impl ApiError {
pub fn too_many_requests(message: impl Into<String>) -> Self {
Self {
status: StatusCode::TOO_MANY_REQUESTS,
code: ErrorCode::TooManyRequests,
code: Some(ErrorCode::TooManyRequests),
message: message.into(),
merge_conflicts: Vec::new(),
manifest_conflict: None,
read_set_conflict: None,
recovery_required: None,
}
}
@ -715,20 +733,51 @@ impl ApiError {
fn merge_conflict(conflicts: Vec<api::MergeConflictOutput>) -> Self {
Self {
status: StatusCode::CONFLICT,
code: ErrorCode::Conflict,
code: Some(ErrorCode::Conflict),
message: summarize_merge_conflicts(&conflicts),
merge_conflicts: conflicts,
manifest_conflict: None,
read_set_conflict: None,
recovery_required: None,
}
}
fn manifest_version_conflict(message: String, details: api::ManifestConflictOutput) -> Self {
Self {
status: StatusCode::CONFLICT,
code: ErrorCode::Conflict,
code: Some(ErrorCode::Conflict),
message,
merge_conflicts: Vec::new(),
manifest_conflict: Some(details),
read_set_conflict: None,
recovery_required: None,
}
}
fn read_set_conflict(message: String, details: api::ReadSetConflictOutput) -> Self {
Self {
status: StatusCode::CONFLICT,
code: Some(ErrorCode::Conflict),
message,
merge_conflicts: Vec::new(),
manifest_conflict: None,
read_set_conflict: Some(details),
recovery_required: None,
}
}
fn recovery_required(message: String, operation_id: String) -> Self {
Self {
status: StatusCode::SERVICE_UNAVAILABLE,
// `ErrorCode` is a closed rolling wire contract. The additive
// `recovery_required` field carries the new meaning while older
// clients continue to deserialize the otherwise familiar body.
code: None,
message,
merge_conflicts: Vec::new(),
manifest_conflict: None,
read_set_conflict: None,
recovery_required: Some(api::RecoveryRequiredOutput { operation_id }),
}
}
@ -752,6 +801,18 @@ impl ApiError {
actual,
},
),
Some(ManifestConflictDetails::ReadSetChanged {
member,
expected,
actual,
}) => Self::read_set_conflict(
err.message,
api::ReadSetConflictOutput {
member,
expected,
actual,
},
),
_ => Self::conflict(err.message),
},
ManifestErrorKind::Internal => Self::internal(err.message),
@ -762,6 +823,13 @@ impl ApiError {
.map(api::MergeConflictOutput::from)
.collect(),
),
OmniError::RecoveryRequired {
operation_id,
reason,
} => Self::recovery_required(
format!("recovery required for operation {operation_id}: {reason}"),
operation_id,
),
OmniError::Lance(message) => Self::internal(format!("storage: {message}")),
OmniError::Io(err) => Self::internal(format!("io: {err}")),
// Engine-layer policy enforcement (MR-722). All denials and
@ -815,7 +883,7 @@ const RETRY_AFTER_SECONDS: &str = "60";
impl IntoResponse for ApiError {
fn into_response(self) -> Response {
let mut headers = axum::http::HeaderMap::new();
if matches!(self.code, ErrorCode::TooManyRequests) {
if matches!(self.code, Some(ErrorCode::TooManyRequests)) {
headers.insert(
axum::http::header::RETRY_AFTER,
axum::http::HeaderValue::from_static(RETRY_AFTER_SECONDS),
@ -826,15 +894,42 @@ impl IntoResponse for ApiError {
headers,
Json(ErrorOutput {
error: self.message,
code: Some(self.code),
code: self.code,
merge_conflicts: self.merge_conflicts,
manifest_conflict: self.manifest_conflict,
read_set_conflict: self.read_set_conflict,
recovery_required: self.recovery_required,
}),
)
.into_response()
}
}
#[cfg(test)]
mod api_error_tests {
use super::*;
#[tokio::test]
async fn recovery_required_503_omits_closed_error_code() {
let response = ApiError::from_omni(OmniError::RecoveryRequired {
operation_id: "01JTESTRECOVERY".to_string(),
reason: "pending recovery intent".to_string(),
})
.into_response();
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
let body = axum::body::to_bytes(response.into_body(), usize::MAX)
.await
.unwrap();
let error: ErrorOutput = serde_json::from_slice(&body).unwrap();
assert_eq!(error.code, None);
assert_eq!(
error.recovery_required.unwrap().operation_id,
"01JTESTRECOVERY"
);
}
}
pub fn init_tracing() {
let filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info"));
let _ = tracing_subscriber::fmt().with_env_filter(filter).try_init();

116
crates/omnigraph-server/tests/data_routes.rs
View file

@ -1099,18 +1099,15 @@ query vector_search_string($q: String) {
}
#[tokio::test(flavor = "multi_thread")]
async fn change_conflict_returns_manifest_conflict_409() {
// A write that races with another writer surfaces as HTTP 409 with
// a structured `manifest_conflict` body — `table_key`, `expected`,
// and `actual` — so clients can detect-and-retry without parsing
// the message.
async fn change_long_lived_handle_refreshes_before_preparing_write() {
// A handle that merely predates another committed write is not stale
// authority: open_write_txn probes the manifest incarnation and prepares
// from the fresh head. ReadSetChanged is reserved for movement *during* an
// already-prepared attempt (covered by the concurrent test below).
let temp = init_loaded_graph().await;
let graph = graph_path(temp.path());
// Build the server first so its handle pins the pre-mutation manifest
// version. Then advance the manifest from outside the server. The
// server's next /change call will capture stale `expected_versions`
// (from its still-pinned snapshot) and the publisher's CAS rejects.
// Build the server first, then advance the graph through another handle.
let state = AppState::open(graph.to_string_lossy().to_string())
.await
.unwrap();
@ -1154,34 +1151,17 @@ async fn change_conflict_returns_manifest_conflict_409() {
)
.await;
assert_eq!(status, StatusCode::CONFLICT);
let error: ErrorOutput = serde_json::from_value(body).unwrap();
assert_eq!(error.code, Some(omnigraph_server::api::ErrorCode::Conflict));
let conflict = error
.manifest_conflict
.expect("publisher CAS rejection must populate manifest_conflict body");
assert_eq!(conflict.table_key, "node:Person");
assert!(
conflict.actual > conflict.expected,
"actual ({}) should be ahead of expected ({})",
conflict.actual,
conflict.expected,
);
assert_eq!(status, StatusCode::OK);
assert_eq!(body["affected_nodes"], 1);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn change_concurrent_inserts_same_key_serialize_without_409() {
// PR 2 Phase 2 (MR-686): pin the design fix for the same-key
// concurrency hazard. Pre-fix, in-process concurrent inserts on
// the same `(table, branch)` rejected with 409 manifest_conflict
// because `ensure_expected_version` fired before the per-table
// queue was acquired and saw Lance HEAD already advanced by a
// peer writer. Post-fix, Insert/Merge skip the strict pre-stage
// check (see `MutationOpKind::strict_pre_stage_version_check`);
// the queue serializes commit_staged; Lance's natural rebase
// handles the in-flight stage; the publisher's CAS on a fresh
// per-branch snapshot under the queue catches genuine cross-
// process drift.
// RFC-022 preservation guard: concurrent retryable inserts still all
// succeed, but not by rebasing an already-validated Lance transaction.
// The coarse branch gate serializes effects; a waiter whose authority
// token changed discards its complete attempt and reprepares from the
// winner's committed branch state.
//
// This test spawns N concurrent /change inserts on a single
// node type and asserts: every request returns 200 (no 409),
@ -1270,36 +1250,10 @@ async fn change_concurrent_inserts_same_key_serialize_without_409() {
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn change_concurrent_updates_same_key_serialize_via_publisher_cas() {
// Pin Update RYW semantics under in-process concurrency on the same
// `(table, branch)`. With per-table queue serialization and op-kind-aware
// drift detection at commit time, exactly one of N concurrent UPDATEs
// on the same row commits; the rest are rejected as 409 manifest_conflict.
//
// Pre-fix bug class: in `MutationStaging::commit_all`, after queue
// acquisition, the staged Lance transaction is handed straight to
// `commit_staged`. For a writer whose staged dataset is at V0 but
// Lance HEAD has advanced to V1 (because the queue's prior winner
// already published), Lance's transaction conflict resolver fires
// `RetryableCommitConflict` on Update vs Update on the same row.
// That error gets wrapped as `OmniError::Lance(<string>)` and the
// API surfaces it as **500 internal**, not 409. Users see "internal
// server error" instead of a retryable conflict, breaking the
// documented 409 contract for in-process drift.
//
// Post-fix invariant: `commit_all` does an op-kind-aware drift check
// before each `commit_staged`. For tables whose tracked op_kind has
// `strict_pre_stage_version_check() == true` (Update / Delete /
// SchemaRewrite), if the staged dataset's version doesn't match the
// fresh manifest pin, return `OmniError::manifest_expected_version_mismatch`
// → 409 ExpectedVersionMismatch. The N-1 losers see a clean 409
// before Lance's commit_staged ever runs.
//
// Why correct-by-design: closing the class "Lance internal conflict
// surfaces as 500 instead of 409" rather than mapping the specific
// Lance error variant. The drift check fires at the right architectural
// layer (engine boundary, under the queue) and respects the existing
// `MutationOpKind` policy.
async fn change_concurrent_updates_same_key_return_typed_pre_effect_conflicts() {
// Strict read-modify-write attempts are never automatically reprepared.
// Exactly one concurrent UPDATE commits; once it changes branch authority,
// every waiter reports a typed 409 before any of its Lance effects begin.
let temp = init_loaded_graph().await;
let graph = graph_path(temp.path());
let state = AppState::open(graph.to_string_lossy().to_string())
@ -1373,27 +1327,37 @@ async fn change_concurrent_updates_same_key_serialize_via_publisher_cas() {
assert_eq!(
ok_count,
1,
"expected exactly one update to commit and N-1 to receive 409 manifest_conflict \
(op-kind-aware drift check rejects stale-V0 staged datasets at commit_all entry). \
Got {} OK + {} 409 + {} other. \
Pre-fix symptom: 1 OK + (N-1) x 500 because Lance's RetryableCommitConflict for \
Update vs Update on the same row bubbles up as `OmniError::Lance(<string>)` and \
the API maps it to 500 internal, not 409. Statuses: {:?}",
"expected exactly one update to commit and N-1 to receive typed 409 conflicts \
before effects. Got {} OK + {} 409 + {} other. Statuses: {:?}",
ok_count,
conflict_count,
statuses.len() - ok_count - conflict_count,
statuses,
);
for (status, bytes) in &results {
if *status != StatusCode::CONFLICT {
continue;
}
let error: ErrorOutput = serde_json::from_slice(bytes).unwrap();
assert_eq!(error.code, Some(omnigraph_server::api::ErrorCode::Conflict));
let conflict = error
.read_set_conflict
.expect("strict OCC loser must include structured read-set authority");
assert_eq!(conflict.member, "graph_head:main");
assert_ne!(conflict.actual, conflict.expected);
assert!(error.manifest_conflict.is_none());
assert!(error.recovery_required.is_none());
}
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn change_disjoint_table_concurrency_succeeds_at_http_level() {
// HTTP-level pin for MR-686's disjoint-table promise: concurrent /change
// requests touching different node types must coexist without admission
// rejection or publisher-CAS conflict. The bench harness measures
// throughput; this test is the regression sentinel that catches a
// future change which accidentally re-introduces graph-wide
// serialization on the disjoint path.
async fn change_disjoint_table_concurrency_succeeds_under_branch_occ_gate() {
// RFC-022 intentionally serializes effect publication per branch because
// graph-head authority protects validation dependencies across tables.
// Disjoint retryable inserts must nevertheless all succeed through bounded
// full-attempt repreparation, without admission rejection or a user-visible
// publisher conflict.
//
// Setup: test.jsonl seeds 4 Persons + 2 Companies. Spawn N=4 concurrent
// /change inserts on `node:Person` and N=4 concurrent inserts on

69
crates/omnigraph-server/tests/openapi.rs
View file

@ -391,7 +391,9 @@ const EXPECTED_SCHEMAS: &[&str] = &[
"MergeConflictOutput",
"ReadOutput",
"ReadRequest",
"ReadSetConflictOutput",
"ReadTargetOutput",
"RecoveryRequiredOutput",
"ManifestConflictOutput",
"SchemaApplyOutput",
"SchemaApplyRequest",
@ -614,6 +616,8 @@ fn error_output_schema_has_expected_fields() {
assert!(props.contains_key("code"));
assert!(props.contains_key("merge_conflicts"));
assert!(props.contains_key("manifest_conflict"));
assert!(props.contains_key("read_set_conflict"));
assert!(props.contains_key("recovery_required"));
}
#[test]
@ -626,6 +630,24 @@ fn manifest_conflict_output_schema_has_expected_fields() {
assert!(props.contains_key("actual"));
}
#[test]
fn read_set_conflict_output_schema_has_expected_fields() {
let doc = openapi_json();
let schema = &doc["components"]["schemas"]["ReadSetConflictOutput"];
let props = schema["properties"].as_object().unwrap();
assert!(props.contains_key("member"));
assert!(props.contains_key("expected"));
assert!(props.contains_key("actual"));
}
#[test]
fn recovery_required_output_schema_has_expected_fields() {
let doc = openapi_json();
let schema = &doc["components"]["schemas"]["RecoveryRequiredOutput"];
let props = schema["properties"].as_object().unwrap();
assert!(props.contains_key("operation_id"));
}
#[test]
fn commit_output_schema_has_expected_fields() {
let doc = openapi_json();
@ -694,12 +716,21 @@ fn error_code_schema_has_expected_variants() {
let schema = &doc["components"]["schemas"]["ErrorCode"];
let variants = schema["enum"].as_array().unwrap();
let values: HashSet<&str> = variants.iter().map(|v| v.as_str().unwrap()).collect();
assert!(values.contains("unauthorized"));
assert!(values.contains("forbidden"));
assert!(values.contains("bad_request"));
assert!(values.contains("not_found"));
assert!(values.contains("conflict"));
assert!(values.contains("internal"));
assert_eq!(
values,
HashSet::from([
"unauthorized",
"forbidden",
"bad_request",
"not_found",
"method_not_allowed",
"conflict",
"too_many_requests",
"internal",
]),
"ErrorCode is a rolling wire contract: new meanings belong in optional \
structured fields, not new closed-enum values",
);
}
#[test]
@ -919,6 +950,32 @@ fn error_responses_reference_error_output_schema() {
}
}
#[test]
fn recovery_barrier_write_endpoints_document_recovery_required() {
let doc = openapi_json();
for (path, method) in [
("/graphs/{graph_id}/change", "post"),
("/graphs/{graph_id}/mutate", "post"),
("/graphs/{graph_id}/queries/{name}", "post"),
("/graphs/{graph_id}/load", "post"),
("/graphs/{graph_id}/ingest", "post"),
("/graphs/{graph_id}/branches", "post"),
("/graphs/{graph_id}/branches/{branch}", "delete"),
("/graphs/{graph_id}/branches/merge", "post"),
] {
let response = &doc["paths"][path][method]["responses"]["503"];
assert!(
response.is_object(),
"{method} {path} must document the recovery-required 503 outcome"
);
assert_eq!(
response["content"]["application/json"]["schema"]["$ref"],
"#/components/schemas/ErrorOutput",
"{method} {path} 503 must use ErrorOutput"
);
}
}
// ---------------------------------------------------------------------------
// Request body reference tests
// ---------------------------------------------------------------------------

28
crates/omnigraph-server/tests/schema_routes.rs
View file

@ -371,8 +371,7 @@ async fn schema_apply_route_can_rename_property() {
#[tokio::test]
async fn schema_apply_route_can_add_index() {
let (temp, app) = app_for_graph_with_auth_tokens_and_policy(
&fs::read_to_string(fixture("test.pg")).unwrap(),
let (temp, app) = app_for_loaded_graph_with_auth_tokens_and_policy(
&[("act-ragnor", "admin-token")],
SCHEMA_APPLY_POLICY_YAML,
)
@ -392,7 +391,9 @@ async fn schema_apply_route_can_add_index() {
.header("authorization", "Bearer admin-token")
.body(Body::from(
serde_json::to_vec(&SchemaApplyRequest {
schema_source: indexed_name_schema(),
schema_source: fs::read_to_string(fixture("test.pg"))
.unwrap()
.replace("age: I32?", "age: I32? @index"),
..Default::default()
})
.unwrap(),
@ -404,9 +405,24 @@ async fn schema_apply_route_can_add_index() {
assert_eq!(payload["applied"], true);
// iss-848: the /schema/apply route accepts the index-add and applies it as a
// metadata change — it records the `@index` intent in the catalog/IR but does
// NOT build the physical index inline (the build is deferred to
// ensure_indices/optimize; on this empty table nothing would build anyway).
// So the physical index count is unchanged by the apply.
// NOT build the physical index inline or spawn a server-only build (the
// build is deferred to explicit ensure_indices/optimize maintenance).
// Pin the detached-work half structurally instead of waiting an arbitrary
// amount of wall-clock time for a task that must not exist. The physical
// assertion below independently covers synchronous/awaited index builds.
let handlers = include_str!("../src/handlers.rs");
let (_, after_apply_signature) = handlers
.split_once("pub(crate) async fn server_schema_apply")
.expect("server_schema_apply handler must remain present");
let (apply_handler, _) = after_apply_signature
.split_once("/// Shared body for `POST /load`")
.expect("schema apply handler must precede the load handler");
assert!(
!apply_handler.contains("tokio::spawn") && !apply_handler.contains(".ensure_indices("),
"schema apply must not schedule implicit physical index convergence"
);
// The physical index count is unchanged by the apply.
let reopened = Omnigraph::open(graph.to_str().unwrap()).await.unwrap();
let snapshot = reopened
.snapshot_of(ReadTarget::branch("main"))

6
crates/omnigraph-server/tests/support/mod.rs
View file

@ -502,12 +502,6 @@ pub fn renamed_age_schema() -> String {
.replace("age: I32?", "years: I32? @rename_from(\"age\")")
}
pub fn indexed_name_schema() -> String {
fs::read_to_string(fixture("test.pg"))
.unwrap()
.replace("name: String @key", "name: String @key @index")
}
pub fn unsupported_schema_change() -> String {
fs::read_to_string(fixture("test.pg"))
.unwrap()

10
crates/omnigraph/src/db/commit_graph.rs
View file

@ -49,6 +49,11 @@ impl CommitGraph {
/// keeps the cache consistent for same-handle reads, with no storage I/O.
/// Head selection matches the manifest-sourced load (`should_replace_head`).
pub fn insert_committed(&mut self, commit: GraphCommit) {
debug_assert_eq!(
commit.manifest_branch.as_deref(),
self.active_branch.as_deref(),
"published lineage must target the commit graph's active branch"
);
if should_replace_head(self.head_commit.as_ref(), &commit) {
self.head_commit = Some(commit.clone());
}
@ -71,7 +76,8 @@ impl CommitGraph {
let root = root_uri.trim_end_matches('/');
// `load_commit_cache_for_branch` opens the branch's `__manifest` (the
// authoritative table), so a truly absent branch fails loudly here.
let (commit_by_id, head_commit) = load_commit_cache_for_branch(root, Some(branch)).await?;
let (commit_by_id, head_commit) =
load_commit_cache_for_branch(root, Some(branch)).await?;
Ok(Self {
root_uri: root.to_string(),
active_branch: Some(branch.to_string()),
@ -180,7 +186,7 @@ async fn load_commit_cache_from_manifest(
root_uri: &str,
branch: Option<&str>,
) -> Result<(HashMap<String, GraphCommit>, Option<GraphCommit>)> {
let (rows, _heads) =
let (rows, _) =
crate::db::manifest::ManifestCoordinator::read_graph_lineage_at(root_uri, branch).await?;
let mut commit_by_id = HashMap::with_capacity(rows.len());
let mut head_commit = None;

50
crates/omnigraph/src/db/graph_coordinator.rs
View file

@ -11,7 +11,8 @@ use crate::storage::{StorageAdapter, normalize_root_uri};
use super::commit_graph::{CommitGraph, GraphCommit};
use super::is_internal_system_branch;
use super::manifest::{
ManifestChange, ManifestCoordinator, ManifestIncarnation, Snapshot, SubTableUpdate,
LineageIntent, ManifestChange, ManifestCoordinator, ManifestIncarnation, PublishPrecondition,
Snapshot, SubTableUpdate,
};
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
@ -167,6 +168,20 @@ impl GraphCoordinator {
self.manifest.incarnation()
}
/// Lance-native identity of the active `__manifest` branch. Stable across
/// commits; changes when a named branch is deleted and recreated.
pub(crate) async fn branch_identifier(&self) -> Result<lance::dataset::refs::BranchIdentifier> {
self.manifest.branch_identifier().await
}
/// Exact `graph_head:<active-branch>` pointer, preserving `None` for a
/// freshly-created named branch even though its inherited commit history has
/// an inferred head. Sourced from the manifest coordinator's SAME pinned
/// state as [`Self::snapshot`], not the separately refreshed lineage cache.
pub(crate) fn exact_graph_head(&self) -> Option<String> {
self.manifest.exact_graph_head()
}
pub fn snapshot(&self) -> Snapshot {
self.manifest.snapshot()
}
@ -393,10 +408,35 @@ impl GraphCoordinator {
actor_id: Option<&str>,
) -> Result<PublishedSnapshot> {
let intent = self.new_lineage_intent(actor_id, None)?;
self.commit_changes_with_intent_and_expected(
changes,
expected_table_versions,
intent,
&PublishPrecondition::Any,
)
.await
}
/// Publish a pre-minted lineage intent under an explicit authority
/// precondition. The intent's identity and timestamp remain stable across
/// publisher retries and can also be persisted by the caller's recovery
/// protocol before this method is invoked.
pub(crate) async fn commit_changes_with_intent_and_expected(
&mut self,
changes: &[ManifestChange],
expected_table_versions: &HashMap<String, u64>,
intent: LineageIntent,
precondition: &PublishPrecondition,
) -> Result<PublishedSnapshot> {
failpoints::maybe_fail(crate::failpoints::names::GRAPH_PUBLISH_BEFORE_COMMIT_APPEND)?;
let outcome = self
.manifest
.commit_changes_with_lineage(changes, expected_table_versions, Some(&intent))
.commit_changes_with_lineage_and_precondition(
changes,
expected_table_versions,
Some(&intent),
precondition,
)
.await?;
failpoints::maybe_fail(crate::failpoints::names::GRAPH_PUBLISH_AFTER_MANIFEST_COMMIT)?;
let snapshot_id = self.apply_lineage_to_cache(intent, &outcome);
@ -434,12 +474,12 @@ impl GraphCoordinator {
/// fresh ULID (stable across the publisher's CAS retries) and a timestamp.
/// The parent is NOT chosen here — the publisher resolves it per attempt
/// against the manifest it commits against.
fn new_lineage_intent(
pub(crate) fn new_lineage_intent(
&self,
actor_id: Option<&str>,
merged_parent_commit_id: Option<String>,
) -> Result<crate::db::manifest::LineageIntent> {
Ok(crate::db::manifest::LineageIntent {
) -> Result<LineageIntent> {
Ok(LineageIntent {
graph_commit_id: ulid::Ulid::new().to_string(),
branch: self.current_branch().map(str::to_string),
actor_id: actor_id.map(str::to_string),

64
crates/omnigraph/src/db/manifest.rs
View file

@ -28,25 +28,26 @@ mod recovery;
mod state;
use graph::{init_manifest_graph, open_manifest_graph, snapshot_state_at};
use layout::{open_manifest_dataset, table_uri_for_path, type_name_hash};
pub(crate) use layout::manifest_uri;
use layout::{open_manifest_dataset, table_uri_for_path, type_name_hash};
pub(crate) use metadata::TableVersionMetadata;
#[cfg(test)]
use metadata::{OMNIGRAPH_ROW_COUNT_KEY, table_version_metadata_for_state};
#[cfg(test)]
use namespace::{branch_manifest_namespace, staged_table_namespace};
pub(crate) use publisher::LineageIntent;
pub(crate) use publisher::{GraphHeadExpectation, LineageIntent, PublishPrecondition};
use publisher::{GraphNamespacePublisher, ManifestBatchPublisher, PublishOutcome};
pub(crate) use recovery::{
RecoveryMode, RecoverySidecar, RecoverySidecarHandle, SidecarKind, SidecarTablePin,
SidecarTableRegistration, SidecarTombstone, confirm_sidecar_phase_b, delete_sidecar,
has_schema_apply_sidecar, heal_pending_sidecars_roll_forward, list_sidecars, new_sidecar,
recover_manifest_drift, schema_apply_serial_queue_key, write_sidecar,
HealPendingOutcome, RecoveryAuthorityToken, RecoveryLineageIntent, RecoveryMode,
RecoverySidecar, RecoverySidecarHandle, SidecarKind, SidecarTablePin, SidecarTableRegistration,
SidecarTombstone, confirm_occ_sidecar_phase_b, confirm_sidecar_phase_b, delete_sidecar,
has_schema_apply_sidecar, heal_pending_sidecars_roll_forward, list_sidecars, new_occ_sidecar,
new_sidecar, recover_manifest_drift, schema_apply_serial_queue_key, write_sidecar,
};
pub use state::SubTableEntry;
pub(crate) use state::{GraphLineageRow, read_graph_lineage};
#[cfg(test)]
use state::string_column;
pub(crate) use state::{GraphLineageRow, read_graph_lineage};
use state::{ManifestState, read_manifest_state};
/// The internal-schema (storage-format) version this binary writes and reads.
@ -498,7 +499,30 @@ impl ManifestCoordinator {
expected_table_versions: &HashMap<String, u64>,
lineage: Option<&LineageIntent>,
) -> Result<CommitOutcome> {
if changes.is_empty() && expected_table_versions.is_empty() && lineage.is_none() {
self.commit_changes_with_lineage_and_precondition(
changes,
expected_table_versions,
lineage,
&PublishPrecondition::Any,
)
.await
}
/// The token-aware form of [`commit_changes_with_lineage`]. Exact authority
/// is checked by the publisher from every CAS attempt's existing one-scan
/// state; legacy callers continue through `Any` above.
pub(crate) async fn commit_changes_with_lineage_and_precondition(
&mut self,
changes: &[ManifestChange],
expected_table_versions: &HashMap<String, u64>,
lineage: Option<&LineageIntent>,
precondition: &PublishPrecondition,
) -> Result<CommitOutcome> {
if changes.is_empty()
&& expected_table_versions.is_empty()
&& lineage.is_none()
&& matches!(precondition, PublishPrecondition::Any)
{
return Ok(CommitOutcome {
version: self.version(),
parent_commit_id: None,
@ -511,7 +535,7 @@ impl ManifestCoordinator {
known_state,
} = self
.publisher
.publish(changes, expected_table_versions, lineage)
.publish_with_precondition(changes, expected_table_versions, lineage, precondition)
.await?;
// RFC-013 PR2 #1b: the publisher folded the new visible state in-memory
// (byte-identical to a re-scan via the shared `assemble_manifest_state`),
@ -550,6 +574,28 @@ impl ManifestCoordinator {
.map_err(|e| OmniError::Lance(e.to_string()))
}
/// Lance-native stable identity for the active manifest branch. Unlike a
/// manifest version/eTag, this remains stable across ordinary commits and
/// changes when a named branch is deleted and recreated (ABA protection).
pub(crate) async fn branch_identifier(&self) -> Result<lance::dataset::refs::BranchIdentifier> {
self.dataset
.branch_identifier()
.await
.map_err(|e| OmniError::Lance(e.to_string()))
}
/// Exact materialized `graph_head:<active-branch>` from the same pinned
/// manifest version as [`Self::snapshot`]. This is write authority, not a
/// lineage-cache query: a read may refresh only the manifest, so consulting
/// `CommitGraph` here would combine a fresh table snapshot with a stale head.
pub(crate) fn exact_graph_head(&self) -> Option<String> {
let branch_key = self
.active_branch
.as_deref()
.unwrap_or(MAIN_BRANCH_HEAD_KEY);
self.known_state.graph_heads.get(branch_key).cloned()
}
pub(crate) fn incarnation(&self) -> ManifestIncarnation {
ManifestIncarnation {
version: self.version(),

230
crates/omnigraph/src/db/manifest/publisher.rs
View file

@ -36,12 +36,12 @@ use super::metadata::{TableVersionMetadata, parse_namespace_version_request};
use super::migrations::{read_stamp, refuse_if_stamp_unsupported};
use super::state::{
GraphLineageRow, GraphLineageRowPart, ManifestState, assemble_manifest_state,
graph_lineage_row_parts, head_lineage_row, manifest_rows_batch, manifest_schema,
read_manifest_state, read_publish_scan,
graph_head_object_id, graph_lineage_row_parts, head_lineage_row, manifest_rows_batch,
manifest_schema, read_manifest_state, read_publish_scan,
};
use super::{
ManifestChange, OBJECT_TYPE_TABLE, OBJECT_TYPE_TABLE_TOMBSTONE, OBJECT_TYPE_TABLE_VERSION,
SubTableEntry, TableRegistration, TableTombstone,
MAIN_BRANCH_HEAD_KEY, ManifestChange, OBJECT_TYPE_TABLE, OBJECT_TYPE_TABLE_TOMBSTONE,
OBJECT_TYPE_TABLE_VERSION, SubTableEntry, TableRegistration, TableTombstone,
};
/// Bound on the publisher-level retry loop that wraps Lance's row-level CAS
@ -54,11 +54,10 @@ const PUBLISHER_RETRY_BUDGET: u32 = 5;
/// The graph-lineage commit to record atomically with a manifest publish
/// (RFC-013 Phase 7). One logical commit per publish: the `graph_commit_id` is
/// minted once by the caller and stays stable across the publisher's CAS
/// retries; only the parent re-resolves per attempt (against the freshly loaded
/// `__manifest`), so a retry after a concurrent commit parents off the new head
/// — the TOCTOU the dual-write era's `commit_graph.refresh()` guarded is closed
/// by construction.
#[derive(Debug, Clone)]
/// retries. Legacy [`PublishPrecondition::Any`] publishes re-resolve the parent
/// per attempt. An exact-head publish instead rejects a retry once that authority
/// changed, so a prepared write can never be silently re-parented.
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
pub(crate) struct LineageIntent {
/// ULID minted once before the publish loop; the graph commit's identity.
pub graph_commit_id: String,
@ -73,6 +72,55 @@ pub(crate) struct LineageIntent {
pub created_at: i64,
}
/// The exact mutable graph-head authority a prepared write observed. A missing
/// row is first-class: a freshly-created named branch inherits lineage commits
/// but has no `graph_head:<its-name>` until its first graph commit.
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
pub(crate) struct GraphHeadExpectation {
/// `None` means main; `Some("main")` is normalized to `None` by [`new`].
pub(crate) branch: Option<String>,
/// Lance-native stable branch identity. This detects delete/recreate ABA;
/// manifest versions/eTags are deliberately not branch identity.
pub(crate) branch_identifier: lance::dataset::refs::BranchIdentifier,
/// Exact commit id stored in the branch's head row, or `None` when absent.
pub(crate) head_commit_id: Option<String>,
}
impl GraphHeadExpectation {
pub(crate) fn new(
branch: Option<&str>,
branch_identifier: lance::dataset::refs::BranchIdentifier,
head_commit_id: Option<String>,
) -> Self {
Self {
branch: branch
.filter(|branch| *branch != "main")
.map(ToOwned::to_owned),
branch_identifier,
head_commit_id,
}
}
fn object_id(&self) -> String {
graph_head_object_id(self.branch.as_deref())
}
}
/// Authority checked by the manifest publisher on every CAS attempt.
///
/// `Any` preserves the legacy dispatcher semantics: row-level contention may
/// retry and re-parent a lineage intent. `ExactGraphHead` is the RFC-022
/// foundation for prepared writes: after contention, any head movement becomes
/// `ReadSetChanged` rather than a transparent re-parent. Its native branch-id
/// check detects delete/recreate ABA on every attempt; it is not a distributed
/// ref-control fence (Lance branch create/delete still lacks conditional CAS),
/// so branch control remains within the documented single-writer-process bound.
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
pub(crate) enum PublishPrecondition {
Any,
ExactGraphHead(GraphHeadExpectation),
}
/// The result of a manifest publish that may have folded in a graph commit.
#[derive(Debug)]
pub(super) struct PublishOutcome {
@ -92,11 +140,29 @@ pub(super) struct PublishOutcome {
#[async_trait]
pub(super) trait ManifestBatchPublisher: Send + Sync {
/// Legacy publish behavior. All existing writers deliberately retain
/// `Any` until enrolled in their RFC-022 adapter.
async fn publish(
&self,
changes: &[ManifestChange],
expected_table_versions: &HashMap<String, u64>,
lineage: Option<&LineageIntent>,
) -> Result<PublishOutcome> {
self.publish_with_precondition(
changes,
expected_table_versions,
lineage,
&PublishPrecondition::Any,
)
.await
}
async fn publish_with_precondition(
&self,
changes: &[ManifestChange],
expected_table_versions: &HashMap<String, u64>,
lineage: Option<&LineageIntent>,
precondition: &PublishPrecondition,
) -> Result<PublishOutcome>;
}
@ -119,15 +185,16 @@ struct PendingVersionRow {
/// Everything one CAS attempt needs out of a single `__manifest` scan
/// (RFC-013 P2): the open dataset, table state for the pre-check + pending-row
/// build, and the `graph_commit` lineage rows for parent resolution. Folding the
/// lineage into this struct is what lets `resolve_lineage_rows` skip its own
/// `read_graph_lineage` scan.
/// build, `graph_commit` lineage rows for parent resolution, and exact
/// `graph_head` rows for OCC. Folding lineage authority into this struct is what
/// lets both checks skip their own `read_graph_lineage` scan.
struct LoadedPublishState {
dataset: Dataset,
registered_tables: HashMap<String, String>,
existing_versions: HashMap<(String, u64), SubTableEntry>,
existing_tombstones: HashMap<(String, u64), ()>,
lineage_rows: Vec<GraphLineageRow>,
graph_heads: HashMap<String, String>,
}
impl GraphNamespacePublisher {
@ -159,12 +226,11 @@ impl GraphNamespacePublisher {
// `db/manifest/migrations.rs`.
refuse_if_stamp_unsupported(read_stamp(&dataset))?;
// ONE `__manifest` scan for everything the publish needs: table
// locations, version entries, tombstones, AND the `graph_commit` lineage
// rows for parent resolution (RFC-013 P2). The lineage extraction rides
// this pass instead of a second `read_graph_lineage` scan in
// `resolve_lineage_rows`; the per-attempt re-read is preserved because
// `load_publish_state` runs once per CAS attempt, so a retry sees the
// advanced head and re-parents correctly.
// locations, version entries, tombstones, `graph_commit` lineage rows
// for parent resolution, AND exact `graph_head` rows for OCC (RFC-013
// P2 / RFC-022). Extraction rides this pass instead of a second
// `read_graph_lineage` scan; the per-attempt re-read is preserved because
// `load_publish_state` runs once per CAS attempt.
let scan = read_publish_scan(&dataset).await?;
let existing_versions = scan
.version_entries
@ -183,6 +249,7 @@ impl GraphNamespacePublisher {
existing_versions,
existing_tombstones,
lineage_rows: scan.lineage_rows,
graph_heads: scan.graph_heads,
})
}
@ -506,12 +573,15 @@ impl GraphNamespacePublisher {
))
})?;
let table_path =
table_locations.get(&row.table_key).cloned().ok_or_else(|| {
OmniError::manifest_internal(format!(
"post-publish fold: missing table row for {}",
row.table_key
))
})?;
table_locations
.get(&row.table_key)
.cloned()
.ok_or_else(|| {
OmniError::manifest_internal(format!(
"post-publish fold: missing table row for {}",
row.table_key
))
})?;
let metadata_json = row.metadata.as_deref().ok_or_else(|| {
OmniError::manifest_internal(format!(
"post-publish fold: table_version row missing metadata for {}",
@ -572,6 +642,80 @@ impl GraphNamespacePublisher {
Ok(())
}
/// Check authority inside the publisher retry loop before pending rows are
/// built. The graph head comes from the SAME scan used to build this CAS
/// attempt; Lance's native branch identifier is re-read from the ref. Thus a
/// row-level-CAS loser with an exact expectation cannot silently re-parent
/// on its next attempt.
async fn check_publish_precondition(
&self,
dataset: &Dataset,
graph_heads: &HashMap<String, String>,
precondition: &PublishPrecondition,
) -> Result<()> {
let PublishPrecondition::ExactGraphHead(expected) = precondition else {
return Ok(());
};
let expected_branch = expected
.branch
.as_deref()
.filter(|branch| *branch != "main");
if expected_branch != self.branch.as_deref() {
return Err(OmniError::manifest_internal(format!(
"publish graph-head precondition targets branch '{}' but publisher is bound to '{}'",
expected_branch.unwrap_or("main"),
self.branch.as_deref().unwrap_or("main"),
)));
}
let branch_identity_member =
format!("branch_identifier:{}", expected_branch.unwrap_or("main"));
let expected_branch_identifier = serde_json::to_string(&expected.branch_identifier)
.map_err(|e| {
OmniError::manifest_internal(format!(
"failed to encode expected Lance branch identifier: {e}"
))
})?;
let actual_branch_identifier = match dataset.branch_identifier().await {
Ok(identifier) => identifier,
Err(LanceError::RefNotFound { .. }) => {
return Err(OmniError::manifest_read_set_changed(
branch_identity_member,
Some(expected_branch_identifier),
None,
));
}
Err(err) => return Err(OmniError::Lance(err.to_string())),
};
if actual_branch_identifier != expected.branch_identifier {
let actual = serde_json::to_string(&actual_branch_identifier).map_err(|e| {
OmniError::manifest_internal(format!(
"failed to encode current Lance branch identifier: {e}"
))
})?;
return Err(OmniError::manifest_read_set_changed(
branch_identity_member,
Some(expected_branch_identifier),
Some(actual),
));
}
let object_id = expected.object_id();
let branch_key = object_id
.strip_prefix("graph_head:")
.expect("graph_head_object_id always supplies the prefix");
let actual = graph_heads.get(branch_key).cloned();
if actual != expected.head_commit_id {
return Err(OmniError::manifest_read_set_changed(
object_id,
expected.head_commit_id.clone(),
actual,
));
}
Ok(())
}
async fn merge_rows(&self, dataset: Dataset, rows: Vec<PendingVersionRow>) -> Result<Dataset> {
let batch = Self::pending_rows_to_batch(rows)?;
let reader = RecordBatchIterator::new(vec![Ok(batch)], manifest_schema());
@ -671,13 +815,18 @@ pub(crate) fn map_lance_publish_error(err: LanceError) -> OmniError {
#[async_trait]
impl ManifestBatchPublisher for GraphNamespacePublisher {
async fn publish(
async fn publish_with_precondition(
&self,
changes: &[ManifestChange],
expected_table_versions: &HashMap<String, u64>,
lineage: Option<&LineageIntent>,
precondition: &PublishPrecondition,
) -> Result<PublishOutcome> {
if changes.is_empty() && expected_table_versions.is_empty() && lineage.is_none() {
if changes.is_empty()
&& expected_table_versions.is_empty()
&& lineage.is_none()
&& matches!(precondition, PublishPrecondition::Any)
{
// Defensive no-op (never reached from `commit_changes_with_lineage`,
// which short-circuits the all-empty case): state is unchanged, so a
// re-scan here is acceptable.
@ -709,8 +858,16 @@ impl ManifestBatchPublisher for GraphNamespacePublisher {
existing_versions,
existing_tombstones,
lineage_rows,
graph_heads,
} = loaded;
// Exact logical authority is checked on EVERY attempt from this
// attempt's single manifest scan. In particular, a CAS retry after
// another writer creates or advances `graph_head:<branch>` fails
// here instead of transparently re-parenting the prepared intent.
self.check_publish_precondition(&dataset, &graph_heads, precondition)
.await?;
let latest_per_table =
Self::latest_visible_per_table(&existing_versions, &existing_tombstones);
// Pre-check on every attempt against freshly loaded state so a
@ -754,6 +911,7 @@ impl ManifestBatchPublisher for GraphNamespacePublisher {
existing_tombstones
.keys()
.map(|(key, version)| (key.clone(), *version)),
graph_heads,
);
return Ok(PublishOutcome {
dataset,
@ -765,8 +923,23 @@ impl ManifestBatchPublisher for GraphNamespacePublisher {
// Build the post-publish fold inputs from the pre-publish state ∪ the
// rows we are about to commit, BEFORE `rows` is moved into merge_rows
// (RFC-013 PR2 #1b). Recomputed per attempt from freshly-loaded state.
let (fold_entries, fold_tombstones) =
Self::fold_inputs(&existing_versions, &existing_tombstones, &rows, &known_tables)?;
let (fold_entries, fold_tombstones) = Self::fold_inputs(
&existing_versions,
&existing_tombstones,
&rows,
&known_tables,
)?;
let mut fold_graph_heads = graph_heads;
if let Some(intent) = lineage {
fold_graph_heads.insert(
intent
.branch
.as_deref()
.unwrap_or(MAIN_BRANCH_HEAD_KEY)
.to_string(),
intent.graph_commit_id.clone(),
);
}
match self.merge_rows(dataset, rows).await {
Ok(new_dataset) => {
@ -774,6 +947,7 @@ impl ManifestBatchPublisher for GraphNamespacePublisher {
new_dataset.version().version,
fold_entries,
fold_tombstones,
fold_graph_heads,
);
return Ok(PublishOutcome {
dataset: new_dataset,

2728
crates/omnigraph/src/db/manifest/recovery.rs
View file

File diff suppressed because it is too large Load diff

129
crates/omnigraph/src/db/manifest/state.rs
View file

@ -29,6 +29,11 @@ pub struct SubTableEntry {
pub(super) struct ManifestState {
pub(super) version: u64,
pub(super) entries: Vec<SubTableEntry>,
/// Exact materialized `graph_head:<branch>` values from this SAME manifest
/// version. Keeping the head beside the table snapshot prevents a
/// manifest-only refresh from leaving coarse write authority split between
/// a fresh table view and the commit graph's older derived cache.
pub(super) graph_heads: HashMap<String, String>,
}
#[derive(Debug, Clone)]
@ -91,10 +96,13 @@ struct ManifestScan {
/// the caller asked (`collect_lineage`). Empty on the table-state read hot
/// path so it never pays the O(commits) lineage JSON decode; populated on the
/// publish path, where `load_publish_state` already needs the parent and would
/// otherwise scan `__manifest` a second time via `read_graph_lineage`. `graph_head`
/// rows are not collected here — parent resolution uses the head-over-commits
/// computation, not the denormalized head pointer (see `resolve_lineage_rows`).
/// otherwise scan `__manifest` a second time via `read_graph_lineage`.
lineage_rows: Vec<GraphLineageRow>,
/// Exact materialized `graph_head:<branch>` values, collected on every
/// table-state/publish pass. This is bounded by branch count; unlike
/// `lineage_rows`, it does not grow with commit history. OCC must distinguish
/// a present head from an absent one (notably on a fresh named branch).
graph_heads: HashMap<String, String>,
}
pub(super) fn manifest_schema() -> SchemaRef {
@ -137,6 +145,7 @@ pub(super) async fn read_manifest_state(dataset: &Dataset) -> Result<ManifestSta
scan.tombstones
.into_iter()
.map(|t| (t.table_key, t.tombstone_version)),
scan.graph_heads,
))
}
@ -152,6 +161,7 @@ pub(super) fn assemble_manifest_state(
version: u64,
version_entries: Vec<SubTableEntry>,
tombstones: impl IntoIterator<Item = (String, u64)>,
graph_heads: HashMap<String, String>,
) -> ManifestState {
let mut latest_versions = HashMap::<String, SubTableEntry>::new();
for entry in version_entries {
@ -183,7 +193,11 @@ pub(super) fn assemble_manifest_state(
})
.collect();
entries.sort_by(|a, b| a.table_key.cmp(&b.table_key));
ManifestState { version, entries }
ManifestState {
version,
entries,
graph_heads,
}
}
// After RFC-013 P2 folded the publish path off this accessor (it now projects
@ -208,6 +222,9 @@ pub(super) struct PublishScan {
pub(super) version_entries: Vec<SubTableEntry>,
pub(super) tombstones: Vec<((String, u64), ())>,
pub(super) lineage_rows: Vec<GraphLineageRow>,
/// Exact `graph_head:<branch>` rows keyed by the branch suffix (`main` for
/// main). Absence is meaningful and is preserved by a missing map entry.
pub(super) graph_heads: HashMap<String, String>,
}
/// One-scan read of everything the publish path needs. `collect_lineage` is
@ -224,6 +241,7 @@ pub(super) async fn read_publish_scan(dataset: &Dataset) -> Result<PublishScan>
.map(|tombstone| ((tombstone.table_key, tombstone.tombstone_version), ()))
.collect(),
lineage_rows: scan.lineage_rows,
graph_heads: scan.graph_heads,
})
}
@ -264,14 +282,43 @@ fn decode_graph_commit_row(
})
}
/// Decode one `graph_head` row into its exact branch-key / commit-id pair.
/// Shared by the dedicated lineage reader and the publisher's folded one-scan
/// path so presence, absence, and malformed-row handling cannot drift.
fn decode_graph_head_row(
object_ids: &StringArray,
metadata: &StringArray,
row: usize,
) -> Result<(String, String)> {
if metadata.is_null(row) {
return Err(OmniError::manifest_internal(format!(
"manifest graph_head row missing metadata for {}",
object_ids.value(row)
)));
}
let head_meta: GraphHeadMetadata = serde_json::from_str(metadata.value(row)).map_err(|e| {
OmniError::manifest_internal(format!("failed to decode graph_head metadata: {e}"))
})?;
let branch_key = object_ids
.value(row)
.strip_prefix("graph_head:")
.ok_or_else(|| {
OmniError::manifest_internal(format!(
"invalid graph_head object id {}",
object_ids.value(row)
))
})?
.to_string();
Ok((branch_key, head_meta.head_commit_id))
}
async fn read_manifest_scan(dataset: &Dataset, collect_lineage: bool) -> Result<ManifestScan> {
// Project only the columns the assembly below reads (RFC-013 PR2 #1c). The
// table-state hot path never touches `object_id` (lineage decode only) or
// `base_objects` (reserved/unused — never read on any path), so reading them
// is wasted bytes on every `__manifest` scan — write publish AND every
// branch-op open. Mirrors Lance's own directory-catalog `__manifest` reads,
// which project to the needed columns rather than scanning all of them.
let mut projection: Vec<&str> = vec![
// `object_id` is needed for the bounded graph-head authority decode on every
// path; `base_objects` remains reserved/unused. Mirrors Lance's own
// directory-catalog `__manifest` reads, which project only needed columns.
let projection: Vec<&str> = vec![
"object_id",
"object_type",
"location",
"metadata",
@ -280,9 +327,6 @@ async fn read_manifest_scan(dataset: &Dataset, collect_lineage: bool) -> Result<
"table_branch",
"row_count",
];
if collect_lineage {
projection.push("object_id");
}
let mut scanner = dataset.scan();
scanner
.project(&projection)
@ -299,6 +343,7 @@ async fn read_manifest_scan(dataset: &Dataset, collect_lineage: bool) -> Result<
let mut version_entries = Vec::new();
let mut tombstones = Vec::new();
let mut lineage_rows = Vec::new();
let mut graph_heads = HashMap::new();
for batch in &batches {
let object_types = string_column(batch, "object_type")?;
@ -308,13 +353,11 @@ async fn read_manifest_scan(dataset: &Dataset, collect_lineage: bool) -> Result<
let versions = u64_column(batch, "table_version")?;
let branches = string_column(batch, "table_branch")?;
let row_counts = u64_column(batch, "row_count")?;
// `object_id` is only needed for lineage decoding; skip the lookup
// entirely on the table-state hot path (`collect_lineage == false`).
let object_ids = if collect_lineage {
Some(string_column(batch, "object_id")?)
} else {
None
};
// `object_id` is needed for the exact graph-head authority even on the
// table-state path. We still skip every `graph_commit` decode there, so
// the added work is bounded by the number of branch-head rows rather
// than growing with commit history.
let object_ids = string_column(batch, "object_id")?;
for row in 0..batch.num_rows() {
let table_key = table_keys.value(row).to_string();
@ -358,21 +401,22 @@ async fn read_manifest_scan(dataset: &Dataset, collect_lineage: bool) -> Result<
tombstone_version,
});
}
// `graph_commit` rows (RFC-013) are decoded into the scan ONLY
// when `collect_lineage` is set (the publish path, which resolves
// a parent). The table-state hot path leaves them — and
// `graph_head` + any future object type — in the `_` arm so it
// never pays the O(commits) lineage JSON decode. When NOT
// collecting, `object_ids` is `None`, so this arm is the same
// forward-compat skip as the `_` arm.
// `graph_commit` rows (RFC-013) are decoded ONLY for the publish
// path, which resolves a parent. The table-state hot path skips
// them, while still decoding the bounded graph-head authority
// rows below.
OBJECT_TYPE_GRAPH_COMMIT if collect_lineage => {
let object_ids = object_ids.expect("object_ids read when collect_lineage");
lineage_rows.push(decode_graph_commit_row(
object_ids, metadata, versions, branches, row,
)?);
}
// Skipped on the table-state path (and for `graph_head` / unknown
// future object types on every path): no table snapshot needs them.
OBJECT_TYPE_GRAPH_HEAD => {
let (branch_key, head_commit_id) =
decode_graph_head_row(object_ids, metadata, row)?;
graph_heads.insert(branch_key, head_commit_id);
}
// Commit rows are skipped on the table-state path; unknown future
// object types are skipped on every path.
_ => {}
}
}
@ -404,6 +448,7 @@ async fn read_manifest_scan(dataset: &Dataset, collect_lineage: bool) -> Result<
version_entries: entries,
tombstones,
lineage_rows,
graph_heads,
})
}
@ -447,26 +492,9 @@ pub(crate) async fn read_graph_lineage(
)?);
}
OBJECT_TYPE_GRAPH_HEAD => {
if metadata.is_null(row) {
return Err(OmniError::manifest_internal(format!(
"manifest graph_head row missing metadata for {}",
object_ids.value(row)
)));
}
let head_meta: GraphHeadMetadata = serde_json::from_str(metadata.value(row))
.map_err(|e| {
OmniError::manifest_internal(format!(
"failed to decode graph_head metadata: {e}"
))
})?;
// `object_id` is `graph_head:<branch>`; the branch key after
// the prefix is the projection's map key (`main` for main).
let branch_key = object_ids
.value(row)
.strip_prefix("graph_head:")
.unwrap_or_default()
.to_string();
graph_heads.insert(branch_key, head_meta.head_commit_id);
let (branch_key, head_commit_id) =
decode_graph_head_row(object_ids, metadata, row)?;
graph_heads.insert(branch_key, head_commit_id);
}
_ => {}
}
@ -623,7 +651,6 @@ pub(super) fn entries_to_batch(
)
}
pub(super) fn manifest_rows_batch(
object_ids: Vec<String>,
object_types: Vec<String>,

266
crates/omnigraph/src/db/manifest/tests.rs
View file

@ -12,7 +12,11 @@ use lance_namespace::models::{
use lance_namespace_impls::DirectoryNamespaceBuilder;
use tokio::sync::Mutex;
use super::publisher::{LineageIntent, ManifestBatchPublisher, PublishOutcome};
use super::publisher::{
GraphHeadExpectation, LineageIntent, ManifestBatchPublisher, PublishOutcome,
PublishPrecondition,
};
use super::state::read_publish_scan;
use super::*;
use omnigraph_compiler::catalog::build_catalog;
use omnigraph_compiler::schema::parser::parse_schema;
@ -154,10 +158,11 @@ async fn test_commit_changes_can_register_new_table_and_tombstone_old_one() {
let ds = crate::table_store::TableStore::create_empty_dataset(&dataset_uri, &schema)
.await
.unwrap();
let state = crate::table_store::TableStore::new(uri, Arc::new(lance::session::Session::default()))
.table_state(&dataset_uri, &ds)
.await
.unwrap();
let state =
crate::table_store::TableStore::new(uri, Arc::new(lance::session::Session::default()))
.table_state(&dataset_uri, &ds)
.await
.unwrap();
mc.commit_changes(&[
ManifestChange::RegisterTable(TableRegistration {
@ -611,7 +616,10 @@ async fn test_branch_manifest_namespace_uses_entry_owner_branch_for_latest_table
.with_branch("feature", None)
.load()
.await;
let err = format!("{:?}", mismatched.expect_err("branch-mismatched open must fail on v9"));
let err = format!(
"{:?}",
mismatched.expect_err("branch-mismatched open must fail on v9")
);
assert!(
err.contains("belonging to branch"),
"expected the v9 branch-consistency error, got: {err}"
@ -1148,11 +1156,12 @@ impl RecordingPublisher {
#[async_trait]
impl ManifestBatchPublisher for RecordingPublisher {
async fn publish(
async fn publish_with_precondition(
&self,
changes: &[ManifestChange],
expected_table_versions: &HashMap<String, u64>,
lineage: Option<&LineageIntent>,
precondition: &PublishPrecondition,
) -> Result<PublishOutcome> {
let requests: Vec<CreateTableVersionRequest> = changes
.iter()
@ -1163,7 +1172,7 @@ impl ManifestBatchPublisher for RecordingPublisher {
.collect();
self.requests.lock().await.extend_from_slice(&requests);
self.inner
.publish(changes, expected_table_versions, lineage)
.publish_with_precondition(changes, expected_table_versions, lineage, precondition)
.await
}
}
@ -1172,11 +1181,12 @@ struct FailingPublisher;
#[async_trait]
impl ManifestBatchPublisher for FailingPublisher {
async fn publish(
async fn publish_with_precondition(
&self,
_changes: &[ManifestChange],
_expected_table_versions: &HashMap<String, u64>,
_lineage: Option<&LineageIntent>,
_precondition: &PublishPrecondition,
) -> Result<PublishOutcome> {
Err(OmniError::manifest(
"injected batch publisher failure".to_string(),
@ -1760,7 +1770,9 @@ async fn sub_current_graph_is_refused_on_open_with_rebuild_hint() {
// (v4) cannot open, since `MIN_SUPPORTED == CURRENT == 4`.
{
let mut ds = open_manifest_dataset(uri, None).await.unwrap();
super::migrations::set_stamp_for_test(&mut ds, 3).await.unwrap();
super::migrations::set_stamp_for_test(&mut ds, 3)
.await
.unwrap();
}
// Read-write open is refused with the rebuild hint.
@ -1820,7 +1832,9 @@ async fn sub_current_graph_is_refused_then_rebuilt_via_export_import() {
// Make it look like a graph from an older release: rewind the stamp below CURRENT.
{
let mut ds = open_manifest_dataset(uri_old, None).await.unwrap();
super::migrations::set_stamp_for_test(&mut ds, 3).await.unwrap();
super::migrations::set_stamp_for_test(&mut ds, 3)
.await
.unwrap();
}
let err = match Omnigraph::open(uri_old).await {
Ok(_) => panic!("a sub-CURRENT graph must be refused on open"),
@ -1866,12 +1880,11 @@ async fn sub_current_graph_is_refused_then_rebuilt_via_export_import() {
// Two (or N) writers committing DISJOINT tables on the same branch still share
// one mutable `graph_head:main` row (one `object_id`, `WhenMatched::UpdateAll`).
// Their table-version rows never collide (distinct `object_id`s), so the *only*
// row-level CAS contention is on `graph_head:main`. The contract under test:
// exactly one writer wins each CAS round; the loser retries, re-resolves its
// parent off the freshly-advanced head (inside the publisher's retry loop), and
// re-commits — so every writer commits and the resulting graph_commit DAG is a
// single LINEAR chain (no fork), not a tree. This is the cross-process
// disjoint-table fork closed by the shared head row (invariants.md §7.1).
// row-level CAS contention is on `graph_head:main`. Two contracts coexist:
// legacy `Any` publishers retry, re-parent, and eventually form one linear DAG;
// RFC-022 `ExactGraphHead` publishers instead reject the stale prepared write
// after the first winner changes authority. Both rely on the same shared row to
// prevent a fork; the tests below pin both behaviors explicitly.
/// A microsecond UNIX timestamp for a `LineageIntent`, matching the genesis /
/// commit-graph `created_at` unit.
@ -1882,6 +1895,180 @@ fn lineage_now_micros() -> i64 {
.as_micros() as i64
}
/// Race two lineage-only publishes against the same exact named-branch head.
/// Returns after proving exactly one committed and the loser surfaced a typed
/// read-set change. `establish_head=false` exercises the load-bearing absent-row
/// case on a fresh branch; `true` exercises ordinary head advancement.
async fn assert_exact_named_head_race(establish_head: bool) {
use crate::error::ManifestConflictDetails;
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().to_str().unwrap();
let catalog = build_test_catalog();
let mut mc = ManifestCoordinator::init(uri, &catalog).await.unwrap();
mc.create_branch("feature").await.unwrap();
let publisher = GraphNamespacePublisher::new(uri, Some("feature"));
let empty = HashMap::new();
let expected_head = if establish_head {
let intent = LineageIntent {
graph_commit_id: ulid::Ulid::new().to_string(),
branch: Some("feature".to_string()),
actor_id: None,
merged_parent_commit_id: None,
created_at: lineage_now_micros(),
};
publisher
.publish(&[], &empty, Some(&intent))
.await
.expect("establish named-branch graph head");
Some(intent.graph_commit_id)
} else {
None
};
// The folded publish scan must preserve exact absence. In particular, the
// inferred lineage head inherited from main must not masquerade as a
// materialized `graph_head:feature` row.
let branch_manifest = open_manifest_dataset(uri, Some("feature")).await.unwrap();
let scan = read_publish_scan(&branch_manifest).await.unwrap();
assert_eq!(scan.graph_heads.get("feature").cloned(), expected_head);
let branch_identifier = branch_manifest.branch_identifier().await.unwrap();
let precondition = PublishPrecondition::ExactGraphHead(GraphHeadExpectation::new(
Some("feature"),
branch_identifier,
expected_head.clone(),
));
let intent_a = LineageIntent {
graph_commit_id: ulid::Ulid::new().to_string(),
branch: Some("feature".to_string()),
actor_id: Some("act-a".to_string()),
merged_parent_commit_id: None,
created_at: lineage_now_micros(),
};
let intent_b = LineageIntent {
graph_commit_id: ulid::Ulid::new().to_string(),
branch: Some("feature".to_string()),
actor_id: Some("act-b".to_string()),
merged_parent_commit_id: None,
created_at: lineage_now_micros(),
};
let publisher_a = GraphNamespacePublisher::new(uri, Some("feature"));
let publisher_b = GraphNamespacePublisher::new(uri, Some("feature"));
let precondition_a = precondition.clone();
let precondition_b = precondition;
let (result_a, result_b) = tokio::join!(
async {
publisher_a
.publish_with_precondition(&[], &empty, Some(&intent_a), &precondition_a)
.await
},
async {
publisher_b
.publish_with_precondition(&[], &empty, Some(&intent_b), &precondition_b)
.await
}
);
let (winner_id, loser_error) = match (result_a, result_b) {
(Ok(_), Err(err)) => (intent_a.graph_commit_id.clone(), err),
(Err(err), Ok(_)) => (intent_b.graph_commit_id.clone(), err),
(Ok(_), Ok(_)) => panic!("exact-head race silently re-parented both writers"),
(Err(a), Err(b)) => panic!("both exact-head writers failed: {a:?} / {b:?}"),
};
let OmniError::Manifest(error) = loser_error else {
panic!("expected typed manifest conflict, got {loser_error:?}");
};
assert_eq!(
error.details,
Some(ManifestConflictDetails::ReadSetChanged {
member: "graph_head:feature".to_string(),
expected: expected_head,
actual: Some(winner_id.clone()),
})
);
let branch_manifest = open_manifest_dataset(uri, Some("feature")).await.unwrap();
let (commits, heads) = read_graph_lineage(&branch_manifest).await.unwrap();
assert_eq!(heads.get("feature"), Some(&winner_id));
assert_eq!(
commits
.iter()
.filter(|commit| {
commit.graph_commit_id == intent_a.graph_commit_id
|| commit.graph_commit_id == intent_b.graph_commit_id
})
.count(),
1,
"the rejected intent must not leave an immutable graph_commit row"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn exact_head_publish_rejects_reparent_on_fresh_and_established_named_branch() {
assert_exact_named_head_race(false).await;
assert_exact_named_head_race(true).await;
}
#[tokio::test]
async fn exact_publish_rejects_named_branch_delete_recreate_aba() {
use crate::error::ManifestConflictDetails;
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().to_str().unwrap();
let catalog = build_test_catalog();
let mut mc = ManifestCoordinator::init(uri, &catalog).await.unwrap();
mc.create_branch("feature").await.unwrap();
let old_branch = open_manifest_dataset(uri, Some("feature")).await.unwrap();
let old_identifier = old_branch.branch_identifier().await.unwrap();
assert!(
read_publish_scan(&old_branch)
.await
.unwrap()
.graph_heads
.get("feature")
.is_none(),
"fresh named branch starts without its own graph_head row"
);
// Recreate the same name at the same logical fork point. Numeric manifest
// version and exact graph-head absence can repeat; only Lance's native
// branch identifier distinguishes the incarnation.
mc.delete_branch("feature").await.unwrap();
mc.create_branch("feature").await.unwrap();
let recreated = open_manifest_dataset(uri, Some("feature")).await.unwrap();
let recreated_identifier = recreated.branch_identifier().await.unwrap();
assert_ne!(old_identifier, recreated_identifier);
let precondition = PublishPrecondition::ExactGraphHead(GraphHeadExpectation::new(
Some("feature"),
old_identifier,
None,
));
let err = GraphNamespacePublisher::new(uri, Some("feature"))
.publish_with_precondition(&[], &HashMap::new(), None, &precondition)
.await
.expect_err("recreated branch must reject the old incarnation token");
let OmniError::Manifest(error) = err else {
panic!("expected typed manifest conflict, got {err:?}");
};
match error.details {
Some(ManifestConflictDetails::ReadSetChanged {
member,
expected: Some(expected),
actual: Some(actual),
}) => {
assert_eq!(member, "branch_identifier:feature");
assert_ne!(expected, actual);
}
other => panic!("expected branch-identifier ReadSetChanged, got {other:?}"),
}
}
/// Append one row to a two-column NODE table (`id`, `name`) and return the
/// resulting `SubTableUpdate` at the new on-disk version. Generalizes
/// `append_person_and_make_update` to any node table whose schema is `(id:
@ -1943,8 +2130,10 @@ async fn assert_linear_chain(uri: &str, expected_total: usize) -> String {
);
// (1) exactly one genesis.
let genesis: Vec<&GraphLineageRow> =
rows.iter().filter(|r| r.parent_commit_id.is_none()).collect();
let genesis: Vec<&GraphLineageRow> = rows
.iter()
.filter(|r| r.parent_commit_id.is_none())
.collect();
assert_eq!(
genesis.len(),
1,
@ -2038,8 +2227,16 @@ async fn concurrent_disjoint_writes_share_head_and_form_linear_chain() {
// version on the other's table; contention is purely the shared head row.
let empty = HashMap::new();
let (res_a, res_b) = tokio::join!(
async { publisher_a.publish(&changes_a, &empty, Some(&intent_a)).await },
async { publisher_b.publish(&changes_b, &empty, Some(&intent_b)).await }
async {
publisher_a
.publish(&changes_a, &empty, Some(&intent_a))
.await
},
async {
publisher_b
.publish(&changes_b, &empty, Some(&intent_b))
.await
}
);
// BOTH commit: disjoint tables → the head-row CAS loser retries within
@ -2114,8 +2311,16 @@ async fn concurrent_disjoint_writes_form_linear_chain_on_s3() {
};
let empty = HashMap::new();
let (res_a, res_b) = tokio::join!(
async { publisher_a.publish(&changes_a, &empty, Some(&intent_a)).await },
async { publisher_b.publish(&changes_b, &empty, Some(&intent_b)).await }
async {
publisher_a
.publish(&changes_a, &empty, Some(&intent_a))
.await
},
async {
publisher_b
.publish(&changes_b, &empty, Some(&intent_b))
.await
}
);
res_a.expect("writer A must commit on S3");
res_b.expect("writer B must commit on S3");
@ -2161,8 +2366,13 @@ async fn n_concurrent_disjoint_writers_converge_to_one_linear_chain() {
.await,
);
updates.push(
append_node_row_and_make_update(uri, &company_entry, &format!("c{i}"), &format!("C{i}"))
.await,
append_node_row_and_make_update(
uri,
&company_entry,
&format!("c{i}"),
&format!("C{i}"),
)
.await,
);
}
assert_eq!(updates.len(), N);
@ -2216,7 +2426,11 @@ async fn n_concurrent_disjoint_writers_converge_to_one_linear_chain() {
// All 8 distinct writer ids committed (no lost commit, no duplicate id).
committed_ids.sort();
committed_ids.dedup();
assert_eq!(committed_ids.len(), N, "every writer must commit exactly once");
assert_eq!(
committed_ids.len(),
N,
"every writer must commit exactly once"
);
// The final DAG is a single linear chain of genesis + 8 = 9, no fork.
assert_linear_chain(uri, N + 1).await;

39
crates/omnigraph/src/db/mod.rs
View file

@ -10,7 +10,7 @@ pub use commit_graph::GraphCommit;
pub use graph_coordinator::{GraphCoordinator, ReadTarget, ResolvedTarget, SnapshotId};
pub use manifest::{Snapshot, SubTableEntry, SubTableUpdate};
pub(crate) use omnigraph::ensure_public_branch_ref;
pub(crate) use omnigraph::WriteTxn;
pub(crate) use omnigraph::{DeferredTableFork, WriteTxn};
pub use omnigraph::{
CleanupPolicyOptions, InitOptions, MergeOutcome, Omnigraph, OpenMode, PendingIndex,
RepairAction, RepairClassification, RepairOptions, RepairStats, SchemaApplyOptions,
@ -21,30 +21,27 @@ use crate::error::{OmniError, Result};
pub(crate) const SCHEMA_APPLY_LOCK_BRANCH: &str = "__schema_apply_lock__";
/// Mutation kind, threaded through the version-check call sites so the
/// engine can apply an op-kind-aware policy:
/// Mutation kind, threaded through the early table-version checks so the
/// engine can apply an op-kind-aware staging policy. This check is not the
/// RFC-022 publish authority: enrolled mutation/load attempts additionally
/// capture an exact branch-wide `WriteTxn`, then revalidate it while holding
/// the root-shared schema → branch → sorted-table gates.
///
/// - `Insert` / `Merge`: skip the strict pre-stage `ensure_expected_version`
/// check. Lance's `MergeInsertBuilder` rebases concurrent appends; the
/// per-(table, branch) writer queue serializes `commit_staged`; the
/// publisher's CAS (refreshed under the queue via
/// `MutationStaging::commit_all`'s `snapshot_for_branch` call) catches
/// genuine cross-process drift as `ManifestConflictDetails::ExpectedVersionMismatch`.
/// The pre-stage strict check would over-reject in-process concurrent
/// inserts, which is exactly the case PR 2 / MR-686 designed the
/// per-table queue to allow.
/// check because their staged files are reclaimable and the complete prepared
/// attempt is checked later against the exact branch authority. On a
/// pre-effect `ReadSetChanged`, mutation Insert and load Append/Merge discard
/// the whole attempt and reprepare with a bounded retry; they never patch
/// table pins beneath an already-validated plan.
///
/// - `Update` / `Delete`: keep the strict check. These have read-modify-write
/// semantics; Lance moving between the read at stage time and the write
/// at commit time means the staged batch is computed against stale state.
/// The strict check guards the per-query SI invariant. SERIALIZABLE
/// opt-in (§VI.36 future seam) is the long-term answer for tighter
/// semantics; today, in-process update-update races on the same key
/// stay rejected as 409 — acceptable.
/// - `Update` / `Delete`: keep the strict early check because these are
/// read-modify-write effects computed from a pinned image. Enrolled attempts
/// still perform the later branch-wide revalidation; a mismatch is strict
/// `ReadSetChanged` (HTTP 409), not a transparent replay.
///
/// - `SchemaRewrite`: keep the strict check. Schema apply runs under the
/// graph-wide `__schema_apply_lock__` AND per-table queues; the strict
/// check is uncontested at that point.
/// - `SchemaRewrite`: keep the strict early check for overwrite/rewrite effects.
/// An enrolled load Overwrite also uses the exact branch-wide gate and
/// surfaces `ReadSetChanged`; schema apply has its own schema/table protocol.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum MutationOpKind {
Insert,

791
crates/omnigraph/src/db/omnigraph.rs
View file

File diff suppressed because it is too large Load diff

166
crates/omnigraph/src/db/omnigraph/optimize.rs
View file

@ -21,8 +21,9 @@
//! retention. Destructive to version history — callers should gate this
//! behind an explicit confirm flag at the CLI layer.
//!
//! Both walk every node + edge table on the `main` branch. Run branches
//! are ephemeral by design so we do not optimize them.
//! Both orchestrate the graph's node + edge datasets from main authority;
//! cleanup preserves Lance-referenced named-branch history according to its
//! retention policy.
use std::time::Duration;
@ -113,10 +114,10 @@ pub struct TableOptimizeStats {
/// `fragments_removed == 0`, `fragments_added == 0`, and `!committed`.
pub skipped: Option<SkipReason>,
/// Manifest table version observed by optimize for drift skips. `None` for
/// normal compaction/no-op/blob skips.
/// normal compaction/no-op outcomes.
pub manifest_version: Option<u64>,
/// Lance HEAD version observed by optimize for drift skips. `None` for
/// normal compaction/no-op/blob skips.
/// normal compaction/no-op outcomes.
pub lance_head_version: Option<u64>,
/// Declared `@index` columns on this table the reconciler could not build
/// this run, each with the `reason` (today: a vector column with no
@ -141,20 +142,6 @@ impl TableOptimizeStats {
}
}
/// Stat for a table that was deliberately skipped (compaction not attempted).
fn skipped(table_key: String, reason: SkipReason) -> Self {
Self {
table_key,
fragments_removed: 0,
fragments_added: 0,
committed: false,
skipped: Some(reason),
manifest_version: None,
lance_head_version: None,
pending_indexes: Vec::new(),
}
}
/// Stat for a table skipped because the manifest and Lance HEAD disagree.
fn skipped_for_drift(
table_key: String,
@ -207,7 +194,6 @@ pub async fn optimize_all_tables(db: &Omnigraph) -> Result<Vec<TableOptimizeStat
recovery sweep before optimizing",
));
}
let snapshot = db.fresh_snapshot_for_branch(None).await?;
// Compute per-table paths up front, in a scope that drops the catalog
@ -402,8 +388,7 @@ async fn optimize_one_table(
// recovery and rolls back siblings).
let needs_reindex = TableStore::has_unindexed_fragments(&ds).await?;
let needs_index_create = if let Some(type_name) = table_key.strip_prefix("node:") {
super::table_ops::needs_index_work_node(db, type_name, &full_path, None)
.await?
super::table_ops::needs_index_work_node(db, type_name, &full_path, None).await?
} else {
super::table_ops::needs_index_work_edge(db, &full_path, None).await?
};
@ -450,7 +435,8 @@ async fn optimize_one_table(
}],
);
sidecar = Some(
crate::db::manifest::write_sidecar(db.root_uri(), db.storage_adapter(), &sc).await?,
crate::db::manifest::write_sidecar(db.root_uri(), db.storage_adapter(), &sc)
.await?,
);
}
@ -493,7 +479,8 @@ async fn optimize_one_table(
// committed (so HEAD is already ahead of the manifest from our own work),
// exercising the own-HEAD (not external) drift classification on the next
// reopened attempt.
if crate::failpoints::maybe_fail(crate::failpoints::names::OPTIMIZE_INJECT_REINDEX_CONFLICT).is_err()
if crate::failpoints::maybe_fail(crate::failpoints::names::OPTIMIZE_INJECT_REINDEX_CONFLICT)
.is_err()
&& attempt < COMPACTION_RETRY_BUDGET
{
continue;
@ -504,7 +491,10 @@ async fn optimize_one_table(
continue;
}
Err(e) => {
return Err(OmniError::Lance(format!("optimize_indices on {}: {}", table_key, e)));
return Err(OmniError::Lance(format!(
"optimize_indices on {}: {}",
table_key, e
)));
}
}
@ -528,7 +518,9 @@ async fn optimize_one_table(
// Pin the per-writer Phase B → Phase C residual: Lance HEAD has advanced but the
// manifest publish below hasn't run.
crate::failpoints::maybe_fail(crate::failpoints::names::OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT)?;
crate::failpoints::maybe_fail(
crate::failpoints::names::OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT,
)?;
// Phase C: monotonic fast-forward publish. The compaction is committed at Lance
// HEAD `N`; publish a manifest pointer that includes it. If a concurrent writer
@ -734,10 +726,7 @@ async fn compact_internal_table(
// conflict we re-open at the new HEAD and rerun — the canonical Lance-consumer
// pattern. Each attempt opens fresh because the conflict means the version moved.
for attempt in 0..COMPACTION_RETRY_BUDGET {
let handle = db
.storage()
.open_dataset_head(&uri, None)
.await?;
let handle = db.storage().open_dataset_head(&uri, None).await?;
let mut ds = handle.into_dataset();
// Keep optimize non-destructive by construction (see clear_stale_auto_cleanup_config).
@ -745,8 +734,7 @@ async fn compact_internal_table(
let cleared_config = match clear_stale_auto_cleanup_config(&mut ds).await {
Ok(cleared) => cleared,
Err(e) => {
if attempt + 1 < COMPACTION_RETRY_BUDGET && is_retryable_lance_conflict(&e)
{
if attempt + 1 < COMPACTION_RETRY_BUDGET && is_retryable_lance_conflict(&e) {
continue;
}
return Err(OmniError::Lance(e.to_string()));
@ -784,10 +772,7 @@ async fn compact_internal_table(
true,
));
}
Err(e)
if attempt + 1 < COMPACTION_RETRY_BUDGET
&& is_retryable_lance_conflict(&e) =>
{
Err(e) if attempt + 1 < COMPACTION_RETRY_BUDGET && is_retryable_lance_conflict(&e) => {
continue;
}
Err(e) => return Err(OmniError::Lance(e.to_string())),
@ -815,11 +800,46 @@ pub async fn cleanup_all_tables(
db.ensure_schema_state_valid().await?;
db.ensure_schema_apply_idle("cleanup").await?;
// Version GC must never run while recovery still needs exact Lance
// transaction/version history to prove effect ownership or resume an
// interrupted compensation. Refuse before orphan reconciliation or any
// per-table cleanup so this operation is all-or-nothing with respect to the
// recovery-history floor. A read-write reopen resolves the sidecar first.
if !crate::db::manifest::list_sidecars(db.root_uri(), db.storage_adapter())
.await?
.is_empty()
{
return Err(OmniError::manifest_conflict(
"cleanup requires a clean recovery state; reopen the graph to run the \
recovery sweep before garbage-collecting versions",
));
}
crate::failpoints::maybe_fail(
crate::failpoints::names::CLEANUP_POST_RECOVERY_CHECK_PRE_GATES,
)?;
// Close the empty-check -> GC race. Mutation/load take schema then branch
// then table gates; current legacy sidecar writers take at least their table
// gates. Cleanup takes the conservative superset and holds it through every
// `cleanup_old_versions` call, then performs the authoritative sidecar check
// under those gates. Without this envelope a writer can arm+commit+fail after
// the fast check and GC can delete the exact transaction/version history
// Full recovery needs to prove ownership or Restore.
let _cleanup_schema_guard = db
.write_queue()
.acquire(&crate::db::manifest::schema_apply_serial_queue_key())
.await;
db.refresh_coordinator_only().await?;
db.ensure_schema_apply_not_locked("cleanup").await?;
let cleanup_catalog = db
.load_accepted_catalog_with_schema_gate_held()
.await?;
// Reclaim orphaned branch forks (from an incomplete prior `branch_delete`)
// before version GC. Authority-derived and idempotent; the eager
// best-effort reclaim in `branch_delete` covers the common case, this is
// the guaranteed backstop. Logged for observability.
let reconciled = reconcile_orphaned_branches(db).await?;
let reconciled = reconcile_orphaned_branches_with_catalog(db, &cleanup_catalog).await?;
if !reconciled.reclaimed.is_empty() {
tracing::info!(
count = reconciled.reclaimed.len(),
@ -835,13 +855,10 @@ pub async fn cleanup_all_tables(
);
}
let before_timestamp = options.older_than.map(|d| Utc::now() - d);
let keep_versions = options.keep_versions;
let resolved = db.resolved_branch_target(None).await?;
let snapshot = resolved.snapshot;
let table_tasks: Vec<_> = all_table_keys(&db.catalog())
let table_tasks: Vec<_> = all_table_keys(&cleanup_catalog)
.into_iter()
.filter_map(|table_key| {
let entry = snapshot.entry(&table_key)?;
@ -850,6 +867,38 @@ pub async fn cleanup_all_tables(
})
.collect();
// Schema gate stability means no native branch create/delete can change this
// set between enumeration and acquisition. Include main canonically as None;
// `all_branches` returns the user-facing "main" spelling.
let mut graph_branches = db
.coordinator
.read()
.await
.all_branches()
.await?
.into_iter()
.map(|branch| if branch == "main" { None } else { Some(branch) })
.collect::<Vec<_>>();
graph_branches.push(None);
graph_branches.sort();
graph_branches.dedup();
let _cleanup_branch_guards = db.write_queue().acquire_branches(&graph_branches).await;
let gc_queue_keys = db.table_queue_keys_for_branches(&graph_branches, &cleanup_catalog);
let _cleanup_table_guards = db.write_queue().acquire_many(&gc_queue_keys).await;
if !crate::db::manifest::list_sidecars(db.root_uri(), db.storage_adapter())
.await?
.is_empty()
{
return Err(OmniError::manifest_conflict(
"cleanup observed a recovery sidecar after acquiring its GC gates; reopen the graph \
read-write to recover before garbage-collecting versions",
));
}
let before_timestamp = options.older_than.map(|d| Utc::now() - d);
let keep_versions = options.keep_versions;
if table_tasks.is_empty() {
return Ok(Vec::new());
}
@ -868,9 +917,7 @@ pub async fn cleanup_all_tables(
// `cleanup_old_versions` is a Lance-only maintenance API not
// surfaced through `TableStorage` — see the optimize path
// above for the same rationale. Unwrap via `into_dataset()`.
let handle = storage
.open_dataset_head(&full_path, None)
.await?;
let handle = storage.open_dataset_head(&full_path, None).await?;
let ds = handle.into_dataset();
let before_version = keep_versions
.map(|n| ds.version().version.saturating_sub(n as u64))
@ -928,8 +975,9 @@ pub struct BranchReconcileStats {
pub failures: Vec<(String, String)>,
}
/// Drop every per-table and commit-graph Lance branch fork the manifest does
/// not reference.
/// Drop every per-table Lance branch fork the manifest does not reference.
/// Graph lineage lives in `__manifest`; the retired standalone commit datasets
/// have no branch-ref cleanup path here.
///
/// Two origins produce a manifest-unreferenced fork:
/// 1. A `branch_delete` flips the manifest authority (atomic) but a
@ -953,7 +1001,16 @@ pub struct BranchReconcileStats {
/// are dropped before parents (longest name first). Idempotent and authority-
/// derived: no-ops once reconciled, and degrades to finding nothing if a future
/// Lance atomic multi-dataset branch op prevents orphans from forming.
#[cfg(all(test, feature = "failpoints"))]
pub async fn reconcile_orphaned_branches(db: &Omnigraph) -> Result<BranchReconcileStats> {
let catalog = db.catalog();
reconcile_orphaned_branches_with_catalog(db, &catalog).await
}
async fn reconcile_orphaned_branches_with_catalog(
db: &Omnigraph,
catalog: &omnigraph_compiler::catalog::Catalog,
) -> Result<BranchReconcileStats> {
use std::collections::{HashMap, HashSet};
// Live manifest branches: the set whose per-table placements are
@ -969,7 +1026,7 @@ pub async fn reconcile_orphaned_branches(db: &Omnigraph) -> Result<BranchReconci
let resolved = db.resolved_branch_target(None).await?;
let snapshot = resolved.snapshot;
let table_targets: Vec<(String, String)> = all_table_keys(&db.catalog())
let table_targets: Vec<(String, String)> = all_table_keys(catalog)
.into_iter()
.filter_map(|table_key| {
let entry = snapshot.entry(&table_key)?;
@ -1021,11 +1078,12 @@ pub async fn reconcile_orphaned_branches(db: &Omnigraph) -> Result<BranchReconci
continue;
}
if !branch_snapshots.contains_key(&branch) {
let branch_snapshot =
match crate::failpoints::maybe_fail(crate::failpoints::names::CLEANUP_RESOLVE_BRANCH_SNAPSHOT) {
Ok(()) => db.snapshot_for_branch(Some(&branch)).await,
Err(injected) => Err(injected),
};
let branch_snapshot = match crate::failpoints::maybe_fail(
crate::failpoints::names::CLEANUP_RESOLVE_BRANCH_SNAPSHOT,
) {
Ok(()) => db.snapshot_for_branch(Some(&branch)).await,
Err(injected) => Err(injected),
};
match branch_snapshot {
Ok(snap) => {
branch_snapshots.insert(branch.clone(), snap);
@ -1083,7 +1141,7 @@ pub async fn reconcile_orphaned_branches(db: &Omnigraph) -> Result<BranchReconci
// skipped and recorded. (Cross-process writers remain the documented
// one-winner-CAS gap.) One key held at a time → no lock-order
// inversion vs multi-table `acquire_many` writers.
match super::table_ops::classify_fork_ref(db, &table_key, &branch).await {
match super::table_ops::classify_fork_ref(db, &table_key, &branch, None).await {
super::table_ops::ForkRefStatus::Orphan => {}
super::table_ops::ForkRefStatus::Legitimate => continue,
super::table_ops::ForkRefStatus::Indeterminate => {
@ -1101,7 +1159,9 @@ pub async fn reconcile_orphaned_branches(db: &Omnigraph) -> Result<BranchReconci
continue;
}
}
let outcome = match crate::failpoints::maybe_fail(crate::failpoints::names::CLEANUP_RECONCILE_FORK) {
let outcome = match crate::failpoints::maybe_fail(
crate::failpoints::names::CLEANUP_RECONCILE_FORK,
) {
Ok(()) => storage.force_delete_branch(&full_path, &branch).await,
Err(injected) => Err(injected),
};

30
crates/omnigraph/src/db/omnigraph/schema_apply.rs
View file

@ -152,6 +152,15 @@ where
// exists, so the heal can never observe it.
db.heal_pending_recovery_sidecars().await?;
// Process-local schema-control gate. RFC-022 mutation/load commit paths
// acquire this before their branch/table gates and retain it through
// publication. Taking it before the durable sentinel closes the old race in
// which schema apply could create the sentinel while a mutation already held
// a table queue, causing that mutation to advance Lance HEAD and only then
// discover the schema lock. The native sentinel remains the cross-handle /
// crash-visible authority; this queue removes the avoidable same-handle race.
let schema_gate_key = crate::db::manifest::schema_apply_serial_queue_key();
let _schema_gate = db.write_queue().acquire(&schema_gate_key).await;
acquire_schema_apply_lock(db).await?;
let result = apply_schema_with_lock(db, desired_schema_source, options, validate_catalog).await;
let release_result = release_schema_apply_lock(db).await;
@ -440,23 +449,17 @@ where
// practice. They exist for symmetry with the recovery reconciler, which
// acquires the same queues before any `Dataset::restore` it issues for
// SchemaApply sidecars.
let mut schema_apply_queue_keys: Vec<(String, Option<String>)> = recovery_pins
let schema_apply_queue_keys: Vec<(String, Option<String>)> = recovery_pins
.iter()
.map(|pin| (pin.table_key.clone(), pin.table_branch.clone()))
.collect();
// The serialization key the write-entry heal acquires before touching
// schema staging or a SchemaApply sidecar. Per-table keys alone don't
// cover a registration-only migration (no pins, but a sidecar and
// staging files on disk) — without this, a concurrent write's heal can
// promote this apply's staging files and publish its registrations out
// from under it. Acquired whenever a sidecar will be written, held
// through Phase D (the guards live to the end of this function).
// The outer `apply_schema` holds the schema-control serialization key from
// before sentinel creation through sentinel release. Per-table guards here
// therefore cover only the concrete table effects; acquiring the schema key
// again would deadlock because these queues are intentionally non-reentrant.
let writes_sidecar = !(recovery_pins.is_empty()
&& sidecar_registrations.is_empty()
&& sidecar_tombstones.is_empty());
if writes_sidecar {
schema_apply_queue_keys.push(crate::db::manifest::schema_apply_serial_queue_key());
}
let _schema_apply_queue_guards = db
.write_queue()
.acquire_many(&schema_apply_queue_keys)
@ -862,10 +865,7 @@ pub(super) async fn ensure_snapshot_entry_head_matches(
let dataset_uri = db.storage().dataset_uri(&entry.table_path);
let ds = db
.storage()
.open_dataset_head(
&dataset_uri,
entry.table_branch.as_deref(),
)
.open_dataset_head(&dataset_uri, entry.table_branch.as_deref())
.await?;
db.storage()
.ensure_expected_version(&ds, &entry.table_key, entry.table_version)

311
crates/omnigraph/src/db/omnigraph/table_ops.rs
View file

@ -118,8 +118,7 @@ pub(super) async fn ensure_indices_for_branch(
continue;
}
let full_path = format!("{}/{}", db.root_uri, entry.table_path);
if needs_index_work_node(db, type_name, &full_path, entry.table_branch.as_deref()).await?
{
if needs_index_work_node(db, type_name, &full_path, entry.table_branch.as_deref()).await? {
recovery_pins.push(crate::db::manifest::SidecarTablePin {
table_key,
table_path: full_path,
@ -213,14 +212,13 @@ pub(super) async fn ensure_indices_for_branch(
entry.table_version,
active_branch,
crate::db::MutationOpKind::SchemaRewrite,
false,
)
.await?
}
},
None => (
db.storage()
.open_dataset_head(&full_path, None)
.await?,
db.storage().open_dataset_head(&full_path, None).await?,
None,
),
};
@ -261,14 +259,13 @@ pub(super) async fn ensure_indices_for_branch(
entry.table_version,
active_branch,
crate::db::MutationOpKind::SchemaRewrite,
false,
)
.await?
}
},
None => (
db.storage()
.open_dataset_head(&full_path, None)
.await?,
db.storage().open_dataset_head(&full_path, None).await?,
None,
),
};
@ -296,7 +293,9 @@ pub(super) async fn ensure_indices_for_branch(
// (one commit_staged per index built) but the manifest publish below
// hasn't run. Used by
// `tests/failpoints.rs::ensure_indices_phase_b_failure_recovered_on_next_open`.
crate::failpoints::maybe_fail(crate::failpoints::names::ENSURE_INDICES_POST_PHASE_B_PRE_MANIFEST_COMMIT)?;
crate::failpoints::maybe_fail(
crate::failpoints::names::ENSURE_INDICES_POST_PHASE_B_PRE_MANIFEST_COMMIT,
)?;
if !updates.is_empty() {
commit_prepared_updates_on_branch(db, branch, &updates, None).await?;
@ -500,6 +499,16 @@ pub(crate) struct OpenedForMutation {
pub(crate) expected_version: u64,
pub(crate) full_path: String,
pub(crate) table_branch: Option<String>,
/// RFC-022 first-touch named-branch writes stage against the inherited
/// source snapshot and defer the durable Lance ref creation until after
/// their v3 recovery intent is armed in `StagedMutation::commit_all`.
pub(crate) deferred_fork: Option<DeferredTableFork>,
}
#[derive(Debug, Clone)]
pub(crate) struct DeferredTableFork {
pub(crate) source_entry: crate::db::SubTableEntry,
pub(crate) target_branch: String,
}
impl OpenedForMutation {
@ -590,6 +599,7 @@ pub(super) async fn open_for_mutation_on_branch(
expected_version: entry.table_version,
full_path,
table_branch: Some(active_branch.to_string()),
deferred_fork: None,
});
}
// Main branch, non-strict → no open. (Main never forks.)
@ -599,6 +609,7 @@ pub(super) async fn open_for_mutation_on_branch(
expected_version: entry.table_version,
full_path,
table_branch: None,
deferred_fork: None,
});
}
// Non-strict but the table isn't on the active branch yet — falls
@ -609,13 +620,19 @@ pub(super) async fn open_for_mutation_on_branch(
match resolved_branch.as_deref() {
None => {
let ds = db
.storage()
.open_dataset_head(&full_path, None)
.await?;
let ds = db.storage().open_dataset_head(&full_path, None).await?;
if op_kind.strict_pre_stage_version_check() {
db.storage()
.ensure_expected_version(&ds, table_key, entry.table_version)?;
if txn.is_some() && ds.version() != entry.table_version {
return Err(OmniError::manifest_read_set_changed(
format!("table_head:{table_key}"),
Some(entry.table_version.to_string()),
Some(ds.version().to_string()),
));
}
if txn.is_none() {
db.storage()
.ensure_expected_version(&ds, table_key, entry.table_version)?;
}
}
let version = ds.version();
Ok(OpenedForMutation {
@ -623,9 +640,28 @@ pub(super) async fn open_for_mutation_on_branch(
expected_version: version,
full_path,
table_branch: None,
deferred_fork: None,
})
}
Some(active_branch) => {
// RFC-022-enrolled mutation/load adapters must arm durable intent
// before creating a per-table Lance branch ref. Read and stage from
// the inherited source entry now; `commit_all` creates the target
// ref after its v3 sidecar is durable, then commits this transaction
// onto the new ref. Legacy writers retain the eager fork path below.
if txn.is_some() && entry.table_branch.as_deref() != Some(active_branch) {
let ds = db.storage().open_snapshot_at_entry(entry).await?;
return Ok(OpenedForMutation {
handle: Some(ds),
expected_version: entry.table_version,
full_path,
table_branch: Some(active_branch.to_string()),
deferred_fork: Some(DeferredTableFork {
source_entry: entry.clone(),
target_branch: active_branch.to_string(),
}),
});
}
let (ds, table_branch) = open_owned_dataset_for_branch_write(
db,
table_key,
@ -634,6 +670,7 @@ pub(super) async fn open_for_mutation_on_branch(
entry.table_version,
active_branch,
op_kind,
txn.is_some(),
)
.await?;
let version = ds.version();
@ -642,6 +679,7 @@ pub(super) async fn open_for_mutation_on_branch(
expected_version: version,
full_path,
table_branch,
deferred_fork: None,
})
}
}
@ -655,6 +693,7 @@ pub(super) async fn open_owned_dataset_for_branch_write(
entry_version: u64,
active_branch: &str,
op_kind: crate::db::MutationOpKind,
occ_enrolled: bool,
) -> Result<(SnapshotHandle, Option<String>)> {
match entry_branch {
Some(branch) if branch == active_branch => {
@ -663,8 +702,17 @@ pub(super) async fn open_owned_dataset_for_branch_write(
.open_dataset_head(full_path, Some(active_branch))
.await?;
if op_kind.strict_pre_stage_version_check() {
db.storage()
.ensure_expected_version(&ds, table_key, entry_version)?;
if occ_enrolled && ds.version() != entry_version {
return Err(OmniError::manifest_read_set_changed(
format!("table_head:{table_key}"),
Some(entry_version.to_string()),
Some(ds.version().to_string()),
));
}
if !occ_enrolled {
db.storage()
.ensure_expected_version(&ds, table_key, entry_version)?;
}
}
Ok((ds, Some(active_branch.to_string())))
}
@ -678,11 +726,19 @@ pub(super) async fn open_owned_dataset_for_branch_write(
let live = db.snapshot_for_branch(Some(active_branch)).await?;
if let Some(entry) = live.entry(table_key) {
if entry.table_branch.as_deref() == Some(active_branch) {
return Err(OmniError::manifest_expected_version_mismatch(
table_key,
entry_version,
entry.table_version,
));
return if occ_enrolled {
Err(OmniError::manifest_read_set_changed(
format!("table_head:{table_key}"),
Some(entry_version.to_string()),
Some(entry.table_version.to_string()),
))
} else {
Err(OmniError::manifest_expected_version_mismatch(
table_key,
entry_version,
entry.table_version,
))
};
}
}
// The fork advances Lance state before the manifest publish. The
@ -760,7 +816,32 @@ pub(crate) async fn classify_fork_ref(
db: &Omnigraph,
table_key: &str,
branch: &str,
excluding_operation_id: Option<&str>,
) -> ForkRefStatus {
// Deferred mutation/load forks are created only after their v3 sidecar is
// durable. Until the manifest publish places this table on `branch`, that
// sidecar is the only durable ownership record for the ref. Treat a
// matching pending intent as indeterminate rather than an orphan so neither
// destructive caller can steal a live writer's fork. The writer that owns
// the intent may exclude itself while reclaiming a genuinely stale ref it
// collided with; every other sidecar remains a hard stop. A list failure is
// likewise indeterminate -- cleanup must never turn missing authority into
// permission to delete.
let sidecars =
match crate::db::manifest::list_sidecars(db.root_uri(), db.storage_adapter()).await {
Ok(sidecars) => sidecars,
Err(_) => return ForkRefStatus::Indeterminate,
};
if sidecars.iter().any(|sidecar| {
Some(sidecar.operation_id.as_str()) != excluding_operation_id
&& sidecar.branch.as_deref() == Some(branch)
&& sidecar.tables.iter().any(|pin| {
pin.table_key == table_key && pin.table_branch.as_deref() == Some(branch)
})
}) {
return ForkRefStatus::Indeterminate;
}
// `classify.fresh_read` failpoint: simulate a transient failure of the
// fresh-authority read (no-op without the `failpoints` feature). Lets a
// test exercise the Indeterminate path — a read failure on a live branch
@ -819,13 +900,34 @@ pub(super) async fn reclaim_orphaned_fork_and_refork(
source_branch: Option<&str>,
source_version: u64,
active_branch: &str,
current_operation_id: Option<&str>,
) -> Result<SnapshotHandle> {
// A v3 mutation/load sidecar is written before its deferred fork. A
// manifest-unreferenced ref claimed by another pending operation is live,
// not an orphan: never force-delete it. Excluding our own operation lets a
// writer reclaim a genuinely stale pre-existing ref after its own intent is
// durable. A sidecar-list failure is indeterminate and therefore loud.
let sidecars = crate::db::manifest::list_sidecars(db.root_uri(), db.storage_adapter()).await?;
if let Some(owner) = sidecars.iter().find(|sidecar| {
Some(sidecar.operation_id.as_str()) != current_operation_id
&& sidecar.branch.as_deref() == Some(active_branch)
&& sidecar.tables.iter().any(|pin| {
pin.table_key == table_key && pin.table_branch.as_deref() == Some(active_branch)
})
}) {
return Err(OmniError::manifest_read_set_changed(
format!("fork_intent:{active_branch}:{table_key}"),
None,
Some(owner.operation_id.clone()),
));
}
// Self-validate against FRESH authority before destroying anything. Only an
// Orphan is reclaimable; a Legitimate status (a concurrent writer published
// a real fork despite the caller's possibly-cached proof) or an
// Indeterminate one (transient read) surfaces a retryable conflict rather
// than stranding the manifest at a version the recreated ref won't have.
match classify_fork_ref(db, table_key, active_branch).await {
match classify_fork_ref(db, table_key, active_branch, current_operation_id).await {
ForkRefStatus::Orphan => {}
ForkRefStatus::Legitimate => {
let actual = db
@ -834,6 +936,13 @@ pub(super) async fn reclaim_orphaned_fork_and_refork(
.ok()
.and_then(|s| s.entry(table_key).map(|e| e.table_version))
.unwrap_or(source_version);
if current_operation_id.is_some() {
return Err(OmniError::manifest_read_set_changed(
format!("table_head:{table_key}"),
Some(source_version.to_string()),
Some(actual.to_string()),
));
}
return Err(OmniError::manifest_expected_version_mismatch(
table_key,
source_version,
@ -1101,7 +1210,9 @@ async fn stage_and_commit_btree(
// to demonstrate that a stage-step failure in the staged-index
// path (`stage_create_btree_index` succeeded; `commit_staged` not
// yet called) leaves no Lance-HEAD drift on the touched table.
crate::failpoints::maybe_fail(crate::failpoints::names::ENSURE_INDICES_POST_STAGE_PRE_COMMIT_BTREE)?;
crate::failpoints::maybe_fail(
crate::failpoints::names::ENSURE_INDICES_POST_STAGE_PRE_COMMIT_BTREE,
)?;
let new_ds = db
.storage()
.commit_staged(ds.clone(), staged)
@ -1153,30 +1264,28 @@ async fn prepare_updates_for_commit(
branch: Option<&str>,
updates: &[crate::db::SubTableUpdate],
txn: Option<&crate::db::WriteTxn>,
// Post-`commit_staged` handles handed out by `StagedMutation::commit_all`
// (RFC-013 step 3b, collapse #4): table_key → the handle already open at
// its just-committed version. When a table's handle is present, the index
// build below reuses it and SKIPS the `reopen_for_mutation` open. Absent
// entries (other writers — schema apply, merge, ensure_indices, tests —
// pass `HashMap::new()`) keep the byte-identical `reopen_for_mutation`
// path. Delete tables ARE staged now (MR-A), so their handle is present
// like any other staged write.
mut committed_handles: std::collections::HashMap<String, SnapshotHandle>,
) -> Result<Vec<crate::db::SubTableUpdate>> {
if updates.is_empty() {
return Ok(Vec::new());
}
// With a `WriteTxn` the schema contract was validated once at capture, so
// reuse the pinned base entries (same per-branch manifest snapshot) instead
// of `snapshot_for_branch` (which re-runs `ensure_schema_state_valid`). Only
// the `entry(table_key).table_path` is read out of it here, identical to the
// no-txn path; the post-`commit_staged` index build below still reopens the
// dataset at its just-committed version. Without a txn, byte-identical.
let snapshot = match txn {
Some(txn) => txn.base.clone(),
None => db.snapshot_for_branch(branch).await?,
};
// RFC-022 mutation/load adapter: the physical effect envelope must be
// closed before its recovery sidecar is armed. Building indexes here can
// add one or several extra Lance commits (and vector-index creation is an
// inline-commit residual), so those commits cannot be represented as the
// staged data transaction promised by the sidecar. Indexes are derived
// state: enrolled mutation/load writes publish the exact data-table result
// and leave declared-index materialization to the existing
// ensure_indices/optimize reconciler. Other writers keep their historical
// behavior until their own effect adapters migrate.
if txn.is_some() {
return Ok(updates.to_vec());
}
// Enrolled mutation/load returned above: derived-index work is deliberately
// outside their exact physical effect envelope. Legacy callers retain the
// historical reopen/build path.
let snapshot = db.snapshot_for_branch(branch).await?;
let mut prepared = Vec::with_capacity(updates.len());
for update in updates {
@ -1190,38 +1299,26 @@ async fn prepare_updates_for_commit(
let mut prepared_update = update.clone();
if prepared_update.row_count > 0 {
let full_path = format!("{}/{}", db.root_uri, entry.table_path);
// Reuse the post-`commit_staged` handle when the caller handed one
// out (collapse #4): it is already open at exactly
// `prepared_update.table_version`, so the defense-in-depth strict
// re-check `reopen_for_mutation` would run is trivially satisfied
// and the open is redundant. When no handle is present (other
// writers, or any non-staged table), fall back to the byte-identical
// `reopen_for_mutation` path.
//
// Strict version check is correct on the fallback: this runs INSIDE
// Strict version check is correct here: this runs INSIDE
// the publisher commit path, after `commit_staged` already
// advanced Lance HEAD to `prepared_update.table_version`.
// The check is a defense-in-depth assertion that the
// dataset state matches what we just committed; not the
// pre-stage race the op-kind policy targets.
let mut ds = match committed_handles.remove(&prepared_update.table_key) {
Some(ds) => ds,
None => {
reopen_for_mutation(
db,
&prepared_update.table_key,
&full_path,
prepared_update.table_branch.as_deref(),
prepared_update.table_version,
crate::db::MutationOpKind::SchemaRewrite,
)
.await?
}
};
let mut ds = reopen_for_mutation(
db,
&prepared_update.table_key,
&full_path,
prepared_update.table_branch.as_deref(),
prepared_update.table_version,
crate::db::MutationOpKind::SchemaRewrite,
)
.await?;
// Any column not yet buildable (e.g. a vector column whose rows
// have null embeddings) is deferred and logged inside
// build_indices; a later ensure_indices/optimize materializes it.
// The load/mutate/merge commit must not fail on it.
// Legacy merge/test callers must not fail on it; enrolled
// mutation/load callers returned before this block.
let _pending =
build_indices_on_dataset(db, &prepared_update.table_key, &mut ds).await?;
let state = db.storage().table_state(&full_path, &ds).await?;
@ -1253,6 +1350,7 @@ async fn commit_prepared_updates(
Ok(manifest_version)
}
#[cfg(feature = "failpoints")]
async fn commit_prepared_updates_with_expected(
db: &Omnigraph,
updates: &[crate::db::SubTableUpdate],
@ -1303,6 +1401,7 @@ pub(super) async fn commit_prepared_updates_on_branch(
Ok(manifest_version)
}
#[cfg(feature = "failpoints")]
pub(super) async fn commit_prepared_updates_on_branch_with_expected(
db: &Omnigraph,
branch: Option<&str>,
@ -1356,14 +1455,8 @@ pub(super) async fn commit_updates(
.await
.current_branch()
.map(str::to_string);
let prepared = prepare_updates_for_commit(
db,
current_branch.as_deref(),
updates,
None,
std::collections::HashMap::new(),
)
.await?;
let prepared =
prepare_updates_for_commit(db, current_branch.as_deref(), updates, None).await?;
commit_prepared_updates(db, &prepared, None).await
}
@ -1390,20 +1483,60 @@ pub(super) async fn commit_updates_on_branch_with_expected(
updates: &[crate::db::SubTableUpdate],
expected_table_versions: &std::collections::HashMap<String, u64>,
actor_id: Option<&str>,
txn: Option<&crate::db::WriteTxn>,
committed_handles: std::collections::HashMap<String, SnapshotHandle>,
txn: &crate::db::WriteTxn,
lineage_intent: crate::db::manifest::LineageIntent,
) -> Result<u64> {
db.ensure_schema_apply_not_locked("write commit").await?;
let prepared =
prepare_updates_for_commit(db, branch, updates, txn, committed_handles).await?;
commit_prepared_updates_on_branch_with_expected(
db,
let prepared = prepare_updates_for_commit(db, branch, updates, Some(txn)).await?;
debug_assert_eq!(lineage_intent.actor_id.as_deref(), actor_id);
let changes = prepared
.iter()
.cloned()
.map(ManifestChange::Update)
.collect::<Vec<_>>();
let expectation = crate::db::manifest::GraphHeadExpectation::new(
branch,
&prepared,
expected_table_versions,
actor_id,
)
.await
txn.authority.branch_identifier.clone(),
txn.authority.graph_head.clone(),
);
let precondition = crate::db::manifest::PublishPrecondition::ExactGraphHead(expectation);
let current_branch = db
.coordinator
.read()
.await
.current_branch()
.map(str::to_string);
let requested_branch = branch.map(str::to_string);
let published = if requested_branch == current_branch {
db.coordinator
.write()
.await
.commit_changes_with_intent_and_expected(
&changes,
expected_table_versions,
lineage_intent,
&precondition,
)
.await?
} else {
let mut coordinator = match requested_branch.as_deref() {
Some(branch) => {
GraphCoordinator::open_branch(db.uri(), branch, Arc::clone(&db.storage)).await?
}
None => GraphCoordinator::open(db.uri(), Arc::clone(&db.storage)).await?,
};
coordinator
.commit_changes_with_intent_and_expected(
&changes,
expected_table_versions,
lineage_intent,
&precondition,
)
.await?
};
Ok(published.manifest_version)
}
pub(super) async fn invalidate_graph_index(db: &Omnigraph) {
@ -1452,7 +1585,7 @@ mod classify_fork_ref_tests {
.await
.unwrap();
assert_eq!(
classify_fork_ref(&db, "node:Company", "feature").await,
classify_fork_ref(&db, "node:Company", "feature", None).await,
ForkRefStatus::Legitimate,
"a manifest-placed fork must classify as Legitimate (never destroyed)"
);
@ -1467,7 +1600,7 @@ mod classify_fork_ref_tests {
ds.create_branch("feature", v, None).await.unwrap();
}
assert_eq!(
classify_fork_ref(&db, "node:Person", "feature").await,
classify_fork_ref(&db, "node:Person", "feature", None).await,
ForkRefStatus::Orphan,
"a ref the manifest does not place on the branch must classify as Orphan"
);
@ -1480,7 +1613,7 @@ mod classify_fork_ref_tests {
ds.create_branch("ghost", v, None).await.unwrap();
}
assert_eq!(
classify_fork_ref(&db, "node:Person", "ghost").await,
classify_fork_ref(&db, "node:Person", "ghost", None).await,
ForkRefStatus::Orphan,
"a ref for a branch absent from the manifest must classify as Orphan"
);

19
crates/omnigraph/src/db/recovery_audit.rs
View file

@ -1,10 +1,11 @@
//! Recovery audit row storage in `_graph_commit_recoveries.lance`.
//!
//! A standalone internal table (not catalog-tracked). Each successful
//! recovery sweep — roll-forward or roll-back — records one row here so
//! operators investigating a sidecar-attributed mutation can correlate
//! `omnigraph commit list --filter actor=omnigraph:recovery` with the
//! original actor whose mutation was rolled forward / back.
//! A standalone internal table (not catalog-tracked). Each completed recovery
//! action records one row here with the original actor and exact per-table
//! outcome. A v3 roll-forward preserves the interrupted writer's lineage and
//! actor, while rollback and legacy recovery lineage use
//! `omnigraph:recovery`; ordinary commit history is therefore not a complete
//! recovery log. This table currently has no public CLI query surface.
//!
//! This standalone table is additive: it doesn't bump
//! `INTERNAL_MANIFEST_SCHEMA_VERSION`. Folding `recovery_for_actor` and
@ -14,10 +15,10 @@
//! Atomicity caveat: append to `_graph_commit_recoveries.lance` is
//! sequential w.r.t. the recovery commit, which RFC-013 Phase 7 records in
//! `__manifest` (folded into the recovery publish CAS via `publish_recovery_commit`).
//! A crash between the publish and this audit append leaves a recovery commit
//! with no audit row. The recovery sweep tolerates it the same way (re-entry
//! sees `NoMovement` for already-restored / already-published tables; the audit
//! append is retried, minting a fresh recovery commit).
//! A crash between the publish and this audit append leaves a visible outcome
//! with no audit row. V3 sidecars carry fixed outcome ids and durable audit
//! payloads, so re-entry appends the missing row without minting another
//! commit; legacy sidecars retain their stale-sidecar cleanup behavior.
use std::sync::Arc;

44
crates/omnigraph/src/db/schema_state.rs
View file

@ -78,7 +78,21 @@ pub(crate) async fn load_or_bootstrap_schema_contract(
pub(crate) async fn validate_schema_contract(
root_uri: &str,
storage: Arc<dyn StorageAdapter>,
) -> Result<()> {
) -> Result<SchemaState> {
load_validated_schema_contract(root_uri, storage)
.await
.map(|(_, state)| state)
}
/// Load the accepted IR and its schema identity from one validated contract
/// read. Mutation/load preparation carries the catalog built from this exact IR
/// beside the identity in its `WriteTxn`; consulting the handle-global catalog
/// would let a long-lived handle combine a newly observed schema token with an
/// older in-memory plan.
pub(crate) async fn load_validated_schema_contract(
root_uri: &str,
storage: Arc<dyn StorageAdapter>,
) -> Result<(SchemaIR, SchemaState)> {
let current_source_ir = read_current_source_ir(root_uri, storage.as_ref()).await?;
let (persisted_ir, state) = match read_schema_contract(root_uri, storage.as_ref()).await? {
SchemaContractRead::Present { ir, state } => (ir, state),
@ -90,7 +104,33 @@ pub(crate) async fn validate_schema_contract(
};
validate_persisted_schema_contract(&persisted_ir, &state)?;
validate_current_source_matches(&state, &current_source_ir)
validate_current_source_matches(&state, &current_source_ir)?;
Ok((persisted_ir, state))
}
/// Read only the durable schema-identity marker. Schema apply promotes this
/// file after `_schema.pg` and `_schema.ir.json`, then releases its sentinel.
/// A capture path that already performed one full contract validation can use a
/// trailing marker read to detect the publish-before-promotion window without
/// paying for a second full source+IR parse.
pub(crate) async fn read_schema_state_identity(
root_uri: &str,
storage: &dyn StorageAdapter,
) -> Result<SchemaState> {
let text = storage.read_text(&schema_state_uri(root_uri)).await?;
let state = serde_json::from_str::<SchemaState>(&text).map_err(|err| {
schema_lock_conflict(format!(
"graph schema state in {} is invalid: {}",
SCHEMA_STATE_FILENAME, err
))
})?;
if state.format_version != SCHEMA_STATE_FORMAT_VERSION {
return Err(schema_lock_conflict(format!(
"graph schema state format {} is unsupported",
state.format_version
)));
}
Ok(state)
}
pub(crate) async fn write_schema_contract(

184
crates/omnigraph/src/db/write_queue.rs
View file

@ -1,15 +1,18 @@
//! Per-`(table_key, branch)` writer queues.
//!
//! These queues are the engine's write-serialization mechanism: the server
//! holds the engine as a lockless `Arc<Omnigraph>` (writes are `&self`), so
//! disjoint-key writes proceed concurrently and only writes to the same
//! `(table_key, branch_ref)` serialize here. This module owns the queue
//! data structure; callers in `MutationStaging::commit_all`, `branch_merge`,
//! `schema_apply`, `ensure_indices`, the fork path (first write to a table on
//! a branch — acquired before the fork, held through the manifest publish),
//! and the recovery reconciler acquire guards before any per-table Lance
//! commit. Serialization is in-process only; cross-process
//! writers on one graph remain one-winner-CAS at the manifest publish.
//! These queues are the engine's process-local, root-scoped write-serialization
//! mechanism. The server normally holds one lockless `Arc<Omnigraph>`, but
//! independently opened handles for the same canonical local root identity
//! (or the same opaque object-store URI) share this manager too. Legacy
//! writers serialize only on `(table_key, branch_ref)` so
//! disjoint keys can proceed concurrently. RFC-022-enrolled mutation/load
//! attempts additionally take a coarse branch effect gate because validation
//! may depend on tables they do not write. This module owns both queue classes;
//! callers in `MutationStaging::commit_all`, branch controls, `branch_merge`,
//! `schema_apply`, `ensure_indices`, cleanup, branch forking, and recovery acquire the applicable guards
//! before a Lance HEAD advance or destructive recovery action. Serialization
//! remains in-process only; cross-process writers on one graph remain
//! one-winner-CAS at publish.
//!
//! ## Why exclusive `tokio::sync::Mutex<()>` per key
//!
@ -32,13 +35,13 @@
//! ## Sorted-order acquisition
//!
//! `acquire_many` accepts a slice of keys and acquires them in
//! lexicographic order. Multi-table writers (mutation finalize,
//! branch_merge, future recovery reconciler) MUST go through
//! lexicographic order. Multi-table writers and control paths (mutation
//! finalize, branch merge, schema apply, maintenance, and recovery) MUST go through
//! `acquire_many` so all callers agree on acquisition order — this is
//! how lock-order inversion deadlock is prevented.
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
use std::sync::{Arc, Mutex, OnceLock, Weak};
use tokio::sync::{Mutex as AsyncMutex, OwnedMutexGuard};
@ -52,14 +55,26 @@ pub(crate) type TableQueueKey = (String, Option<String>);
/// Per-`(table_key, branch)` writer queue manager.
///
/// Lives on `Omnigraph` as `Arc<WriteQueueManager>` so HTTP handlers,
/// engine internals, the CLI binary, and future background reconcilers
/// (MR-870 recovery, MR-848 index) all reach it via the engine handle.
/// Every `Omnigraph` handle for one canonical root identity shares the same
/// manager via a process-global weak registry. This matters beyond HTTP's usual
/// `Arc<Omnigraph>` shape: a separately-opened handle can run recovery, and
/// Lance Restore/ref deletion must serialize with a live writer owned by the
/// first handle. The registry deliberately keys only by the queue root
/// identity; custom storage adapters for the same URI conservatively serialize
/// too.
#[derive(Default)]
pub(crate) struct WriteQueueManager {
/// Held only briefly per `acquire` call: clone out the per-key Arc,
/// release the std mutex, then await the per-key tokio Mutex.
queues: Mutex<HashMap<TableQueueKey, Arc<AsyncMutex<()>>>>,
/// Coarse per-branch effect gate used by RFC-022-enrolled writers.
///
/// This is deliberately separate from `queues`: a branch is authority,
/// not a synthetic table key. Enrolled writers acquire this gate before
/// any table queue and hold it through manifest publication. Legacy
/// writers continue to use only the table queues until their adapters are
/// migrated.
branch_queues: Mutex<HashMap<Option<String>, Arc<AsyncMutex<()>>>>,
}
impl WriteQueueManager {
@ -67,6 +82,24 @@ impl WriteQueueManager {
Self::default()
}
/// Return the process-wide queue manager for one canonical graph-root
/// identity.
/// Weak values avoid retaining every graph URI ever opened by a long-lived
/// multi-tenant process; lookup opportunistically removes dead entries.
pub(crate) fn for_root(root_identity: &str) -> Arc<Self> {
static REGISTRY: OnceLock<Mutex<HashMap<String, Weak<WriteQueueManager>>>> =
OnceLock::new();
let registry = REGISTRY.get_or_init(|| Mutex::new(HashMap::new()));
let mut roots = registry.lock().expect("root write queue registry poisoned");
if let Some(existing) = roots.get(root_identity).and_then(Weak::upgrade) {
return existing;
}
roots.retain(|_, manager| manager.strong_count() > 0);
let manager = Arc::new(Self::new());
roots.insert(root_identity.to_string(), Arc::downgrade(&manager));
manager
}
/// Get-or-create the per-key queue and clone its Arc.
fn slot(&self, key: &TableQueueKey) -> Arc<AsyncMutex<()>> {
let mut map = self.queues.lock().expect("write queue map poisoned");
@ -78,6 +111,55 @@ impl WriteQueueManager {
fresh
}
fn branch_slot(&self, branch: &Option<String>) -> Arc<AsyncMutex<()>> {
let mut map = self
.branch_queues
.lock()
.expect("branch write queue map poisoned");
if let Some(existing) = map.get(branch) {
return Arc::clone(existing);
}
let fresh = Arc::new(AsyncMutex::new(()));
map.insert(branch.clone(), Arc::clone(&fresh));
fresh
}
/// Acquire the coarse effect gate for one graph branch.
///
/// RFC-022-enrolled callers MUST acquire this before any per-table queue.
/// It is an in-process contention optimization only; publisher OCC and
/// recovery remain the correctness authorities.
pub(crate) async fn acquire_branch(
&self,
branch: Option<&str>,
) -> OwnedMutexGuard<()> {
let key = branch.map(str::to_string);
self.branch_slot(&key).lock_owned().await
}
/// Acquire several graph-branch control gates in one deterministic order.
///
/// Native branch create-from reads a source ref and mutates a target ref, so
/// both incarnations must remain stable across its fresh revalidation and
/// visibility point. Sorting/deduping gives branch control the same
/// deadlock-free acquisition rule as [`Self::acquire_many`] gives tables.
pub(crate) async fn acquire_branches(
&self,
branches: &[Option<String>],
) -> Vec<OwnedMutexGuard<()>> {
if branches.is_empty() {
return Vec::new();
}
let mut sorted = branches.to_vec();
sorted.sort();
sorted.dedup();
let mut guards = Vec::with_capacity(sorted.len());
for branch in sorted {
guards.push(self.branch_slot(&branch).lock_owned().await);
}
guards
}
/// Acquire exclusive access to the queue for one `(table_key, branch)`.
///
/// Blocks until the lock is available. Drop the returned guard to
@ -112,6 +194,8 @@ impl WriteQueueManager {
#[cfg(test)]
mod tests {
use super::*;
use crate::storage::write_queue_root_identity;
use std::path::PathBuf;
use std::time::{Duration, Instant};
use tokio::time::timeout;
@ -140,6 +224,23 @@ mod tests {
assert_eq!(guards.len(), 1);
}
#[tokio::test]
async fn acquire_branches_dedupes_main_and_named_keys() {
let qm = WriteQueueManager::new();
let guards = timeout(
Duration::from_secs(2),
qm.acquire_branches(&[
Some("feature".to_string()),
None,
Some("feature".to_string()),
None,
]),
)
.await
.expect("duplicate branch keys must not self-deadlock");
assert_eq!(guards.len(), 2);
}
#[tokio::test]
async fn acquire_many_sorts_keys_deterministically() {
// Two callers passing keys in different orders must acquire in
@ -232,4 +333,55 @@ mod tests {
.await
.expect("same-table-different-branch should not serialize");
}
#[test]
fn opaque_root_registry_shares_manager_across_handles() {
let root = format!("memory://write-queue-registry/{}", ulid::Ulid::new());
let first = WriteQueueManager::for_root(&root);
let second = WriteQueueManager::for_root(&root);
assert!(Arc::ptr_eq(&first, &second));
let other = WriteQueueManager::for_root(&format!("{root}/other"));
assert!(!Arc::ptr_eq(&first, &other));
}
#[test]
fn relative_and_absolute_local_roots_share_manager() {
let relative = PathBuf::from("target")
.join("write-queue-identities")
.join(ulid::Ulid::new().to_string())
.join("graph.omni");
let absolute = std::env::current_dir().unwrap().join(&relative);
let relative_identity = write_queue_root_identity(relative.to_str().unwrap()).unwrap();
let absolute_identity = write_queue_root_identity(absolute.to_str().unwrap()).unwrap();
assert_eq!(relative_identity, absolute_identity);
let first = WriteQueueManager::for_root(&relative_identity);
let second = WriteQueueManager::for_root(&absolute_identity);
assert!(Arc::ptr_eq(&first, &second));
}
#[cfg(unix)]
#[test]
fn real_and_symlinked_local_roots_share_manager_before_init() {
use std::os::unix::fs::symlink;
let parent = tempfile::tempdir().unwrap();
let real_parent = parent.path().join("real");
let alias_parent = parent.path().join("alias");
std::fs::create_dir(&real_parent).unwrap();
symlink(&real_parent, &alias_parent).unwrap();
// The graph suffix deliberately does not exist: init computes its
// queue identity before creating the graph directory.
let real_root = real_parent.join("future").join("graph.omni");
let alias_root = alias_parent.join("future").join("graph.omni");
let real_identity = write_queue_root_identity(real_root.to_str().unwrap()).unwrap();
let alias_identity = write_queue_root_identity(alias_root.to_str().unwrap()).unwrap();
assert_eq!(real_identity, alias_identity);
let first = WriteQueueManager::for_root(&real_identity);
let second = WriteQueueManager::for_root(&alias_identity);
assert!(Arc::ptr_eq(&first, &second));
}
}

62
crates/omnigraph/src/error.rs
View file

@ -22,6 +22,15 @@ pub enum ManifestConflictDetails {
expected: u64,
actual: u64,
},
/// A logical authority value captured during write preparation changed
/// before the manifest visibility decision. Unlike a touched-table
/// version mismatch, this may name a read-only dependency such as the
/// target branch's graph head or schema identity.
ReadSetChanged {
member: String,
expected: Option<String>,
actual: Option<String>,
},
/// Lance's row-level CAS rejected the publish because a concurrent writer
/// landed a row with the same `object_id`. Distinct from
/// `ExpectedVersionMismatch`: the caller's expectations (if any) still
@ -85,6 +94,16 @@ pub enum OmniError {
Manifest(ManifestError),
#[error("merge conflicts: {0:?}")]
MergeConflicts(Vec<MergeConflict>),
/// A durable recovery intent overlaps this write. Its physical effects may
/// already have landed, or it may still be armed before its first effect;
/// either way the sidecar named by `operation_id` must be resolved before
/// the caller retries. Treating this as ordinary OCC would let a writer
/// advance around unresolved commit ownership.
#[error("recovery required for operation {operation_id}: {reason}")]
RecoveryRequired {
operation_id: String,
reason: String,
},
/// Engine-layer policy enforcement (MR-722). Wraps either a policy
/// denial ("you can't do that") or a policy-evaluation failure
/// ("the policy engine itself blew up"). The HTTP layer maps
@ -103,6 +122,16 @@ pub enum OmniError {
}
impl OmniError {
pub(crate) fn is_read_set_changed(&self) -> bool {
matches!(
self,
Self::Manifest(ManifestError {
details: Some(ManifestConflictDetails::ReadSetChanged { .. }),
..
})
)
}
pub fn manifest(message: impl Into<String>) -> Self {
Self::Manifest(ManifestError::new(ManifestErrorKind::BadRequest, message))
}
@ -146,4 +175,37 @@ impl OmniError {
.with_details(ManifestConflictDetails::RowLevelCasContention),
)
}
pub fn manifest_read_set_changed(
member: impl Into<String>,
expected: Option<String>,
actual: Option<String>,
) -> Self {
let member = member.into();
let message = format!(
"write authority '{}' changed during preparation (expected {}, current {}) — reprepare from the current branch state",
member,
expected.as_deref().unwrap_or("<absent>"),
actual.as_deref().unwrap_or("<absent>"),
);
Self::Manifest(
ManifestError::new(ManifestErrorKind::Conflict, message).with_details(
ManifestConflictDetails::ReadSetChanged {
member,
expected,
actual,
},
),
)
}
pub fn recovery_required(
operation_id: impl Into<String>,
reason: impl Into<String>,
) -> Self {
Self::RecoveryRequired {
operation_id: operation_id.into(),
reason: reason.into(),
}
}
}

169
crates/omnigraph/src/exec/merge.rs
View file

@ -635,16 +635,16 @@ fn row_signature(batch: &RecordBatch, row: usize) -> Result<String> {
/// it is validated like `AdoptWithDelta`; only an empty-delta adopt is skipped.
async fn build_merge_changeset(
db: &Omnigraph,
catalog: &Catalog,
candidates: &HashMap<String, CandidateTableState>,
) -> Result<crate::validate::ChangeSet> {
let catalog = db.catalog();
let mut changeset = crate::validate::ChangeSet::new();
for (table_key, candidate) in candidates {
// Validation reads only id/src/dst + scalar constraint columns; project
// out Vector/Blob so the change-set never holds embeddings (holding the
// delta with embeddings would re-introduce the memory pressure the
// streaming append exists to avoid).
let projection = validation_projection(&catalog, table_key);
let projection = validation_projection(catalog, table_key);
let projection: Vec<&str> = projection.iter().map(String::as_str).collect();
let mut change = crate::validate::TableChange::default();
match candidate {
@ -736,7 +736,7 @@ async fn scan_staged_for_validation(
}
async fn validate_merge_candidates(
db: &Omnigraph,
catalog: &Catalog,
target_snapshot: &Snapshot,
changeset: &crate::validate::ChangeSet,
) -> Result<()> {
@ -746,9 +746,9 @@ async fn validate_merge_candidates(
// uniqueness, edge-RI, and cardinality all route through one evaluator shared
// with (eventually) the write path — closing the merge-vs-write drift.
let committed = crate::validate::CommittedState::merge(target_snapshot);
let constraints = crate::validate::constraints_for(&db.catalog());
let constraints = crate::validate::constraints_for(catalog);
let violations =
crate::validate::evaluate(&constraints, changeset, &committed, &db.catalog()).await?;
crate::validate::evaluate(&constraints, changeset, &committed, catalog).await?;
if violations.is_empty() {
Ok(())
} else {
@ -1233,13 +1233,6 @@ impl Omnigraph {
actor_id,
)?;
self.ensure_schema_apply_idle("branch_merge").await?;
// Converge any pending recovery sidecar before the merge
// captures its target snapshot: the merge's publish would
// otherwise make the drifted Phase-B commit visible as an
// unattributed side effect (manifest catches up to HEAD with no
// recovery audit row) and leave the stale sidecar behind. Runs
// before the merge's own sidecar exists.
self.heal_pending_recovery_sidecars().await?;
self.branch_merge_impl(source, target, actor_id).await
}
@ -1263,6 +1256,34 @@ impl Omnigraph {
));
}
let relevant_branches = [source_branch.as_deref(), target_branch.as_deref()];
// Branch merge is still a legacy per-table publisher, but its graph-ref
// authority must be stable for the complete prepare -> publish window.
// First converge or reject relevant recovery intent, then join the same
// root-shared schema -> branch order used by native branch controls.
// Holding both branch gates through publication prevents a target
// delete/recreate from reusing the branch name underneath a plan (ABA).
self.heal_pending_recovery_sidecars_for_write(&relevant_branches)
.await?;
let _schema_guard = self
.write_queue()
.acquire(&crate::db::manifest::schema_apply_serial_queue_key())
.await;
let _branch_guards = self
.write_queue()
.acquire_branches(&[source_branch.clone(), target_branch.clone()])
.await;
self.ensure_no_pending_recovery_sidecars_under_gates(
&relevant_branches,
"branch_merge",
)
.await?;
self.refresh_coordinator_only().await?;
self.ensure_schema_apply_not_locked("branch_merge").await?;
let merge_catalog = self
.load_accepted_catalog_with_schema_gate_held()
.await?;
let source_head_commit_id = self
.head_commit_id_for_branch(source_branch.as_deref())
.await?
@ -1298,6 +1319,15 @@ impl Omnigraph {
))
.await?
.snapshot;
let target_snapshot = self
.resolved_target(ReadTarget::Branch(
target_branch.clone().unwrap_or_else(|| "main".to_string()),
))
.await?
.snapshot;
crate::failpoints::maybe_fail(
crate::failpoints::names::BRANCH_MERGE_POST_AUTHORITY_CAPTURE,
)?;
// Hold the merge-exclusive mutex across the full swap → operate
// → restore window. Two concurrent branch_merge calls would
// otherwise interleave their three separate `coordinator.write()`
@ -1316,6 +1346,10 @@ impl Omnigraph {
.branch_merge_on_current_target(
&base_snapshot,
&source_snapshot,
&target_snapshot,
merge_catalog.as_ref(),
source_branch.as_deref(),
target_branch.as_deref(),
&target_head_commit_id,
&source_head_commit_id,
is_fast_forward,
@ -1353,8 +1387,8 @@ impl Omnigraph {
// `crates/omnigraph/tests/failpoints.rs`.
//
// Err-path refresh is best-effort: the merge body's error
// (typically the structured `manifest_conflict` from the
// post_queue_snapshot drift check) is the value the caller
// (typically the structured read-set conflict from the fresh
// post-table-gate manifest check) is the value the caller
// needs to see. A refresh-time storage error would replace
// that with a less informative error; the next op or the next
// `Omnigraph::open` will re-sync the coord anyway.
@ -1378,13 +1412,15 @@ impl Omnigraph {
&self,
base_snapshot: &Snapshot,
source_snapshot: &Snapshot,
target_snapshot: &Snapshot,
catalog: &Catalog,
source_branch: Option<&str>,
target_branch: Option<&str>,
target_head_commit_id: &str,
source_head_commit_id: &str,
is_fast_forward: bool,
actor_id: Option<&str>,
) -> Result<MergeOutcome> {
let target_snapshot = self.snapshot().await;
let mut table_keys = HashSet::new();
for entry in base_snapshot.entries() {
table_keys.insert(entry.table_key.clone());
@ -1415,10 +1451,10 @@ impl Omnigraph {
if same_manifest_state(base_entry, target_entry) {
let candidate = classify_adopt(
self,
&self.catalog(),
catalog,
base_snapshot,
source_snapshot,
&target_snapshot,
target_snapshot,
table_key,
)
.await?;
@ -1428,10 +1464,10 @@ impl Omnigraph {
if let Some(staged) = stage_streaming_table_merge(
table_key,
&self.catalog(),
catalog,
base_snapshot,
source_snapshot,
&target_snapshot,
target_snapshot,
&mut conflicts,
)
.await?
@ -1447,8 +1483,8 @@ impl Omnigraph {
return Err(OmniError::MergeConflicts(conflicts));
}
let changeset = build_merge_changeset(self, &candidates).await?;
validate_merge_candidates(self, &target_snapshot, &changeset).await?;
let changeset = build_merge_changeset(self, catalog, &candidates).await?;
validate_merge_candidates(catalog, target_snapshot, &changeset).await?;
// Recovery sidecar: protect the per-table commit_staged loop.
// Pin `RewriteMerged` and `AdoptWithDelta` candidates — both advance
@ -1469,54 +1505,48 @@ impl Omnigraph {
// HEAD unpinned — is closed: `classify_adopt` pre-computes the delta, so a
// HEAD-advancing adopt is `AdoptWithDelta` (pinned here) and an empty-delta
// adopt stays `AdoptSourceState`.
// Acquire per-(table_key, target_branch) queues for every table
// touched by the merge plan. Sorted-order acquisition prevents
// lock-order inversion against concurrent multi-table writers.
// The active branch (set by the caller's `swap_coordinator_for_branch`)
// is the merge target; queue keys are scoped to it because a
// branch_merge writes only to the target branch.
//
// Held across the per-table publish loop and the manifest
// commit + record_merge_commit calls below, so no concurrent
// writer to a touched (table, target_branch) can interleave
// between our commit_staged and our publish.
let active_branch_for_keys = self.active_branch().await;
let merge_queue_keys: Vec<(String, Option<String>)> = ordered_table_keys
.iter()
.filter(|table_key| {
matches!(
candidates.get(*table_key),
Some(CandidateTableState::RewriteMerged(_))
| Some(CandidateTableState::AdoptSourceState { .. })
| Some(CandidateTableState::AdoptWithDelta(_))
)
})
.map(|table_key| (table_key.clone(), active_branch_for_keys.clone()))
.collect();
// This bridge still coexists with legacy maintenance writers that take
// only `(table, branch)` queues. Acquire the conservative all-catalog
// envelope for BOTH source and target, in the global sorted order, then
// re-run the sidecar barrier and re-read both manifest branches before
// Phase A. Planning remains outside table queues, but no plan derived
// from a stale source/target snapshot can cross into physical effects.
let active_branch_for_keys = target_branch.map(str::to_string);
let merge_branches = [
source_branch.map(str::to_string),
active_branch_for_keys.clone(),
];
let merge_queue_keys = self.table_queue_keys_for_branches(&merge_branches, catalog);
let _merge_queue_guards = self.write_queue().acquire_many(&merge_queue_keys).await;
let post_queue_snapshot = self.snapshot().await;
for table_key in &ordered_table_keys {
let Some(candidate) = candidates.get(table_key) else {
continue;
};
if !matches!(
candidate,
CandidateTableState::RewriteMerged(_)
| CandidateTableState::AdoptSourceState { .. }
| CandidateTableState::AdoptWithDelta(_)
) {
continue;
}
let expected = target_snapshot.entry(table_key).map(|e| e.table_version);
let current = post_queue_snapshot
.entry(table_key)
.map(|e| e.table_version);
if expected != current {
return Err(OmniError::manifest_expected_version_mismatch(
table_key.clone(),
expected.unwrap_or(0),
current.unwrap_or(0),
self.ensure_no_pending_recovery_sidecars_under_gates(
&[source_branch, target_branch],
"branch_merge after acquiring source/target table gates",
)
.await?;
let fresh_source_snapshot = self
.fresh_snapshot_for_branch_unchecked(source_branch)
.await?;
let fresh_target_snapshot = self
.fresh_snapshot_for_branch_unchecked(target_branch)
.await?;
for (member, prepared, current) in [
("source", source_snapshot, &fresh_source_snapshot),
("target", target_snapshot, &fresh_target_snapshot),
] {
if prepared.version() != current.version() {
return Err(OmniError::manifest_read_set_changed(
format!(
"branch_merge_{member}:{}",
if member == "source" {
source_branch.unwrap_or("main")
} else {
target_branch.unwrap_or("main")
}
),
Some(prepared.version().to_string()),
Some(current.version().to_string()),
));
}
}
@ -1607,7 +1637,7 @@ impl Omnigraph {
};
let update = match candidate_state {
CandidateTableState::AdoptSourceState { .. } => {
publish_adopted_source_state(self, source_snapshot, &target_snapshot, table_key)
publish_adopted_source_state(self, source_snapshot, target_snapshot, table_key)
.await?
}
CandidateTableState::AdoptWithDelta(delta) => {
@ -1631,6 +1661,9 @@ impl Omnigraph {
// `updates` carry the real per-table final versions (multiple
// commit_staged calls per table, so not derivable from `post_commit_pin`
// alone). A failure here leaves the unconfirmed sidecar → roll back.
crate::failpoints::maybe_fail(
crate::failpoints::names::BRANCH_MERGE_POST_EFFECTS_PRE_CONFIRM,
)?;
if let Some((sidecar, _)) = recovery.as_mut() {
let confirmed_versions: std::collections::HashMap<String, u64> = updates
.iter()

525
crates/omnigraph/src/exec/mutation.rs
View file

@ -382,21 +382,13 @@ fn predicate_to_sql(
/// Replace specific columns in a RecordBatch with new literal values.
///
/// Blob columns may or may not be present in `batch` depending on the
/// caller's scan projection:
/// - If `batch` does NOT contain a blob column AND it has no assignment,
/// the column is OMITTED from the output. `merge_insert` leaves it
/// untouched.
/// - If `batch` DOES contain a blob column AND it has no assignment, the
/// column is COPIED to the output. This enables coalescing of
/// different-shape updates into a single full-schema merge batch (the
/// per-table accumulator in `MutationStaging` requires consistent
/// schemas across pending batches for `concat_batches`). The
/// round-tripping cost is acceptable for typical agent-driven
/// mutations; tables with large blobs and unassigned-blob updates may
/// want to be split into separate queries.
/// - If a blob column has a string-URI assignment, build the blob array
/// inline.
/// Blob-bearing updates always arrive with the full logical schema. Committed
/// blob payloads were materialized by the caller and rebuilt as logical
/// `Struct<data,uri>` arrays; pending batches already have that shape. An
/// unassigned blob is copied through, while an assigned string URI is rebuilt
/// with the same blob writer used by inserts. Consequently every update batch
/// has the catalog schema and can safely share one pending merge stream with
/// inserts and earlier updates.
fn apply_assignments(
full_schema: &SchemaRef,
batch: &RecordBatch,
@ -404,8 +396,6 @@ fn apply_assignments(
blob_properties: &HashSet<String>,
) -> Result<RecordBatch> {
let mut columns: Vec<ArrayRef> = Vec::with_capacity(full_schema.fields().len());
let mut out_fields: Vec<Field> = Vec::with_capacity(full_schema.fields().len());
for field in full_schema.fields().iter() {
if blob_properties.contains(field.name()) {
if let Some(Literal::String(uri)) = assignments.get(field.name()) {
@ -414,26 +404,25 @@ fn apply_assignments(
for _ in 0..batch.num_rows() {
crate::loader::append_blob_value(&mut builder, uri)?;
}
let blob_field = lance::blob::blob_field(field.name(), true);
out_fields.push(blob_field);
columns.push(
builder
.finish()
.map_err(|e| OmniError::Lance(e.to_string()))?,
);
} else if let Some(col) = batch.column_by_name(field.name()) {
// Unassigned but scan included it: copy through (writes
// back the same blob, no observable change but uniform
// schema for the accumulator).
let blob_field = lance::blob::blob_field(field.name(), field.is_nullable());
out_fields.push(blob_field);
} else {
// Unassigned: the materializing scan must have normalized the
// committed value (or pending value) to the logical blob
// schema, so copying it preserves both bytes and full-schema
// merge compatibility.
let col = batch.column_by_name(field.name()).ok_or_else(|| {
OmniError::Lance(format!(
"blob column '{}' not found in full-schema mutation scan",
field.name()
))
})?;
columns.push(col.clone());
}
// else: scan did not include this blob column and no
// assignment — omit. Caller's accumulator must accept the
// narrower schema (legacy single-merge_insert path).
} else if let Some(lit) = assignments.get(field.name()) {
out_fields.push(field.as_ref().clone());
columns.push(literal_to_typed_array(
lit,
field.data_type(),
@ -446,13 +435,11 @@ fn apply_assignments(
field.name()
))
})?;
out_fields.push(field.as_ref().clone());
columns.push(col.clone());
}
}
RecordBatch::try_new(Arc::new(Schema::new(out_fields)), columns)
.map_err(|e| OmniError::Lance(e.to_string()))
RecordBatch::try_new(full_schema.clone(), columns).map_err(|e| OmniError::Lance(e.to_string()))
}
// ─── Mutation execution ──────────────────────────────────────────────────────
@ -461,15 +448,15 @@ use super::staging::{MutationStaging, PendingMode};
/// Open a sub-table dataset for read or staged write within the current
/// mutation query, capturing pre-write metadata in `staging` on first touch.
/// The captured version is the publisher's CAS fence at end-of-query
/// (per-table OCC).
/// The captured table version is the physical staging baseline. The publisher's
/// logical CAS fence is the enclosing `WriteTxn` authority: native branch
/// identity, exact optional graph head, and accepted schema identity.
///
/// On first touch, opens the dataset at HEAD on the requested branch
/// via `open_for_mutation_on_branch`, which compares Lance HEAD against
/// the manifest's pinned version — that fence is the engine's
/// publisher-style OCC catching cross-writer drift before we make any
/// changes. For delete-only queries, this strict open is also the uncovered
/// drift guard.
/// On first touch, resolves the table from the transaction's pinned base.
/// Strict read-modify-write operations open that exact version and retain the
/// early HEAD-vs-pin drift guard; Insert/Merge may skip the physical open and
/// carry only a reclaimable stage plan. Neither path substitutes for the final
/// branch-wide token check under the effect gates.
///
/// On subsequent touches *within the same query*, Lance HEAD has not moved
/// since first touch — inserts, updates AND deletes all stage their work and
@ -481,8 +468,7 @@ use super::staging::{MutationStaging, PendingMode};
/// touch records another predicate (`record_delete`), and `stage_all` combines
/// them into one staged delete — there is no post-inline-commit reopen to
/// special-case anymore.
impl Omnigraph {
}
impl Omnigraph {}
async fn open_table_for_mutation(
db: &Omnigraph,
@ -495,8 +481,8 @@ async fn open_table_for_mutation(
// `open_for_mutation_on_branch` returns the expected version even when it
// skips the open (collapse #1, the non-strict insert/merge path): the version
// is the pinned base's, identical to the opened handle's `.version()`. Use it
// directly for `ensure_path` so the no-open path still captures the publisher
// CAS fence.
// directly for `ensure_path` so the no-open path retains its exact physical
// staging baseline; the branch-wide WriteTxn is the publisher CAS fence.
let opened = db
.open_for_mutation_on_branch(branch, table_key, op_kind, txn)
.await?;
@ -512,6 +498,7 @@ async fn open_table_for_mutation(
table_key,
opened.full_path.clone(),
opened.table_branch.clone(),
opened.deferred_fork.clone(),
opened.expected_version,
op_kind,
);
@ -648,18 +635,18 @@ impl Omnigraph {
txn: &crate::db::WriteTxn,
) -> Result<()> {
// RI/uniqueness read the write's already-validated pinned base (`txn.base`),
// NOT a fresh `snapshot_for_branch` — which would re-run the schema-contract
// validation the WriteTxn already did once (RFC-013 step 3b capture-once).
// Cardinality reads LIVE HEAD per edge table (the #298 stale-handle fix) via
// the live opener in `CommittedState::write`.
// NOT a fresh `snapshot_for_branch` — per-table resolution must not add a
// schema-contract validation beyond capture + the pre-effect gate.
// Cardinality reads a fresh manifest-visible branch snapshot (the #298
// stale-handle fix) via `CommittedState::write`; it never follows an
// unpublished raw Lance HEAD or a not-yet-created first-touch ref.
let committed =
crate::validate::CommittedState::write(&txn.base, self, txn.branch.as_deref());
// `to_changeset` carries both constructive batches and the ids the delete
// ops captured from their own scans (`deleted_ids`), so the evaluator
// recounts the srcs a delete empties (`@card`) and sees removed rows for
// RI — the faithful change-set the merge path also builds.
crate::validate::validate_changeset(&staging.to_changeset(), &committed, &self.catalog())
.await
crate::validate::validate_changeset(&staging.to_changeset(), &committed, &txn.catalog).await
}
async fn mutate_with_current_actor(
@ -670,15 +657,51 @@ impl Omnigraph {
params: &ParamMap,
actor_id: Option<&str>,
) -> Result<MutationResult> {
// Converge any pending recovery sidecar (a previously failed
// writer's Phase B → Phase C residual) before executing: the
// inline delete path advances Lance HEAD during execution and
// the staged path's commit-time drift guard refuses
// sidecar-covered drift, so a long-lived handle must heal here
// — not at restart. One `list_dir` when no sidecars exist (the
// steady state). MUST run before `open_write_txn` below — the heal
// may advance the manifest, so the pinned base must be captured after.
self.heal_pending_recovery_sidecars().await?;
const MAX_PRE_EFFECT_REPREPARES: usize = 32;
// Resolve request-scoped values such as now() once so a safe
// pre-effect retry does not change the logical input.
let resolved_params = enrich_mutation_params(params)?;
for attempt in 0..=MAX_PRE_EFFECT_REPREPARES {
let mut retryable = false;
match self
.mutate_one_attempt(
branch,
query_source,
query_name,
&resolved_params,
actor_id,
&mut retryable,
)
.await
{
Err(err)
if retryable
&& err.is_read_set_changed()
&& attempt < MAX_PRE_EFFECT_REPREPARES =>
{
tracing::debug!(
attempt = attempt + 1,
branch,
"prepared mutation authority changed before effects; repreparing"
);
self.refresh().await?;
}
result => return result,
}
}
unreachable!("bounded mutation retry loop always returns")
}
async fn mutate_one_attempt(
&self,
branch: &str,
query_source: &str,
query_name: &str,
params: &ParamMap,
actor_id: Option<&str>,
retryable: &mut bool,
) -> Result<MutationResult> {
let requested = Self::normalize_branch_name(branch)?;
// Reject internal `__run__*` / system-prefixed branches at the
// public write boundary. Direct-publish paths assert this
@ -687,62 +710,42 @@ impl Omnigraph {
if let Some(name) = requested.as_deref() {
crate::db::ensure_public_branch_ref(name, "mutate")?;
}
// Capture-once write transaction (RFC-013 step 3b). `open_write_txn`
// validates the schema contract ONCE (it resolves the branch target,
// whose first line is `ensure_schema_state_valid`) and pins the base
// snapshot for this write. Threaded as `Some(&txn)` through execution,
// staging commit, and the manifest publish so the per-table opens and
// the commit-time OCC re-read reuse the pinned base instead of
// re-validating the contract at every resolve point. Captured AFTER the
// recovery heal (which may advance the manifest) and AFTER `requested`
// is known so it pins the post-heal snapshot for the correct branch.
// Stage A: converge any roll-forward-eligible sidecars, then close the
// barrier on every unresolved intent for this graph branch. This MUST
// run before `open_write_txn`: healing may advance the manifest, and a
// deferred Armed intent remains ownership even when no table HEAD moved.
self.heal_pending_recovery_sidecars_for_write(&[requested.as_deref()])
.await?;
// Capture one branch-wide write authority after the recovery barrier:
// native branch identity, exact optional graph head, accepted schema
// identity/catalog, and the base table snapshot. Execution, validation,
// staging, and publication all use this immutable attempt. `commit_all`
// revalidates the complete token under the root-shared schema → branch →
// sorted-table gates before it arms recovery or advances Lance HEAD.
let txn = self.open_write_txn(requested.as_deref()).await?;
let resolved_params = enrich_mutation_params(params)?;
let resolved_params = params.clone();
// Per-query staging accumulator. Inserts and updates push batches
// into `pending`; deletes push predicates into `delete_predicates`. At
// end-of-query, `finalize` issues one `stage_*` + `commit_staged` per
// touched table (inserts/updates/deletes alike), then the publisher
// commits the manifest atomically across all touched tables. Branch is
// threaded explicitly — no coordinator swap.
// Per-query staging accumulator. Inserts and updates push batches into
// `pending`; deletes push predicates into `delete_predicates`. At the
// boundary, `stage_all` prepares one exact transaction per touched table
// and `commit_all` records those identities in a durable schema-v3
// recovery intent before independently advancing the table HEADs. The
// publisher then makes the complete result graph-visible in one manifest
// CAS. Branch is threaded explicitly — no coordinator swap.
let mut staging = MutationStaging::default();
// Lower + validate up front so the touched-table set is known before
// execution. A lowering/validation error returns exactly as it did
// when this happened inside execute_named_mutation.
let ir = self.lower_named_mutation(query_source, query_name)?;
// Up-front fork-queue acquisition (see the loader for the full
// rationale): if this mutation will fork any touched table onto a
// non-main branch, acquire the per-(table, branch) write queues for
// every touched table before the first fork and hold them through the
// publish, so the orphan-fork reclaim can't race a concurrent
// in-process fork. The touched set is derived from the lowered IR.
let fork_queue_guards: Option<(
Vec<(String, Option<String>)>,
Vec<tokio::sync::OwnedMutexGuard<()>>,
)> = if let Some(active) = requested.as_deref() {
let snapshot = self.snapshot_for_branch(Some(active)).await?;
let touched: Vec<(String, Option<String>)> = self
.touched_table_keys(&ir)
.into_iter()
.map(|k| (k, Some(active.to_string())))
.collect();
let needs_fork = touched.iter().any(|(table_key, _)| {
snapshot
.entry(table_key)
.map(|e| e.table_branch.as_deref() != Some(active))
.unwrap_or(false)
});
if needs_fork {
let guards = self.write_queue().acquire_many(&touched).await;
Some((touched, guards))
} else {
None
}
} else {
None
};
let ir = self.lower_named_mutation(&txn.catalog, query_source, query_name)?;
// Only an insert-only mutation is safe to replay automatically after a
// pre-effect authority mismatch. Update/Delete keep strict caller-visible
// `ReadSetChanged`; replaying their stale read-modify-write plan would be
// a semantic rebase rather than a fresh execution contract.
*retryable = ir
.ops
.iter()
.all(|op| matches!(op, MutationOpIR::Insert { .. }));
let exec_result = self
.execute_named_mutation(
@ -750,7 +753,7 @@ impl Omnigraph {
&resolved_params,
requested.as_deref(),
&mut staging,
Some(&txn),
&txn,
)
.await;
@ -760,62 +763,74 @@ impl Omnigraph {
Ok(total) => {
self.validate_staged_mutation(&staging, &txn).await?;
let staged = staging.stage_all(self, requested.as_deref()).await?;
// `_queue_guards` holds per-(table_key, branch) write
// queues acquired inside `commit_all`. Held across the
// manifest publish below so no concurrent writer can
// interleave between our commit_staged and our publish
// (which would correctly fail our CAS but leave Lance
// HEAD advanced — the residual class MR-870 recovers).
crate::failpoints::maybe_fail(
crate::failpoints::names::MUTATION_POST_STAGE_PRE_EFFECT_GATE,
)?;
let lineage_intent = self
.new_lineage_intent_for_branch(requested.as_deref(), actor_id)
.await?;
// `_queue_guards` holds the root-shared schema gate, branch
// effect gate, and sorted table gates acquired by `commit_all`.
// They remain held through manifest publication, covering the
// complete same-process sidecar/effect lifetime. They are a
// local serialization aid; the exact publisher precondition and
// durable v3 recovery plan remain the correctness authorities.
let super::staging::CommittedMutation {
updates,
expected_versions,
sidecar_handle,
guards: _queue_guards,
committed_handles,
} = staged
.commit_all(
self,
requested.as_deref(),
crate::db::manifest::SidecarKind::Mutation,
actor_id,
fork_queue_guards,
Some(&txn),
&txn,
&lineage_intent,
)
.await?;
// Failpoint that wedges the documented finalize→publisher
// residual: per-table `commit_staged` calls already
// advanced Lance HEAD on every touched table; a failure
// injected here mirrors the production-rare case where
// the publisher's CAS pre-check rejects (or the manifest
// write throws) after staged commits succeeded. The
// sidecar written inside `staging.finalize()` persists
// across this failure so the next `Omnigraph::open`'s
// recovery sweep can roll forward — see
// Failpoint for the confirmed-effects → publisher boundary:
// table HEADs have advanced but graph visibility has not. The
// v3 sidecar already contains exact transaction identities,
// immutable manifest delta, and fixed lineage/rollback outcomes.
// Any failure from here is `RecoveryRequired`; synchronous heal
// or a read-write open converges the recorded outcome. See
// `tests/failpoints.rs::recovery_rolls_forward_after_finalize_publisher_failure`.
crate::failpoints::maybe_fail(crate::failpoints::names::MUTATION_POST_FINALIZE_PRE_PUBLISHER)?;
self.commit_updates_on_branch_with_expected(
requested.as_deref(),
&updates,
&expected_versions,
actor_id,
Some(&txn),
committed_handles,
)
.await?;
// Phase C succeeded — sidecar can be deleted. If this
// delete fails, the next open's sweep classifies every
// table as NoMovement (manifest pin == Lance HEAD ==
// post_commit_pin) and the sidecar is treated as a
// stale artifact (cleaned up via the Phase 2 logic).
crate::failpoints::maybe_fail(
crate::failpoints::names::MUTATION_POST_FINALIZE_PRE_PUBLISHER,
)?;
let publish_result = self
.commit_updates_on_branch_with_expected(
requested.as_deref(),
&updates,
&expected_versions,
actor_id,
&txn,
lineage_intent,
)
.await;
if let Err(err) = publish_result {
// A sidecar exists iff at least one table effect was
// committed. Lineage-only / zero-row mutations have no
// physical residual to recover, so preserve their original
// publish error (notably ReadSetChanged) and let the normal
// retry/409 path handle it.
return match sidecar_handle.as_ref() {
Some(handle) => Err(OmniError::recovery_required(
handle.operation_id.clone(),
err.to_string(),
)),
None => Err(err),
};
}
if let Some(handle) = sidecar_handle {
// Best-effort cleanup: the manifest publish already
// succeeded, so the user's mutation is durable. A
// failed delete leaves the sidecar on disk; the
// next open's recovery sweep classifies every table
// as `NoMovement` (manifest pin == Lance HEAD ==
// post_commit_pin) and tidies up. Failing the user
// here would return an error for a write that
// already landed.
// succeeded, so the user's mutation is durable. A failed
// delete leaves a fixed, idempotent v3 outcome for the next
// synchronous heal or read-write open to audit and remove.
// Failing the user here would report an error for a write
// that already landed.
if let Err(err) =
crate::db::manifest::delete_sidecar(&handle, self.storage_adapter()).await
{
@ -841,13 +856,14 @@ impl Omnigraph {
/// is unchanged.
fn lower_named_mutation(
&self,
catalog: &omnigraph_compiler::catalog::Catalog,
query_source: &str,
query_name: &str,
) -> Result<omnigraph_compiler::ir::MutationIR> {
let query_decl = omnigraph_compiler::find_named_query(query_source, query_name)
.map_err(|e| OmniError::manifest(e.to_string()))?;
let checked = typecheck_query_decl(&self.catalog(), &query_decl)?;
let checked = typecheck_query_decl(catalog, &query_decl)?;
match checked {
CheckedQuery::Mutation(_) => {}
CheckedQuery::Read(_) => {
@ -863,58 +879,13 @@ impl Omnigraph {
Ok(ir)
}
/// The COMPLETE set of `(node|edge):{type}` table keys a mutation IR can
/// touch at execution time, keyed as `MutationStaging`/`commit_all` key
/// them. Must be a superset of everything execution forks/commits, since
/// it drives the up-front fork-queue acquisition and `commit_all`'s
/// held-guard coverage check — a miss means an unserialized fork/commit.
///
/// The set is a pure function of (IR ops + catalog). For each op it mirrors
/// the execute path's node-vs-edge dispatch (`node_types` first, then
/// `edge_types`). A `delete <Node>` additionally **cascades** to every edge
/// type whose endpoint is that node (see `execute_delete_node`), forking
/// those edge tables during execution — so they are included here, derived
/// the same way the executor derives them (`from_type`/`to_type` match).
/// Unknown types are skipped (the execute path surfaces the error).
/// Sorted + deduped for one-shot `acquire_many`.
fn touched_table_keys(&self, ir: &omnigraph_compiler::ir::MutationIR) -> Vec<String> {
use omnigraph_compiler::ir::MutationOpIR;
let catalog = self.catalog();
let mut keys: Vec<String> = Vec::new();
for op in &ir.ops {
let type_name = match op {
MutationOpIR::Insert { type_name, .. }
| MutationOpIR::Update { type_name, .. }
| MutationOpIR::Delete { type_name, .. } => type_name,
};
if catalog.node_types.contains_key(type_name) {
keys.push(format!("node:{type_name}"));
// A node delete cascades to every edge touching this node type,
// forking those edge tables. Include them so the up-front
// acquisition covers the cascade (mirrors execute_delete_node).
if matches!(op, MutationOpIR::Delete { .. }) {
for (edge_name, edge_type) in &catalog.edge_types {
if edge_type.from_type == *type_name || edge_type.to_type == *type_name {
keys.push(format!("edge:{edge_name}"));
}
}
}
} else if catalog.edge_types.contains_key(type_name) {
keys.push(format!("edge:{type_name}"));
}
}
keys.sort();
keys.dedup();
keys
}
async fn execute_named_mutation(
&self,
ir: &omnigraph_compiler::ir::MutationIR,
params: &ParamMap,
branch: Option<&str>,
staging: &mut MutationStaging,
txn: Option<&crate::db::WriteTxn>,
txn: &crate::db::WriteTxn,
) -> Result<MutationResult> {
let mut total = MutationResult::default();
for op in &ir.ops {
@ -932,7 +903,13 @@ impl Omnigraph {
predicate,
} => {
self.execute_update(
type_name, assignments, predicate, params, branch, staging, txn,
type_name,
assignments,
predicate,
params,
branch,
staging,
txn,
)
.await?
}
@ -957,18 +934,19 @@ impl Omnigraph {
params: &ParamMap,
branch: Option<&str>,
staging: &mut MutationStaging,
txn: Option<&crate::db::WriteTxn>,
txn: &crate::db::WriteTxn,
) -> Result<MutationResult> {
let mut resolved: HashMap<String, Literal> = HashMap::new();
for a in assignments {
resolved.insert(a.property.clone(), resolve_expr_value(&a.value, params)?);
}
let is_node = self.catalog().node_types.contains_key(type_name);
let is_edge = self.catalog().edge_types.contains_key(type_name);
let catalog = &txn.catalog;
let is_node = catalog.node_types.contains_key(type_name);
let is_edge = catalog.edge_types.contains_key(type_name);
if is_node {
let node_type = &self.catalog().node_types[type_name];
let node_type = &catalog.node_types[type_name];
let schema = node_type.arrow_schema.clone();
let blob_props = node_type.blob_properties.clone();
let id = if let Some(key_prop) = node_type.key_property() {
@ -1001,7 +979,8 @@ impl Omnigraph {
// only `ensure_path`'s captured version (read inside
// `open_table_for_mutation`) is used downstream.
let (_ds, _full_path, _table_branch) =
open_table_for_mutation(self, staging, branch, &table_key, insert_kind, txn).await?;
open_table_for_mutation(self, staging, branch, &table_key, insert_kind, Some(txn))
.await?;
// Accumulate. @key inserts go into the Merge stream (so a
// later update on the same id coalesces correctly); no-key
// inserts go into the Append stream.
@ -1017,26 +996,28 @@ impl Omnigraph {
affected_edges: 0,
})
} else if is_edge {
let edge_type = &self.catalog().edge_types[type_name];
let edge_type = &catalog.edge_types[type_name];
let schema = edge_type.arrow_schema.clone();
let blob_props = edge_type.blob_properties.clone();
let id = ulid::Ulid::new().to_string();
let batch = build_insert_batch(&schema, &id, &resolved, &blob_props)?;
// Validation (edge-RI, enum, unique, @card against LIVE HEAD) runs
// Validation (edge-RI, enum, unique, @card against the live
// manifest-visible branch snapshot) runs
// end-of-query via the evaluator.
let table_key = format!("edge:{}", type_name);
// Capture pre-write metadata on first touch (ensure_path). Edge
// inserts are non-strict, so with a `WriteTxn` this opens NOTHING
// (collapse #1) and the handle is discarded — validation, including
// `@card` against LIVE HEAD, runs end-of-query via the evaluator.
// `@card` against the live committed branch snapshot, runs
// end-of-query via the evaluator.
let (_handle, _full_path, _table_branch) = open_table_for_mutation(
self,
staging,
branch,
&table_key,
crate::db::MutationOpKind::Insert,
txn,
Some(txn),
)
.await?;
// Accumulate the new edge row. Edge IDs are ULID-generated so
@ -1062,10 +1043,11 @@ impl Omnigraph {
params: &ParamMap,
branch: Option<&str>,
staging: &mut MutationStaging,
txn: Option<&crate::db::WriteTxn>,
txn: &crate::db::WriteTxn,
) -> Result<MutationResult> {
let catalog = &txn.catalog;
// Defense in depth: ensure this is a node type
if !self.catalog().node_types.contains_key(type_name) {
if !catalog.node_types.contains_key(type_name) {
return Err(OmniError::manifest(format!(
"update is only supported for node types, not '{}'",
type_name
@ -1073,7 +1055,7 @@ impl Omnigraph {
}
// Reject updates to @key properties — identity is immutable
if let Some(key_prop) = self.catalog().node_types[type_name].key_property() {
if let Some(key_prop) = catalog.node_types[type_name].key_property() {
if assignments.iter().any(|a| a.property == key_prop) {
return Err(OmniError::manifest(format!(
"cannot update @key property '{}' — delete and re-insert instead",
@ -1083,8 +1065,8 @@ impl Omnigraph {
}
let pred_sql = predicate_to_sql(predicate, params, false)?;
let schema = self.catalog().node_types[type_name].arrow_schema.clone();
let blob_props = self.catalog().node_types[type_name].blob_properties.clone();
let schema = catalog.node_types[type_name].arrow_schema.clone();
let blob_props = catalog.node_types[type_name].blob_properties.clone();
let table_key = format!("node:{}", type_name);
let (handle, _full_path, _table_branch) = open_table_for_mutation(
@ -1093,7 +1075,7 @@ impl Omnigraph {
branch,
&table_key,
crate::db::MutationOpKind::Update,
txn,
Some(txn),
)
.await?;
// Update is a STRICT op, so collapse #1 never skips its open — the
@ -1101,25 +1083,9 @@ impl Omnigraph {
let ds = handle.expect("strict Update op always opens its dataset");
// Scan committed via Lance + apply the same predicate to pending
// batches via DataFusion `MemTable` (read-your-writes for prior
// ops in this query). The pending side may include rows from
// earlier `insert` / `update` ops on the same table.
//
// For blob tables we project away the blob columns: Lance's
// scanner doesn't accept the standard projection path on blob
// descriptors and would panic with a `Field::project` assertion.
// The downstream `apply_assignments` synthesizes blob columns
// from explicit assignments and omits unassigned blobs (Lance's
// merge_insert leaves them untouched). Tables without blob
// columns scan the full schema unprojected.
let non_blob_cols: Vec<&str> = schema
.fields()
.iter()
.filter(|f| !blob_props.contains(f.name()))
.map(|f| f.name().as_str())
.collect();
let projection: Option<&[&str]> =
(!blob_props.is_empty()).then_some(non_blob_cols.as_slice());
// batches via DataFusion `MemTable` (read-your-writes for prior ops in
// this query). The pending side may include rows from earlier
// `insert` / `update` ops on the same table.
let pending_batches = staging.pending_batches(&table_key);
let pending_schema = staging.pending_schema(&table_key);
// Use merge semantics on the union: a committed row whose `id`
@ -1128,17 +1094,34 @@ impl Omnigraph {
// otherwise the predicate runs against stale committed values
// and a chained `update where <pred>` can match a row whose
// pending value no longer satisfies <pred>.
let batches = self
.storage()
.scan_with_pending(
&ds,
pending_batches,
pending_schema,
projection,
Some(&pred_sql),
Some("id"),
)
.await?;
// A blob-v2 scan normally yields physical descriptor structs, which
// cannot be fed back to the full-schema merge writer. Select matched
// committed row ids without projecting blobs, then take and rebuild
// only those payloads into the logical blob schema before unioning
// pending rows. This keeps correctness independent of whether an id
// index happens to steer Lance onto its legacy partial-column plan.
let batches = if blob_props.is_empty() {
self.storage()
.scan_with_pending(
&ds,
pending_batches,
pending_schema,
None,
Some(&pred_sql),
Some("id"),
)
.await?
} else {
self.storage()
.scan_with_pending_materialized_blobs(
&ds,
pending_batches,
pending_schema,
Some(&pred_sql),
Some("id"),
)
.await?
};
if batches.is_empty() || batches.iter().all(|b| b.num_rows() == 0) {
return Ok(MutationResult {
@ -1148,14 +1131,9 @@ impl Omnigraph {
}
// Concat the matched batches (committed + pending) into one. The
// helper trusts that both sides share a schema — Lance returns
// dataset-schema-ordered columns and DataFusion returns
// MemTable-schema-ordered columns; both should match the catalog's
// arrow_schema when the projection is consistent. If they
// diverge (typically a blob-table mid-schema-shift), the helper
// surfaces a clear error directing the caller to split the
// mutation.
let matched = concat_match_batches_to_schema(&schema, &blob_props, batches)?;
// helper binds both sides to the catalog's full logical schema. Any
// divergence here is an internal scan/staging contract violation.
let matched = concat_match_batches_to_schema(&schema, batches)?;
let affected_count = matched.num_rows();
@ -1187,9 +1165,9 @@ impl Omnigraph {
params: &ParamMap,
branch: Option<&str>,
staging: &mut MutationStaging,
txn: Option<&crate::db::WriteTxn>,
txn: &crate::db::WriteTxn,
) -> Result<MutationResult> {
let is_node = self.catalog().node_types.contains_key(type_name);
let is_node = txn.catalog.node_types.contains_key(type_name);
if is_node {
self.execute_delete_node(type_name, predicate, params, branch, staging, txn)
.await
@ -1206,7 +1184,7 @@ impl Omnigraph {
params: &ParamMap,
branch: Option<&str>,
staging: &mut MutationStaging,
txn: Option<&crate::db::WriteTxn>,
txn: &crate::db::WriteTxn,
) -> Result<MutationResult> {
let pred_sql = predicate_to_sql(predicate, params, false)?;
@ -1217,7 +1195,7 @@ impl Omnigraph {
branch,
&table_key,
crate::db::MutationOpKind::Delete,
txn,
Some(txn),
)
.await?;
// Delete is a STRICT op, so collapse #1 never skips its open.
@ -1256,7 +1234,9 @@ impl Omnigraph {
// HEAD only at the unified end-of-query commit — no inline residual.
// `open_table_for_mutation` above already captured the table's
// path/version/op-kind via `ensure_path`.
crate::failpoints::maybe_fail(crate::failpoints::names::MUTATION_DELETE_NODE_PRE_PRIMARY_DELETE)?;
crate::failpoints::maybe_fail(
crate::failpoints::names::MUTATION_DELETE_NODE_PRE_PRIMARY_DELETE,
)?;
staging.record_deleted_ids(&table_key, &deleted_ids);
staging.record_delete(&table_key, pred_sql.clone());
@ -1267,8 +1247,8 @@ impl Omnigraph {
.collect();
let id_list = escaped.join(", ");
let edge_info: Vec<(String, String, String)> = self
.catalog()
let edge_info: Vec<(String, String, String)> = txn
.catalog
.edge_types
.iter()
.map(|(name, et)| (name.clone(), et.from_type.clone(), et.to_type.clone()))
@ -1294,7 +1274,7 @@ impl Omnigraph {
branch,
&edge_table_key,
crate::db::MutationOpKind::Delete,
txn,
Some(txn),
)
.await?;
// Delete is a STRICT op, so collapse #1 never skips its open.
@ -1310,8 +1290,10 @@ impl Omnigraph {
// by both a cascade and an explicit `delete <Edge>` — is counted
// once. Record the ORIGINAL cascade filter (the combined staged
// delete removes the union); skip only when nothing NEW matches.
let count_filter =
dedup_delete_filter(&cascade_filter, staging.recorded_delete_predicates(&edge_table_key));
let count_filter = dedup_delete_filter(
&cascade_filter,
staging.recorded_delete_predicates(&edge_table_key),
);
// Scan (not count) the cascade-removed edge ids so validation
// recounts the OTHER endpoint's @card after the cascade; `len()` is
// the affected count.
@ -1347,7 +1329,7 @@ impl Omnigraph {
params: &ParamMap,
branch: Option<&str>,
staging: &mut MutationStaging,
txn: Option<&crate::db::WriteTxn>,
txn: &crate::db::WriteTxn,
) -> Result<MutationResult> {
let pred_sql = predicate_to_sql(predicate, params, true)?;
@ -1358,7 +1340,7 @@ impl Omnigraph {
branch,
&table_key,
crate::db::MutationOpKind::Delete,
txn,
Some(txn),
)
.await?;
// Delete is a STRICT op, so collapse #1 never skips its open.
@ -1420,23 +1402,20 @@ fn ids_from_batches(batches: &[RecordBatch]) -> Vec<String> {
/// `scan_with_pending` returns committed-side and pending-side batches in
/// order; both should share a schema if pending was produced through
/// `apply_assignments` with full-schema scan input. If schemas drift,
/// surface a clear error so the user can split the query.
/// surface the internal contract failure at the mutation boundary.
fn concat_match_batches_to_schema(
_schema: &SchemaRef,
_blob_properties: &HashSet<String>,
schema: &SchemaRef,
batches: Vec<RecordBatch>,
) -> Result<RecordBatch> {
if batches.len() == 1 {
return Ok(batches.into_iter().next().unwrap());
let batch = batches.into_iter().next().unwrap();
return RecordBatch::try_new(schema.clone(), batch.columns().to_vec())
.map_err(|e| OmniError::Lance(e.to_string()));
}
let common = batches[0].schema();
arrow_select::concat::concat_batches(&common, &batches).map_err(|e| {
arrow_select::concat::concat_batches(schema, &batches).map_err(|e| {
OmniError::Lance(format!(
"scan_with_pending returned batches with mismatched schemas \
across the committed/pending boundary; this typically indicates \
a blob-column shape mismatch between the committed table and a \
prior in-query insert/update. Split blob-touching mutations \
into separate queries. ({})",
"mutation scan returned batches that violate the full logical schema \
across the committed/pending boundary ({})",
e
))
})

778
crates/omnigraph/src/exec/staging.rs
View file

File diff suppressed because it is too large Load diff

77
crates/omnigraph/src/failpoints.rs
View file

@ -39,31 +39,87 @@ pub(crate) fn maybe_fail_retryable_contention(name: &str) -> Result<()> {
/// compile error rather than a silently-never-firing failpoint.
pub mod names {
pub const BRANCH_DELETE_BEFORE_TABLE_CLEANUP: &str = "branch_delete.before_table_cleanup";
pub const BRANCH_MERGE_ADOPT_AFTER_APPEND_PRE_UPSERT: &str = "branch_merge.adopt_after_append_pre_upsert";
pub const BRANCH_MERGE_ADOPT_AFTER_UPSERT_PRE_DELETE: &str = "branch_merge.adopt_after_upsert_pre_delete";
pub const BRANCH_MERGE_POST_PHASE_B_PRE_MANIFEST_COMMIT: &str = "branch_merge.post_phase_b_pre_manifest_commit";
pub const BRANCH_MERGE_REWRITE_AFTER_DELETE_PRE_INDEX: &str = "branch_merge.rewrite_after_delete_pre_index";
pub const BRANCH_MERGE_REWRITE_AFTER_MERGE_PRE_DELETE: &str = "branch_merge.rewrite_after_merge_pre_delete";
/// Branch delete holds the schema, target-branch, and fresh-catalog table
/// envelope and has completed its final recovery check, before the native
/// manifest-ref mutation.
pub const BRANCH_DELETE_POST_TABLE_GATES: &str = "branch_delete.post_table_gates";
/// After native branch control completed its first recovery barrier, before
/// it acquires schema -> branch -> table gates and performs the final check.
pub const BRANCH_CONTROL_POST_RECOVERY_BARRIER: &str =
"branch_control.post_recovery_barrier";
pub const BRANCH_MERGE_ADOPT_AFTER_APPEND_PRE_UPSERT: &str =
"branch_merge.adopt_after_append_pre_upsert";
pub const BRANCH_MERGE_ADOPT_AFTER_UPSERT_PRE_DELETE: &str =
"branch_merge.adopt_after_upsert_pre_delete";
/// Source/target heads and snapshots have been captured while the schema
/// and both branch-incarnation gates are held, before merge planning or
/// any durable table effect.
pub const BRANCH_MERGE_POST_AUTHORITY_CAPTURE: &str =
"branch_merge.post_authority_capture";
pub const BRANCH_MERGE_POST_PHASE_B_PRE_MANIFEST_COMMIT: &str =
"branch_merge.post_phase_b_pre_manifest_commit";
/// Every merge table effect is complete, but the sidecar is still in its
/// pre-confirmation shape.
pub const BRANCH_MERGE_POST_EFFECTS_PRE_CONFIRM: &str =
"branch_merge.post_effects_pre_confirm";
pub const BRANCH_MERGE_REWRITE_AFTER_DELETE_PRE_INDEX: &str =
"branch_merge.rewrite_after_delete_pre_index";
pub const BRANCH_MERGE_REWRITE_AFTER_MERGE_PRE_DELETE: &str =
"branch_merge.rewrite_after_merge_pre_delete";
pub const CLASSIFY_FRESH_READ: &str = "classify.fresh_read";
pub const CLEANUP_RECONCILE_FORK: &str = "cleanup.reconcile_fork";
/// After cleanup's fast empty-sidecar probe, before it acquires the closed
/// schema/branch/table GC gate set and performs the authoritative recheck.
pub const CLEANUP_POST_RECOVERY_CHECK_PRE_GATES: &str =
"cleanup.post_recovery_check_pre_gates";
pub const CLEANUP_RESOLVE_BRANCH_SNAPSHOT: &str = "cleanup.resolve_branch_snapshot";
pub const CLEANUP_TABLE_GC: &str = "cleanup.table_gc";
pub const ENSURE_INDICES_POST_PHASE_B_PRE_MANIFEST_COMMIT: &str = "ensure_indices.post_phase_b_pre_manifest_commit";
pub const ENSURE_INDICES_POST_STAGE_PRE_COMMIT_BTREE: &str = "ensure_indices.post_stage_pre_commit_btree";
pub const ENSURE_INDICES_POST_PHASE_B_PRE_MANIFEST_COMMIT: &str =
"ensure_indices.post_phase_b_pre_manifest_commit";
pub const ENSURE_INDICES_POST_STAGE_PRE_COMMIT_BTREE: &str =
"ensure_indices.post_stage_pre_commit_btree";
pub const FORK_BEFORE_CLASSIFY: &str = "fork.before_classify";
pub const FORK_BEFORE_RECLAIM: &str = "fork.before_reclaim";
/// After Lance durably creates a target table ref, before the caller can
/// reopen and verify it. An error here is post-effect and must retain the
/// recovery sidecar.
pub const FORK_POST_CREATE_PRE_OPEN: &str = "fork.post_create_pre_open";
pub const GRAPH_PUBLISH_AFTER_MANIFEST_COMMIT: &str = "graph_publish.after_manifest_commit";
pub const GRAPH_PUBLISH_BEFORE_COMMIT_APPEND: &str = "graph_publish.before_commit_append";
pub const INIT_AFTER_COORDINATOR_INIT: &str = "init.after_coordinator_init";
pub const INIT_AFTER_SCHEMA_CONTRACT_WRITTEN: &str = "init.after_schema_contract_written";
pub const INIT_AFTER_SCHEMA_PG_WRITTEN: &str = "init.after_schema_pg_written";
pub const MUTATION_DELETE_NODE_PRE_PRIMARY_DELETE: &str = "mutation.delete_node_pre_primary_delete";
pub const MUTATION_DELETE_NODE_PRE_PRIMARY_DELETE: &str =
"mutation.delete_node_pre_primary_delete";
/// After every deferred first-touch table ref is created under a durable
/// v3 sidecar, before any staged data transaction advances target HEAD.
pub const MUTATION_POST_FORK_PRE_COMMIT: &str = "mutation.post_fork_pre_commit";
/// After the v3 ownership sidecar is durable but before the first deferred
/// named-table ref is created. Recovery must accept the absent target ref.
pub const MUTATION_POST_SIDECAR_PRE_FORK: &str = "mutation.post_sidecar_pre_fork";
/// Deterministic OCC rendezvous after a mutation has validated and staged
/// its complete attempt, but before the RFC-022 branch effect gate is
/// acquired and the write authority token is revalidated. Tests park the
/// first writer here, commit a conflicting second writer, then prove the
/// first attempt is discarded and validation is rerun from a fresh token.
pub const MUTATION_POST_STAGE_PRE_EFFECT_GATE: &str = "mutation.post_stage_pre_effect_gate";
pub const MUTATION_POST_FINALIZE_PRE_PUBLISHER: &str = "mutation.post_finalize_pre_publisher";
/// Open owns the schema gate and is about to read source/IR/state as one
/// catalog view.
pub const OPEN_BEFORE_SCHEMA_CONTRACT_READ: &str = "open.before_schema_contract_read";
pub const OPTIMIZE_BEFORE_COMPACT: &str = "optimize.before_compact";
pub const OPTIMIZE_INJECT_REINDEX_CONFLICT: &str = "optimize.inject_reindex_conflict";
pub const OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT: &str = "optimize.post_phase_b_pre_manifest_commit";
pub const OPTIMIZE_POST_PHASE_B_PRE_MANIFEST_COMMIT: &str =
"optimize.post_phase_b_pre_manifest_commit";
pub const RECOVERY_BEFORE_ROLL_FORWARD_PUBLISH: &str = "recovery.before_roll_forward_publish";
/// Recovery has listed/parsed its discovery snapshot but has not yet taken
/// per-sidecar gates. Tests rewrite confirmation state in this window.
pub const RECOVERY_POST_LIST_PRE_GATES: &str = "recovery.post_list_pre_gates";
pub const RECOVERY_ORPHAN_DISCARD_AUDIT_APPEND: &str = "recovery.orphan_discard_audit_append";
/// After the fixed rollback lineage/table-pin publish is durable, before
/// its operator-facing audit row is appended.
pub const RECOVERY_POST_ROLLBACK_PUBLISH_PRE_AUDIT: &str =
"recovery.post_rollback_publish_pre_audit";
pub const RECOVERY_RECORD_AUDIT: &str = "recovery.record_audit";
pub const RECOVERY_SIDECAR_CONFIRM: &str = "recovery.sidecar_confirm";
pub const RECOVERY_SIDECAR_DELETE: &str = "recovery.sidecar_delete";
@ -72,6 +128,9 @@ pub mod names {
pub const SCHEMA_APPLY_AFTER_MANIFEST_COMMIT: &str = "schema_apply.after_manifest_commit";
pub const SCHEMA_APPLY_AFTER_STAGING_WRITE: &str = "schema_apply.after_staging_write";
pub const SCHEMA_APPLY_BEFORE_STAGING_WRITE: &str = "schema_apply.before_staging_write";
/// Reload owns the schema gate and is about to read/publish one contract view.
pub const SCHEMA_RELOAD_BEFORE_CONTRACT_READ: &str =
"schema_reload.before_contract_read";
/// Injects a retryable `RowLevelCasContention` from `load_publish_state` so a
/// test can prove the publisher's outer retry re-runs the load.
pub const PUBLISH_LOAD_STATE_RETRYABLE_CONTENTION: &str =

10
crates/omnigraph/src/instrumentation.rs
View file

@ -313,6 +313,8 @@ pub struct StorageReadCounts {
pub exists: AtomicU64,
pub read_text_versioned: AtomicU64,
pub list_dir: AtomicU64,
pub write_text: AtomicU64,
pub delete: AtomicU64,
}
impl StorageReadCounts {
@ -328,6 +330,12 @@ impl StorageReadCounts {
pub fn list_dir(&self) -> u64 {
self.list_dir.load(Ordering::Relaxed)
}
pub fn write_text(&self) -> u64 {
self.write_text.load(Ordering::Relaxed)
}
pub fn delete(&self) -> u64 {
self.delete.load(Ordering::Relaxed)
}
}
/// Boundary decorator over a [`StorageAdapter`] that counts read-facing calls.
@ -360,6 +368,7 @@ impl StorageAdapter for CountingStorageAdapter {
}
async fn write_text(&self, uri: &str, contents: &str) -> Result<()> {
self.counts.write_text.fetch_add(1, Ordering::Relaxed);
self.inner.write_text(uri, contents).await
}
@ -377,6 +386,7 @@ impl StorageAdapter for CountingStorageAdapter {
}
async fn delete(&self, uri: &str) -> Result<()> {
self.counts.delete.fetch_add(1, Ordering::Relaxed);
self.inner.delete(uri).await
}

216
crates/omnigraph/src/loader/mod.rs
View file

@ -187,17 +187,6 @@ impl Omnigraph {
&omnigraph_policy::ResourceScope::Branch(branch.to_string()),
actor_id,
)?;
// Schema-contract validation is captured ONCE per write via the
// `WriteTxn` opened in `load_jsonl_reader` (after branch resolution).
// The redundant `ensure_schema_state_valid` that used to run here is
// subsumed by `open_write_txn`'s `resolved_branch_target` call.
// Converge any pending recovery sidecar (a previously failed
// writer's Phase B → Phase C residual) before staging anything:
// without this, sidecar-covered drift wedges every load on the
// commit-time drift guard until a process restart — `repair`
// refuses while a sidecar is pending. One `list_dir` when no
// sidecars exist (the steady state).
self.heal_pending_recovery_sidecars().await?;
// Reject internal `__run__*` / system-prefixed branches at the
// public write boundary. Direct-publish paths assert this
// explicitly so a caller can't write to legacy or system
@ -215,6 +204,23 @@ impl Omnigraph {
}
None => None,
};
// Schema/catalog authority is captured once via the `WriteTxn` (plus its
// cheap trailing identity-marker fence); the only second full validation
// is the required pre-effect recheck under gates. Per-table resolution
// performs no additional contract reads.
//
// Stage A precedes both an implicit target-branch fork and data staging.
// The target branch and an explicit base are read/write authority for the
// operation, so an unresolved intent on either closes the barrier. The
// helper folds `Some("main")` to main's canonical `None` identity.
let mut recovery_branches = vec![requested.as_deref()];
if base.is_some() {
// `base_branch` retains `Some("main")` for the result DTO; the
// barrier helper canonicalizes it to main's `None` identity.
recovery_branches.push(base_branch.as_deref());
}
self.heal_pending_recovery_sidecars_for_write(&recovery_branches)
.await?;
// Fork-if-missing only when a base branch was explicitly given.
// `requested == None` is `main`, which always exists.
let mut branch_created = false;
@ -238,7 +244,7 @@ impl Omnigraph {
}
// Direct-to-target writes: no Run state machine, no `__run__` staging
// branch. Cross-table OCC is enforced by the publisher's
// `expected_table_versions` CAS inside `load_jsonl_reader`.
// `expected_table_versions` CAS inside the load attempt.
let mut result = self
.load_direct_on_branch(requested.as_deref(), data, mode, actor_id)
.await?;
@ -274,8 +280,7 @@ impl Omnigraph {
mode: LoadMode,
actor_id: Option<&str>,
) -> Result<LoadResult> {
let reader = BufReader::new(Cursor::new(data.as_bytes()));
load_jsonl_reader(self, branch, reader, mode, actor_id).await
load_jsonl_data(self, branch, data, mode, actor_id).await
}
}
@ -312,14 +317,55 @@ impl LoadResult {
}
}
async fn load_jsonl_reader<R: BufRead>(
async fn load_jsonl_data(
db: &Omnigraph,
branch: Option<&str>,
data: &str,
mode: LoadMode,
actor_id: Option<&str>,
) -> Result<LoadResult> {
const MAX_PRE_EFFECT_REPREPARES: usize = 32;
// Every public load entry point already owns a stable `&str` payload
// (`load_file_as` reads its file once). Replay that slice directly on a
// pre-effect retry; copying it into a second raw byte buffer would double
// peak input memory before the parser's per-type materialization.
let retryable = matches!(mode, LoadMode::Append | LoadMode::Merge);
for attempt in 0..=MAX_PRE_EFFECT_REPREPARES {
let replay = BufReader::new(Cursor::new(data.as_bytes()));
match load_jsonl_reader_once(db, branch, replay, mode, actor_id).await {
Err(err)
if retryable
&& err.is_read_set_changed()
&& attempt < MAX_PRE_EFFECT_REPREPARES =>
{
tracing::debug!(
attempt = attempt + 1,
branch = branch.unwrap_or("main"),
"prepared load authority changed before effects; repreparing"
);
db.refresh().await?;
}
result => return result,
}
}
unreachable!("bounded load retry loop always returns")
}
async fn load_jsonl_reader_once<R: BufRead>(
db: &Omnigraph,
branch: Option<&str>,
reader: R,
mode: LoadMode,
actor_id: Option<&str>,
) -> Result<LoadResult> {
let catalog = db.catalog().clone();
// Capture the manifest/schema authority before interpreting any input. The
// catalog rides the WriteTxn and was built from the exact accepted IR named
// by its schema token; a long-lived handle's global catalog may legitimately
// lag a schema apply completed through another handle.
let txn = db.open_write_txn(branch).await?;
let catalog = Arc::clone(&txn.catalog);
let snapshot = txn.base.clone();
// Phase 1: Parse all lines, spool into per-type collections
let mut node_rows: HashMap<String, Vec<JsonValue>> = HashMap::new();
@ -400,16 +446,10 @@ async fn load_jsonl_reader<R: BufRead>(
// inline path.
let mut result = LoadResult::default();
// Capture-once write transaction (RFC-013 step 3b). `open_write_txn`
// validates the schema contract ONCE and pins the base snapshot. Threaded
// as `Some(&txn)` through the per-table opens and the manifest publish so
// each resolve point reuses the pinned base instead of re-validating the
// contract. The branch already exists here (fork-if-missing ran in
// `load_as` before this), so this captures the post-fork snapshot. The
// load's own base read (`db.snapshot_for_branch` previously) is the same
// per-branch snapshot, so reuse `txn.base` for it — dropping a validation.
let txn = db.open_write_txn(branch).await?;
let snapshot = txn.base.clone();
// The branch-wide WriteTxn captured above is threaded through every table
// open and the manifest publish, so the parsed batches, validation catalog,
// base snapshot, native branch identity, exact graph head, and schema
// identity form one immutable authority unit.
let mut staging = MutationStaging::default();
let pending_mode = match mode {
LoadMode::Merge => PendingMode::Merge,
@ -419,56 +459,17 @@ async fn load_jsonl_reader<R: BufRead>(
LoadMode::Append => PendingMode::Append,
LoadMode::Overwrite => PendingMode::Overwrite,
};
// Map LoadMode to MutationOpKind for the version-check policy.
// Append/Merge skip the strict pre-stage check (concurrency-safe
// under the per-(table, branch) queue + publisher CAS); Overwrite
// uses the strict check because it truncates and replaces the
// dataset — concurrent advances change what "replace" means.
// Map LoadMode to the early table-version policy. Append/Merge may stage
// reclaimable files before the effect gates, then revalidate the complete
// branch token and fully reprepare on a bounded pre-effect conflict.
// Overwrite keeps the strict early check because it replaces the image; its
// later branch-wide mismatch surfaces `ReadSetChanged` without replay.
let load_op_kind = match mode {
LoadMode::Append => crate::db::MutationOpKind::Insert,
LoadMode::Merge => crate::db::MutationOpKind::Merge,
LoadMode::Overwrite => crate::db::MutationOpKind::SchemaRewrite,
};
// Up-front fork-queue acquisition. The first write to a table on a
// non-main branch forks it (create_branch), which advances Lance state
// before the manifest publish; the reclaim of any manifest-unreferenced
// leftover (`reclaim_orphaned_fork_and_refork`) must not race a concurrent
// in-process fork. So when this load will fork at least one touched table,
// acquire the per-(table, branch) write queues for ALL touched tables up
// front (one sorted `acquire_many`, keyed uniformly by the target branch
// so it covers what `commit_all` recomputes) and hold them through the
// publish. Main-branch loads never fork; branch loads where every touched
// table is already forked skip this and let `commit_all` acquire at commit.
let fork_queue_guards: Option<(
Vec<(String, Option<String>)>,
Vec<tokio::sync::OwnedMutexGuard<()>>,
)> = if let Some(active) = branch {
let touched: Vec<(String, Option<String>)> = node_rows
.keys()
.map(|t| (format!("node:{t}"), Some(active.to_string())))
.chain(
edge_rows
.keys()
.map(|e| (format!("edge:{e}"), Some(active.to_string()))),
)
.collect();
let needs_fork = touched.iter().any(|(table_key, _)| {
snapshot
.entry(table_key)
.map(|e| e.table_branch.as_deref() != Some(active))
.unwrap_or(false)
});
if needs_fork {
let guards = db.write_queue().acquire_many(&touched).await;
Some((touched, guards))
} else {
None
}
} else {
None
};
// Phase 2a: build and validate every node batch up front. Cheap and
// synchronous — surfaces validation errors before any S3 traffic.
let mut prepared_nodes: Vec<(String, String, RecordBatch, usize)> =
@ -499,6 +500,7 @@ async fn load_jsonl_reader<R: BufRead>(
&table_key,
opened.full_path,
opened.table_branch,
opened.deferred_fork,
opened.expected_version,
load_op_kind,
);
@ -534,6 +536,7 @@ async fn load_jsonl_reader<R: BufRead>(
&table_key,
opened.full_path,
opened.table_branch,
opened.deferred_fork,
opened.expected_version,
load_op_kind,
);
@ -581,43 +584,60 @@ async fn load_jsonl_reader<R: BufRead>(
let staged = staging
.stage_all_with_concurrency(db, branch, load_write_concurrency())
.await?;
// `_queue_guards` holds per-(table_key, branch) write queues
// across the manifest publish below — see exec/mutation.rs for
// the rationale (interleaving prevention).
crate::failpoints::maybe_fail(crate::failpoints::names::MUTATION_POST_STAGE_PRE_EFFECT_GATE)?;
let lineage_intent = db.new_lineage_intent_for_branch(branch, actor_id).await?;
// `_queue_guards` holds the root-shared schema → branch → sorted-table
// gates across manifest publication. This closes same-process
// interleaving across the v3 sidecar/effect lifetime. The exact publisher
// token and durable sidecar remain persistent correctness authorities, but
// these local gates do not expand the documented single-writer-process
// recovery boundary.
let crate::exec::staging::CommittedMutation {
updates,
expected_versions,
sidecar_handle,
guards: _queue_guards,
committed_handles,
} = staged
.commit_all(
db,
branch,
crate::db::manifest::SidecarKind::Load,
actor_id,
fork_queue_guards,
Some(&txn),
&txn,
&lineage_intent,
)
.await?;
// Same finalize → publisher residual as mutations: per-table
// staged commits have advanced Lance HEAD, but the manifest
// publish has not run yet. Reuse the mutation failpoint name so
// one failpoint pins the shared `MutationStaging` boundary.
// Same confirmed-effects → publisher boundary as mutations: table HEADs
// have advanced and the v3 sidecar contains their exact transaction
// identities, but the graph manifest has not published the result. Reuse
// the mutation failpoint name so one failpoint pins the shared boundary.
crate::failpoints::maybe_fail(crate::failpoints::names::MUTATION_POST_FINALIZE_PRE_PUBLISHER)?;
db.commit_updates_on_branch_with_expected(
branch,
&updates,
&expected_versions,
actor_id,
Some(&txn),
committed_handles,
)
.await?;
// The recovery sidecar protects the per-table commit_staged →
// manifest publish window. Phase C succeeded — clean up
// best-effort: failing the user here would error out a write
// that already landed durably.
let publish_result = db
.commit_updates_on_branch_with_expected(
branch,
&updates,
&expected_versions,
actor_id,
&txn,
lineage_intent,
)
.await;
if let Err(err) = publish_result {
// Empty loads can still publish lineage but have no table effect and
// therefore no recovery sidecar. Preserve that publish error instead
// of manufacturing an "unknown" recovery operation.
return match sidecar_handle.as_ref() {
Some(handle) => Err(OmniError::recovery_required(
handle.operation_id.clone(),
err.to_string(),
)),
None => Err(err),
};
}
// The v3 recovery sidecar protects every independently durable table effect
// through the one manifest visibility point. Phase C succeeded — clean up
// best-effort: failing the user here would error out a write that already
// landed durably; a leftover fixed outcome is idempotently finalized later.
if let Some(handle) = sidecar_handle {
if let Err(err) = crate::db::manifest::delete_sidecar(&handle, db.storage_adapter()).await {
tracing::warn!(
@ -1843,7 +1863,10 @@ edge WorksAt: Person -> Company
let result = db
.load_as("nonexistent", None, TEST_DATA, LoadMode::Merge, None)
.await;
assert!(result.is_err(), "load without base must not create branches");
assert!(
result.is_err(),
"load without base must not create branches"
);
assert!(
!db.branch_list()
.await
@ -1853,7 +1876,10 @@ edge WorksAt: Person -> Company
);
// Loads to main carry the default branch metadata.
let main_load = db.load("main", TEST_DATA, LoadMode::Overwrite).await.unwrap();
let main_load = db
.load("main", TEST_DATA, LoadMode::Overwrite)
.await
.unwrap();
assert_eq!(main_load.branch, "main");
assert_eq!(main_load.base_branch, None);
assert!(!main_load.branch_created);

71
crates/omnigraph/src/storage.rs
View file

@ -502,6 +502,54 @@ pub fn normalize_root_uri(uri: &str) -> Result<String> {
}
}
/// Process-local identity used to share writer queues across handles for the
/// same graph root.
///
/// The storage URI remains unchanged: this identity is used only as the key in
/// the in-process queue registry. Local paths are first made absolute
/// lexically, then the deepest canonicalizable ancestor is resolved so aliases
/// through symlinks converge. Any suffix that does not exist yet is appended
/// unchanged, which makes the identity safe to compute before `init` creates
/// the graph directory. Object-store and caller-defined URI schemes are opaque
/// and retain their normalized spelling.
pub(crate) fn write_queue_root_identity(normalized_root: &str) -> Result<String> {
let local_path = if normalized_root.starts_with(FILE_SCHEME_PREFIX) {
local_path_from_file_uri(normalized_root)?
} else if Path::new(normalized_root).is_absolute() {
PathBuf::from(normalized_root)
} else if has_uri_scheme(normalized_root) {
return Ok(normalized_root.to_string());
} else {
PathBuf::from(normalized_root)
};
let absolute = absolutize_lexically(local_path)?;
let mut ancestor = absolute.as_path();
let mut suffix = Vec::new();
loop {
if let Ok(canonical) = std::fs::canonicalize(ancestor) {
let mut identity = canonical;
for component in suffix.iter().rev() {
identity.push(component);
}
return Ok(normalize_local_path(&identity));
}
let Some(name) = ancestor.file_name() else {
// Filesystem roots should always canonicalize. Falling back to the
// lexical absolute path preserves queue availability on unusual
// platforms/filesystems without changing the storage root.
return Ok(normalize_local_path(&absolute));
};
suffix.push(name.to_os_string());
let Some(parent) = ancestor.parent() else {
return Ok(normalize_local_path(&absolute));
};
ancestor = parent;
}
}
pub fn join_uri(root_uri: &str, relative_path: &str) -> String {
let relative_path = relative_path.trim_start_matches('/');
match storage_kind_for_uri(root_uri) {
@ -534,6 +582,18 @@ fn local_path_from_uri(uri: &str) -> Result<PathBuf> {
Ok(PathBuf::from(uri))
}
fn has_uri_scheme(value: &str) -> bool {
let Some(colon) = value.find(':') else {
return false;
};
let scheme = &value[..colon];
!scheme.is_empty()
&& scheme.as_bytes()[0].is_ascii_alphabetic()
&& scheme
.bytes()
.all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'+' | b'-' | b'.'))
}
/// Lexically absolutize a local path: join relative paths onto the current
/// working directory and fold `.` / `..` components, without touching the
/// filesystem. Required because `object_store::path::Path` rejects
@ -908,6 +968,17 @@ mod tests {
);
}
#[test]
fn write_queue_identity_keeps_remote_and_custom_schemes_opaque() {
for root in [
"s3://bucket/prefix",
"memory://write-queue/custom",
"custom+transport:opaque-root",
] {
assert_eq!(write_queue_root_identity(root).unwrap(), root);
}
}
#[test]
fn join_uri_handles_local_file_and_s3_roots() {
assert_eq!(

113
crates/omnigraph/src/storage_layer.rs
View file

@ -66,7 +66,7 @@ use lance::dataset::{WhenMatched, WhenNotMatched};
use crate::db::{Snapshot, SubTableEntry};
use crate::error::Result;
use crate::table_store::{StagedWrite, TableState, TableStore};
use crate::table_store::{StagedTransactionIdentity, StagedWrite, TableState, TableStore};
// ─── sealed module ──────────────────────────────────────────────────────────
@ -174,6 +174,59 @@ impl StagedHandle {
pub(crate) fn into_staged(self) -> StagedWrite {
self.inner
}
/// Lance transaction identity captured when this effect was staged.
pub fn transaction_identity(&self) -> StagedTransactionIdentity {
self.inner.transaction_identity()
}
/// Replace Lance's random transaction UUID with the identity durably armed
/// before a deferred first-touch fork. The read version must still match.
pub(crate) fn bind_transaction_identity(
&mut self,
planned: &StagedTransactionIdentity,
) -> Result<()> {
self.inner.bind_transaction_identity(planned)
}
}
/// Result of the no-conflict-retry commit path used by RFC-022-enrolled
/// writers. `is_exact` checks both transaction identity and achieved version:
/// Lance's initial conflict-resolution pass can preserve `(read_version, uuid)`
/// while committing at a later version. The table effect is durable when that
/// happens, so the caller must leave its recovery sidecar armed.
#[derive(Debug)]
pub struct ExactCommitOutcome {
snapshot: SnapshotHandle,
planned_transaction: StagedTransactionIdentity,
committed_transaction: StagedTransactionIdentity,
}
impl ExactCommitOutcome {
pub fn is_exact(&self) -> bool {
self.planned_transaction == self.committed_transaction
&& self.snapshot.version() == self.planned_transaction.read_version + 1
}
pub fn planned_transaction(&self) -> &StagedTransactionIdentity {
&self.planned_transaction
}
pub fn committed_transaction(&self) -> &StagedTransactionIdentity {
&self.committed_transaction
}
pub fn committed_version(&self) -> u64 {
self.snapshot.version()
}
pub fn snapshot(&self) -> &SnapshotHandle {
&self.snapshot
}
pub fn into_snapshot(self) -> SnapshotHandle {
self.snapshot
}
}
/// Helper: clone the inner `StagedWrite` out of each `StagedHandle` and
@ -324,6 +377,20 @@ pub trait TableStorage: sealed::Sealed + Send + Sync + Debug {
key_column: Option<&str>,
) -> Result<Vec<RecordBatch>>;
/// Full-schema blob-aware sibling of `scan_with_pending` for mutation
/// updates. The committed predicate scan retains row ids without projecting
/// blobs; only matched rows are then taken and rebuilt as Lance's logical
/// blob input arrays before unioning the in-memory pending view. This keeps
/// the eventual merge source schema independent of scalar-index state.
async fn scan_with_pending_materialized_blobs(
&self,
snapshot: &SnapshotHandle,
pending: &[RecordBatch],
pending_schema: Option<SchemaRef>,
filter: Option<&str>,
key_column: Option<&str>,
) -> Result<Vec<RecordBatch>>;
async fn first_row_id_for_filter(
&self,
snapshot: &SnapshotHandle,
@ -366,6 +433,15 @@ pub trait TableStorage: sealed::Sealed + Send + Sync + Debug {
staged: StagedHandle,
) -> Result<SnapshotHandle>;
/// Commit one staged effect with Lance conflict retries disabled and expose
/// the transaction identity that actually landed. Legacy callers retain
/// `commit_staged`; RFC-022 adapters opt into this method explicitly.
async fn commit_staged_exact(
&self,
snapshot: SnapshotHandle,
staged: StagedHandle,
) -> Result<ExactCommitOutcome>;
/// Stage an overwrite (Operation::Overwrite). MR-793 Phase 2.
async fn stage_overwrite(
&self,
@ -636,6 +712,25 @@ impl TableStorage for TableStore {
.await
}
async fn scan_with_pending_materialized_blobs(
&self,
snapshot: &SnapshotHandle,
pending: &[RecordBatch],
pending_schema: Option<SchemaRef>,
filter: Option<&str>,
key_column: Option<&str>,
) -> Result<Vec<RecordBatch>> {
TableStore::scan_with_pending_materialized_blobs(
self,
snapshot.dataset(),
pending,
pending_schema,
filter,
key_column,
)
.await
}
async fn first_row_id_for_filter(
&self,
snapshot: &SnapshotHandle,
@ -701,6 +796,22 @@ impl TableStorage for TableStore {
.map(SnapshotHandle::new)
}
async fn commit_staged_exact(
&self,
snapshot: SnapshotHandle,
staged: StagedHandle,
) -> Result<ExactCommitOutcome> {
let planned_transaction = staged.transaction_identity();
let ds_arc = snapshot.into_arc();
let (dataset, committed_transaction) =
TableStore::commit_staged_exact(self, ds_arc, staged.into_staged()).await?;
Ok(ExactCommitOutcome {
snapshot: SnapshotHandle::new(dataset),
planned_transaction,
committed_transaction,
})
}
async fn stage_overwrite(
&self,
snapshot: &SnapshotHandle,

304
crates/omnigraph/src/table_store.rs
View file

@ -16,13 +16,14 @@ use lance::dataset::{
use lance::datatypes::{BlobKind, Schema as LanceSchema};
use lance::index::DatasetIndexExt;
use lance::index::scalar::IndexDetails;
use lance_select::mask::RowAddrTreeMap;
use lance_file::version::LanceFileVersion;
use lance_index::scalar::{InvertedIndexParams, ScalarIndexParams};
use lance_index::{IndexType, is_system_index};
use lance_linalg::distance::MetricType;
use lance_select::mask::RowAddrTreeMap;
use lance_table::format::{Fragment, IndexMetadata, RowIdMeta};
use lance_table::rowids::{RowIdSequence, write_row_ids};
use serde::{Deserialize, Serialize};
use std::sync::Arc;
use crate::db::manifest::TableVersionMetadata;
@ -50,6 +51,30 @@ pub enum IndexCoverage {
Degraded { reason: String },
}
/// Stable identity of one Lance transaction.
///
/// Lance persists both fields in the transaction file referenced by the
/// committed manifest. Recovery uses the pair, rather than a numeric table
/// version alone, to prove that an observed HEAD was produced by the staged
/// effect named in a recovery sidecar. The UUID distinguishes two writers that
/// started from the same version. Lance may preserve both fields while
/// rebasing, so enrolled callers must also require the achieved table version
/// to be exactly `read_version + 1`.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct StagedTransactionIdentity {
pub read_version: u64,
pub uuid: String,
}
impl From<&Transaction> for StagedTransactionIdentity {
fn from(transaction: &Transaction) -> Self {
Self {
read_version: transaction.read_version,
uuid: transaction.uuid.clone(),
}
}
}
/// A Lance write that has produced fragment files on object storage but is
/// not yet committed to the dataset's manifest. The staged-write primitives
/// are consumed by `MutationStaging` (`exec/staging.rs`,
@ -137,6 +162,30 @@ impl StagedWrite {
pub fn removed_fragment_ids(&self) -> &[u64] {
&self.removed_fragment_ids
}
/// Identity Lance assigned when this effect was staged.
pub fn transaction_identity(&self) -> StagedTransactionIdentity {
StagedTransactionIdentity::from(&self.transaction)
}
/// Bind a pre-minted recovery identity to a transaction staged after a
/// deferred branch fork. The operation and read version still come from
/// Lance; only its otherwise-random UUID is replaced so the sidecar can be
/// durable before the target ref (and its branch-local fragment paths)
/// exist.
pub(crate) fn bind_transaction_identity(
&mut self,
planned: &StagedTransactionIdentity,
) -> Result<()> {
if self.transaction.read_version != planned.read_version {
return Err(OmniError::manifest_internal(format!(
"staged transaction read version {} does not match pre-minted recovery pin {}",
self.transaction.read_version, planned.read_version
)));
}
self.transaction.uuid.clone_from(&planned.uuid);
Ok(())
}
}
#[derive(Debug, Clone)]
@ -376,6 +425,11 @@ impl TableStore {
return Err(OmniError::Lance(create_err.to_string()));
}
// The ref is now independently durable. Any error from this point is an
// ambiguous/post-effect outcome to the caller and must retain an armed
// recovery intent rather than being treated as a safe pre-effect retry.
crate::failpoints::maybe_fail(crate::failpoints::names::FORK_POST_CREATE_PRE_OPEN)?;
let ds = self
.open_dataset_head(dataset_uri, Some(target_branch))
.await?;
@ -446,6 +500,30 @@ impl TableStore {
.copied()
.collect::<Vec<_>>();
Self::materialize_blob_batch_with_row_ids(ds, batch, &row_ids).await
}
/// Rebuild the blob columns in `batch` using explicit stable row ids.
///
/// Most rewrite callers scan with `_rowid` and use
/// [`Self::materialize_blob_batch`]. A predicate-filtered blob mutation
/// cannot include blob descriptors in that scan on the pinned Lance
/// revision (the filter projection panics), so it first scans only
/// non-blob columns + `_rowid`, takes the full descriptor rows by id, and
/// calls this sibling with the ids captured by the safe scan.
async fn materialize_blob_batch_with_row_ids(
ds: &Dataset,
batch: RecordBatch,
row_ids: &[u64],
) -> Result<RecordBatch> {
if batch.num_rows() != row_ids.len() {
return Err(OmniError::Lance(format!(
"blob materialization row count {} does not match {} row ids",
batch.num_rows(),
row_ids.len()
)));
}
let schema: SchemaRef = Arc::new(ds.schema().into());
let mut columns = Vec::with_capacity(schema.fields().len());
for field in schema.fields() {
@ -1263,6 +1341,44 @@ impl TableStore {
/// the publisher at end-of-query to materialize all staged writes before
/// the meta-manifest commit.
pub async fn commit_staged(&self, ds: Arc<Dataset>, staged: StagedWrite) -> Result<Dataset> {
self.commit_staged_with_retry_budget(ds, staged, None)
.await
.map(|(dataset, _)| dataset)
}
/// Commit an RFC-022-enrolled staged effect with no commit-conflict retry.
///
/// `CommitBuilder::with_max_retries(0)` gives Lance one commit attempt. It
/// can still perform its initial conflict-resolution pass before that
/// attempt, so this method also reads back and returns the identity of the
/// transaction that actually landed. Callers must compare it with
/// [`StagedWrite::transaction_identity`] AND require the returned dataset
/// version to equal `read_version + 1`: Lance's preflight rebase can
/// preserve the transaction fields while committing at a later version.
/// Either mismatch is a post-effect recovery case, not permission to widen
/// the prepared plan.
pub async fn commit_staged_exact(
&self,
ds: Arc<Dataset>,
staged: StagedWrite,
) -> Result<(Dataset, StagedTransactionIdentity)> {
let (dataset, committed_identity) = self
.commit_staged_with_retry_budget(ds, staged, Some(0))
.await?;
let committed_identity = committed_identity.ok_or_else(|| {
OmniError::manifest_internal(
"Lance committed a staged effect without a readable transaction identity",
)
})?;
Ok((dataset, committed_identity))
}
async fn commit_staged_with_retry_budget(
&self,
ds: Arc<Dataset>,
staged: StagedWrite,
max_retries: Option<u32>,
) -> Result<(Dataset, Option<StagedTransactionIdentity>)> {
// Skip Lance's auto-cleanup hook on every commit. OmniGraph owns version
// GC explicitly (optimize.rs::cleanup_all_tables); Lance's hook fires off
// the *dataset's stored* `lance.auto_cleanup.*` config, which graphs
@ -1272,13 +1388,27 @@ impl TableStore {
// data path) for new and legacy datasets alike, preventing Lance from
// GC'ing versions the __manifest still pins for snapshots/time-travel.
let mut builder = CommitBuilder::new(ds).with_skip_auto_cleanup(true);
if let Some(max_retries) = max_retries {
builder = builder.with_max_retries(max_retries);
}
if let Some(affected_rows) = staged.commit_metadata.affected_rows {
builder = builder.with_affected_rows(affected_rows);
}
builder
let dataset = builder
.execute(staged.transaction)
.await
.map_err(|e| OmniError::Lance(e.to_string()))
.map_err(|e| OmniError::Lance(e.to_string()))?;
let committed_identity = if max_retries.is_some() {
dataset
.read_transaction()
.await
.map_err(|e| OmniError::Lance(e.to_string()))?
.as_ref()
.map(StagedTransactionIdentity::from)
} else {
None
};
Ok((dataset, committed_identity))
}
/// Stage an overwrite (write_fragments + Operation::Overwrite { schema, fragments }).
@ -1581,33 +1711,119 @@ impl TableStore {
}
let committed = self.scan(committed_ds, projection, filter, None).await?;
if pending_batches.is_empty() {
return Ok(committed);
combine_committed_with_pending(
committed,
pending_batches,
pending_schema,
projection,
filter,
key_column,
)
.await
}
/// Read a blob-bearing table as a full logical merge source while retaining
/// [`Self::scan_with_pending`]'s merge-shadow semantics.
///
/// Lance normally scans blob-v2 columns as physical descriptor structs.
/// Those descriptors are a read representation, not valid writer input: a
/// full-row merge sends them through the blob writer, which requires the
/// logical `Struct<data, uri>` shape and fails with `Blob struct missing
/// data field`. This remained hidden while an `id` BTREE forced Lance's
/// partial-column merge plan; once index creation became reconciler-owned,
/// an index-absent table correctly selected the full-scan plan and exposed
/// the representation mismatch.
///
/// A first scan evaluates `filter` with blob columns excluded and retains
/// stable row ids. Full descriptor rows are then taken by those ids without
/// a filter, and only their payloads are rebuilt as logical
/// [`BlobArrayBuilder`] columns. Pending rows already carry logical blob
/// arrays; both sides have the dataset's full logical schema before the
/// existing shadow union. The resulting batch is therefore valid for either
/// of Lance's merge plans, making physical index presence a performance
/// detail rather than a correctness precondition.
pub async fn scan_with_pending_materialized_blobs(
&self,
committed_ds: &Dataset,
pending_batches: &[RecordBatch],
pending_schema: Option<SchemaRef>,
filter: Option<&str>,
key_column: Option<&str>,
) -> Result<Vec<RecordBatch>> {
let blob_columns = committed_ds
.schema()
.fields
.iter()
.filter(|field| field.is_blob())
.map(|field| field.name.clone())
.collect::<Vec<_>>();
if blob_columns.is_empty() {
return self
.scan_with_pending(
committed_ds,
pending_batches,
pending_schema,
None,
filter,
key_column,
)
.await;
}
// Shadow committed rows whose key value also appears in pending.
// This makes scan_with_pending implement merge semantics rather
// than naive union: any row that has a pending update is
// represented ONLY by its pending value, never by both its
// (stale) committed value and its (current) pending value.
let committed = match key_column {
Some(key_col) => {
let pending_keys = collect_string_column_values(pending_batches, key_col)?;
if pending_keys.is_empty() {
committed
} else {
filter_out_rows_where_string_in(committed, key_col, &pending_keys)?
}
// The pinned Lance revision cannot combine a predicate filter with a
// full blob-v2 projection: `FilteredReadExec` applies the descriptor
// child projection to the logical blob field and panics. Select the
// matched rows without blob columns, retaining stable `_rowid`, then
// take exactly those full descriptor rows without a filter and rebuild
// their logical blobs. Thus payload I/O remains proportional to matched
// rows while avoiding any dependence on which merge/index plan Lance
// selects later.
let non_blob_columns = committed_ds
.schema()
.fields
.iter()
.filter(|field| !field.is_blob())
.map(|field| field.name.as_str())
.collect::<Vec<_>>();
let matched = Self::scan_stream(committed_ds, Some(&non_blob_columns), filter, None, true)
.await?
.try_collect::<Vec<RecordBatch>>()
.await
.map_err(|error| OmniError::Lance(error.to_string()))?;
let mut committed = Vec::with_capacity(matched.len());
for batch in matched {
let row_ids = batch
.column_by_name("_rowid")
.and_then(|column| column.as_any().downcast_ref::<UInt64Array>())
.ok_or_else(|| {
OmniError::Lance("expected _rowid in predicate-matched blob scan".to_string())
})?
.values()
.to_vec();
if row_ids.is_empty() {
continue;
}
None => committed,
};
let descriptors = committed_ds
.take_rows(&row_ids, committed_ds.schema().clone())
.await
.map_err(|error| OmniError::Lance(error.to_string()))?;
committed.push(
Self::materialize_blob_batch_with_row_ids(committed_ds, descriptors, &row_ids)
.await?,
);
}
let pending =
scan_pending_batches(pending_batches, pending_schema, projection, filter).await?;
let mut out = committed;
out.extend(pending);
Ok(out)
let combined = combine_committed_with_pending(
committed,
pending_batches,
pending_schema,
None,
filter,
key_column,
)
.await?;
Ok(combined)
}
/// `count_rows` variant that respects staged writes. Used for
@ -1844,6 +2060,42 @@ fn assign_row_id_meta(fragments: &mut [Fragment], start_row_id: u64) -> Result<(
Ok(())
}
/// Apply the in-memory half of `scan_with_pending` to an already-scanned
/// committed view. Kept as one helper so the ordinary descriptor scan and the
/// blob-materializing scan cannot drift in their key-shadow semantics.
async fn combine_committed_with_pending(
committed: Vec<RecordBatch>,
pending_batches: &[RecordBatch],
pending_schema: Option<SchemaRef>,
projection: Option<&[&str]>,
filter: Option<&str>,
key_column: Option<&str>,
) -> Result<Vec<RecordBatch>> {
if pending_batches.is_empty() {
return Ok(committed);
}
// Shadow committed rows whose key value also appears in pending. This is
// deliberately applied before filtering pending: a pending row that no
// longer matches must still suppress its stale committed predecessor.
let committed = match key_column {
Some(key_col) => {
let pending_keys = collect_string_column_values(pending_batches, key_col)?;
if pending_keys.is_empty() {
committed
} else {
filter_out_rows_where_string_in(committed, key_col, &pending_keys)?
}
}
None => committed,
};
let pending = scan_pending_batches(pending_batches, pending_schema, projection, filter).await?;
let mut out = committed;
out.extend(pending);
Ok(out)
}
/// Collect the set of values in a Utf8 column across multiple batches.
/// Used by `scan_with_pending`'s merge-semantic path to identify
/// committed rows that are shadowed by pending writes. NULL values are

35
crates/omnigraph/src/validate.rs
View file

@ -238,9 +238,10 @@ pub(crate) struct CommittedState<'a> {
/// tables, not a global flag — an edges-only overwrite still sees committed
/// nodes for RI. Empty on the merge / mutation / append / merge-load paths.
overwritten: HashSet<String>,
/// Write path only: open edge tables at LIVE HEAD for `@card` (the #298
/// stale-handle fix). `None` on the merge and load paths — there the snapshot
/// (or empty) is the committed view for every check.
/// Write path only: open edge tables from a fresh graph-branch manifest
/// snapshot for `@card` (the #298 stale-handle fix). This is the live
/// committed graph view, not a raw Lance HEAD that may be unpublished or
/// belong to an inherited source ref. `None` on merge/load.
live: Option<(&'a Omnigraph, Option<&'a str>)>,
}
@ -255,7 +256,8 @@ impl<'a> CommittedState<'a> {
}
/// Write path: existence/uniqueness read `committed` (the write's pinned
/// base); cardinality reads LIVE HEAD per edge table via `db` (#298).
/// base); cardinality reads the live committed branch snapshot via `db`
/// (#298).
pub(crate) fn write(committed: &'a Snapshot, db: &'a Omnigraph, branch: Option<&'a str>) -> Self {
Self {
committed: Some(committed),
@ -300,10 +302,12 @@ impl<'a> CommittedState<'a> {
}
}
/// Open an edge table for cardinality counting: LIVE HEAD on the write path
/// (so an edge a concurrent writer committed since the base was pinned is
/// counted — #298), the committed snapshot otherwise. `None` if there is no
/// committed view (Overwrite load) or the table isn't in it.
/// Open an edge table for cardinality counting: the current manifest-visible
/// graph-branch snapshot on the write path (so a concurrent published edge
/// is counted — #298), the pinned committed snapshot otherwise. Resolving
/// through the fresh graph snapshot is load-bearing for first-touch named
/// branches: their table still inherits another Lance ref until this write's
/// sidecar is armed, so opening the target ref directly would be invalid.
async fn open_cardinality(&self, table_key: &str) -> Result<Option<Dataset>> {
if self.overwritten.contains(table_key) {
return Ok(None);
@ -311,14 +315,21 @@ impl<'a> CommittedState<'a> {
let Some(committed) = self.committed else {
return Ok(None);
};
let Some(entry) = committed.entry(table_key) else {
let Some(_entry) = committed.entry(table_key) else {
return Ok(None);
};
match self.live {
Some((db, branch)) => {
let full_path = db.storage().dataset_uri(&entry.table_path);
let handle = db.storage().open_dataset_head(&full_path, branch).await?;
Ok(Some(handle.dataset().clone()))
// `CommittedState::write` is constructed only after WriteTxn
// schema validation, so use the unchecked manifest refresh to
// avoid another full contract read while retaining live branch
// authority. Snapshot::open follows the entry's actual
// `table_branch` and pinned version (including inheritance).
let live = db.fresh_snapshot_for_branch_unchecked(branch).await?;
match live.entry(table_key) {
Some(_) => Ok(Some(live.open(table_key).await?)),
None => Ok(None),
}
}
None => Ok(Some(committed.open(table_key).await?)),
}

31
crates/omnigraph/tests/branching.rs
View file

@ -351,6 +351,22 @@ async fn branch_merge_with_blob_columns_preserves_blob_data() {
)
.await
.unwrap();
// This regression must not rely on an incidental physical index selecting
// Lance's legacy partial-column merge plan. The materialized-blob update
// path is correct even when the table has no user index at all.
let ds = snapshot_main(&main)
.await
.unwrap()
.open("node:Document")
.await
.unwrap();
let indices = ds.load_indices().await.unwrap();
assert!(
indices.iter().all(is_system_index),
"blob correctness regression requires an index-absent table"
);
main.branch_create("feature").await.unwrap();
let mut feature = Omnigraph::open(uri).await.unwrap();
@ -629,7 +645,10 @@ async fn same_branch_insert_after_external_commit_is_linear() {
.iter()
.filter(|c| c.parent_commit_id.as_deref() == Some(c0.graph_commit_id.as_str()))
.count();
assert_eq!(c0_children, 1, "C0 must have exactly one child; two is the fork");
assert_eq!(
c0_children, 1,
"C0 must have exactly one child; two is the fork"
);
}
/// Strict update after a read: Fix 1's `refresh_manifest_only` makes the read
@ -676,7 +695,10 @@ async fn same_branch_update_after_external_commit_and_read_is_linear() {
.await
.unwrap()
.unwrap();
assert_eq!(cb.parent_commit_id.as_deref(), Some(ca.graph_commit_id.as_str()));
assert_eq!(
cb.parent_commit_id.as_deref(),
Some(ca.graph_commit_id.as_str())
);
// A reads main: the stale-probe path refreshes A's MANIFEST (via
// refresh_manifest_only) but not its commit-graph head, freshening the
@ -713,7 +735,10 @@ async fn same_branch_update_after_external_commit_and_read_is_linear() {
.iter()
.filter(|c| c.parent_commit_id.as_deref() == Some(ca.graph_commit_id.as_str()))
.count();
assert_eq!(ca_children, 1, "Ca must have exactly one child; two is the fork");
assert_eq!(
ca_children, 1,
"Ca must have exactly one child; two is the fork"
);
}
#[tokio::test]

37
crates/omnigraph/tests/consistency.rs
View file

@ -815,16 +815,15 @@ query high_value() {
assert_eq!(values.value(8), 499);
}
// ─── Stale handle must refresh-and-retry (no silent rebase) ──────────────
// ─── Long-lived strict handle refreshes before preparation ───────────────
#[tokio::test]
async fn stale_handle_public_mutation_must_refresh_then_retry() {
// With the Run state machine removed, the engine no longer
// auto-rebases stale-handle mutations onto the latest target head.
// The publisher's `expected_table_versions` CAS makes the contract
// explicit — a stale writer fails loudly with
// `ExpectedVersionMismatch` and the client decides whether to
// refresh-and-retry.
async fn long_lived_handle_prepares_strict_mutation_from_current_head() {
// Merely opening a handle before another completed commit does not make the
// next attempt stale: open_write_txn probes the manifest incarnation and
// captures the current branch authority before it plans the strict update.
// ReadSetChanged is reserved for movement during an already-prepared
// attempt (covered deterministically in the failpoint suite).
let dir = tempfile::tempdir().unwrap();
let _db = init_and_load(&dir).await;
drop(_db);
@ -843,26 +842,8 @@ async fn stale_handle_public_mutation_must_refresh_then_retry() {
.await
.unwrap();
// Writer 2 is now stale. Its first attempt must fail with
// ExpectedVersionMismatch — no silent rebase.
let stale_err = mutate_main(
&mut db2,
MUTATION_QUERIES,
"set_age",
&mixed_params(&[("$name", "Alice")], &[("$age", 99)]),
)
.await
.expect_err("stale writer must hit ExpectedVersionMismatch");
let omnigraph::error::OmniError::Manifest(manifest_err) = stale_err else {
panic!("expected Manifest error");
};
assert!(matches!(
manifest_err.details,
Some(omnigraph::error::ManifestConflictDetails::ExpectedVersionMismatch { .. })
));
// Refresh and retry — the canonical client recovery path.
db2.sync_branch("main").await.unwrap();
// Writer 2's handle predates Eve, but its strict attempt prepares from the
// current head and succeeds in one call.
mutate_main(
&mut db2,
MUTATION_QUERIES,

2373
crates/omnigraph/tests/failpoints.rs
View file

File diff suppressed because it is too large Load diff

9
crates/omnigraph/tests/helpers/cost.rs
View file

@ -202,7 +202,8 @@ impl ObjectStore for PrefixCountingStore {
}
fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, OSResult<ObjectMeta>> {
self.counter.record_read(&prefix.cloned().unwrap_or_default());
self.counter
.record_read(&prefix.cloned().unwrap_or_default());
self.target.list(prefix)
}
@ -211,12 +212,14 @@ impl ObjectStore for PrefixCountingStore {
prefix: Option<&Path>,
offset: &Path,
) -> BoxStream<'static, OSResult<ObjectMeta>> {
self.counter.record_read(&prefix.cloned().unwrap_or_default());
self.counter
.record_read(&prefix.cloned().unwrap_or_default());
self.target.list_with_offset(prefix, offset)
}
async fn list_with_delimiter(&self, prefix: Option<&Path>) -> OSResult<ListResult> {
self.counter.record_read(&prefix.cloned().unwrap_or_default());
self.counter
.record_read(&prefix.cloned().unwrap_or_default());
self.target.list_with_delimiter(prefix).await
}

12
crates/omnigraph/tests/helpers/mod.rs
View file

@ -61,6 +61,9 @@ pub async fn init_and_load(dir: &tempfile::TempDir) -> Omnigraph {
load_jsonl(&mut db, TEST_DATA, LoadMode::Overwrite)
.await
.unwrap();
// Mutation/load publish only exact data effects; physical indexes are
// reconciled separately as derived state.
db.ensure_indices().await.unwrap();
db
}
@ -211,15 +214,18 @@ pub async fn commit_many(db: &mut Omnigraph, n: usize) {
}
}
/// Like [`commit_many`] but every commit carries an actor, so it grows
/// `_graph_commit_actors.lance` too — the authenticated (server/CLI) write path.
/// Like [`commit_many`] but every commit carries an actor in its inline
/// `__manifest` lineage row — the authenticated (server/CLI) write path.
pub async fn commit_many_as(db: &mut Omnigraph, n: usize, actor: &str) {
for i in 0..n {
db.mutate_as(
"main",
MUTATION_QUERIES,
"insert_person",
&mixed_params(&[("$name", &format!("commit_many_as_{i}"))], &[("$age", 30)]),
&mixed_params(
&[("$name", &format!("commit_many_as_{i}"))],
&[("$age", 30)],
),
Some(actor),
)
.await

51
crates/omnigraph/tests/helpers/recovery.rs
View file

@ -13,8 +13,17 @@ const RECOVERY_ACTOR: &str = "omnigraph:recovery";
#[derive(Debug)]
pub enum RecoveryExpectation {
RolledForward { tables: Vec<TableExpectation> },
RolledBack { tables: Vec<TableExpectation> },
RolledForward {
tables: Vec<TableExpectation>,
},
/// Protocol-v3 mutation/load recovery republishes the writer's fixed
/// lineage intent rather than minting a synthetic recovery commit.
RolledForwardOriginalLineage {
tables: Vec<TableExpectation>,
},
RolledBack {
tables: Vec<TableExpectation>,
},
Deferred,
NoOp,
}
@ -41,6 +50,7 @@ pub struct FollowUpMutation {
struct RecoveryAuditRow {
graph_commit_id: String,
recovery_kind: String,
recovery_for_actor: Option<String>,
operation_id: String,
sidecar_writer_kind: String,
per_table_outcomes: Vec<TableOutcome>,
@ -201,7 +211,21 @@ pub async fn assert_post_recovery_invariants(
);
assert_manifest_pins_match_lance_heads(graph_root, &tables).await?;
assert_audit_to_versions_match_lance_heads(graph_root, &audit, &tables).await?;
assert_recovery_commit_shape(graph_root, &audit, &tables).await?;
assert_recovery_commit_shape(graph_root, &audit, &tables, false).await?;
assert_non_main_did_not_move_main(graph_root, &tables).await?;
assert_idempotent_reopen(graph_root, operation_id).await?;
run_follow_up_mutations(graph_root, tables).await?;
}
RecoveryExpectation::RolledForwardOriginalLineage { tables } => {
assert_sidecar_absent(graph_root, operation_id);
let audit = read_audit_row(graph_root, operation_id).await?;
assert_eq!(
audit.recovery_kind, "RolledForward",
"audit row for {operation_id} recorded the wrong recovery_kind",
);
assert_manifest_pins_match_lance_heads(graph_root, &tables).await?;
assert_audit_to_versions_match_lance_heads(graph_root, &audit, &tables).await?;
assert_recovery_commit_shape(graph_root, &audit, &tables, true).await?;
assert_non_main_did_not_move_main(graph_root, &tables).await?;
assert_idempotent_reopen(graph_root, operation_id).await?;
run_follow_up_mutations(graph_root, tables).await?;
@ -217,7 +241,7 @@ pub async fn assert_post_recovery_invariants(
// Roll-back now publishes the restored HEAD, so manifest == Lance
// HEAD afterward (symmetric with roll-forward) — no residual drift.
assert_manifest_pins_match_lance_heads(graph_root, &tables).await?;
assert_recovery_commit_shape(graph_root, &audit, &tables).await?;
assert_recovery_commit_shape(graph_root, &audit, &tables, false).await?;
assert_non_main_did_not_move_main(graph_root, &tables).await?;
assert_idempotent_reopen(graph_root, operation_id).await?;
run_follow_up_mutations(graph_root, tables).await?;
@ -366,19 +390,29 @@ async fn assert_recovery_commit_shape(
graph_root: &Path,
audit: &RecoveryAuditRow,
tables: &[TableExpectation],
preserves_original_intent: bool,
) -> Result<()> {
let branch = branch_context(tables);
let expected_parent = expected_recovery_parent(tables)?;
let branch = branch.as_deref();
let commit = read_recovery_commit(graph_root, audit, branch).await?;
// RFC-022 mutation/load roll-forward publishes the writer's original,
// pre-minted lineage intent. That preserves its original actor (including
// `None`) and makes a crash after publish distinguishable from a new
// synthetic recovery content commit. Rollback and legacy writer recovery
// still publish an explicit `omnigraph:recovery` commit.
let expected_actor = if preserves_original_intent {
audit.recovery_for_actor.as_deref()
} else {
Some(RECOVERY_ACTOR)
};
assert_eq!(
commit.actor_id.as_deref(),
Some(RECOVERY_ACTOR),
"recovery commit {} for operation {} must use actor {}",
expected_actor,
"recovery commit {} for operation {} recorded the wrong actor",
commit.graph_commit_id,
audit.operation_id,
RECOVERY_ACTOR,
);
if let Some(expected_parent) = expected_parent {
@ -573,6 +607,7 @@ async fn matching_audit_rows(
for batch in batches {
let graph_commit_ids = string_column(&batch, "graph_commit_id")?;
let kinds = string_column(&batch, "recovery_kind")?;
let recovery_actors = string_column(&batch, "recovery_for_actor")?;
let ops = string_column(&batch, "operation_id")?;
let writers = string_column(&batch, "sidecar_writer_kind")?;
let outcomes_json = string_column(&batch, "per_table_outcomes_json")?;
@ -589,6 +624,8 @@ async fn matching_audit_rows(
rows.push(RecoveryAuditRow {
graph_commit_id: graph_commit_ids.value(row).to_string(),
recovery_kind: kinds.value(row).to_string(),
recovery_for_actor: (!recovery_actors.is_null(row))
.then(|| recovery_actors.value(row).to_string()),
operation_id: ops.value(row).to_string(),
sidecar_writer_kind: writers.value(row).to_string(),
per_table_outcomes,

2
crates/omnigraph/tests/lineage_projection.rs
View file

@ -1,7 +1,7 @@
//! RFC-013 Phase 7 acceptance gate: graph lineage lives ONLY in `__manifest`.
//!
//! The `graph_commit` + `graph_head` rows ride the same publish CAS as the
//! table-version rows, so `_graph_commits.lance` carries NO commit rows. This
//! table-version rows, so no standalone commit-lineage dataset exists. This
//! gate proves two things over a realistic history (commits on main, a branch,
//! a merge, all with actors):
//!

38
crates/omnigraph/tests/maintenance.rs
View file

@ -361,7 +361,8 @@ node Doc {
let uri = dir.path().to_str().unwrap();
let mut db = Omnigraph::init(uri, SCHEMA).await.unwrap();
// First load builds the id + rank BTREEs over the initial fragment.
// Loads publish only data effects; establish the initial id + rank BTREEs
// explicitly through the reconciler before creating partial coverage.
load_jsonl(
&mut db,
"{\"type\":\"Doc\",\"data\":{\"slug\":\"d1\",\"rank\":1}}\n\
@ -370,6 +371,7 @@ node Doc {
)
.await
.unwrap();
db.ensure_indices().await.unwrap();
// A second load with NEW keys appends a fragment the existing BTREEs do not
// cover (the existence gate skips re-building an index that already exists).
@ -411,13 +413,9 @@ node Doc {
);
}
// Regression: `optimize` must not crash on a graph that has a `Blob` table.
//
// Lance `compact_files` forces `BlobHandling::AllBinary`, which mis-decodes
// blob-v2 columns ("more fields in the schema than provided column indices"),
// failing even a pristine uniform-V2_2 multi-fragment blob table. `optimize`
// must skip blob-bearing tables (and report the skip) rather than aborting the
// whole sweep.
// Regression: `optimize` must compact a graph that has a `Blob` table through
// the same positive path as every other data table; a reintroduced skip would
// hide both fragment growth and a Lance compatibility regression.
//
// History: through Lance 7.0.0 `compact_files` mis-decoded blob-v2 columns, so
// `optimize` skipped blob tables (`SkipReason::BlobColumnsUnsupportedByLance`)
@ -431,16 +429,17 @@ node Doc {
async fn optimize_compacts_blob_table_alongside_plain_table() {
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().to_str().unwrap();
// One Blob node type (`Doc`) + one plain node type (`Tag`): proves the blob
// table is skipped while a non-blob table in the same sweep still compacts.
// One Blob node type (`Doc`) + one plain node type (`Tag`): proves both use
// the normal compaction path in the same sweep.
let schema = "\
node Doc {\n slug: String @key\n content: Blob\n}\n\
node Tag {\n slug: String @key\n}\n";
let mut db = Omnigraph::init(uri, schema).await.unwrap();
// Multi-fragment blob table: Overwrite creates fragment 1; each Merge of
// new keys appends another. A >=2-fragment blob table is exactly what
// crashes `compact_files` today (single fragment would no-op and not crash).
// new keys appends another. A >=2-fragment blob table exercises the rewrite
// path that exposed the historical Lance regression (a single fragment
// would be a no-op).
load_jsonl(
&mut db,
"{\"type\":\"Doc\",\"data\":{\"slug\":\"d1\",\"content\":\"base64:aGVsbG8x\"}}\n{\"type\":\"Doc\",\"data\":{\"slug\":\"d2\",\"content\":\"base64:aGVsbG8y\"}}",
@ -1133,15 +1132,12 @@ async fn cleanup_reconciles_live_branch_orphan_fork_but_keeps_legitimate_fork()
}
// Regression (iss-848): a table with rows but NULL vectors (the load-before-
// embed window) must not abort index building. The vector (IVF) index cannot
// train on 0 vectors, so `create_vector_index` errors with "KMeans cannot
// train 1 centroids with 0 vectors". `build_indices_on_dataset_for_catalog`
// is the chokepoint every caller funnels through (load/mutate via
// prepare_updates_for_commit, ensure_indices, optimize, schema apply, merge),
// so per-index fault isolation there must defer that one column (pending) and
// still build the sibling scalar indexes, instead of propagating the error.
// This exercises both the load path (which builds indices inline) and the
// ensure_indices reconciler. Pre-fix this fails at the load step.
// embed window) must remain writable and reconcilable. RFC-022-enrolled writes
// publish only their logical data effect; physical indexes are derived work.
// The vector (IVF) index cannot train on 0 vectors, so the index chokepoint must
// defer that column as pending while still building eligible sibling indexes.
// This exercises both halves of the contract: the logical load succeeds without
// inline index work, then `ensure_indices` tolerates the untrainable vector.
#[tokio::test]
async fn index_build_tolerates_null_vector_rows() {
let dir = tempfile::tempdir().unwrap();

6
crates/omnigraph/tests/merge_fast_forward.rs
View file

@ -127,8 +127,8 @@ async fn fast_forward_merge_defers_vector_index_to_reconciler() {
let main = Omnigraph::init(uri, VEC_SCHEMA).await.unwrap();
main.branch_create("feature").await.unwrap();
// Load embedding-bearing chunks onto the branch. The branch builds its own
// index here (outside the probe scope) — irrelevant to the merge's cost.
// Load embedding-bearing chunks onto the branch. Load publishes only the
// data effect, so the declared vector index remains pending here too.
let mut rows = String::new();
for i in 0..24 {
let v: Vec<String> = (0..8).map(|j| format!("{}.0", (i + j) % 5)).collect();
@ -140,7 +140,7 @@ async fn fast_forward_merge_defers_vector_index_to_reconciler() {
let feature = Omnigraph::open(uri).await.unwrap();
feature.load("feature", &rows, LoadMode::Merge).await.unwrap();
// Merge, counting inline vector-index builds the publish path performs.
// Merge, asserting that its publish path performs no inline vector-index build.
let probes = MergeWriteProbes::default();
let outcome = with_merge_write_probes(probes.clone(), main.branch_merge("feature", "main"))
.await

10
crates/omnigraph/tests/recovery.rs
View file

@ -1172,12 +1172,10 @@ async fn recovery_ensure_indices_steady_state_no_sidecar() {
/// `count_rows == 0 → return false` short-circuit in `needs_index_work_*`
/// is what makes this work.
///
/// A stronger assertion that captured the sidecar mid-flight and verified
/// the persisted JSON omits empty tables would require bypassing
/// `load_jsonl` (which auto-builds indices via
/// `prepare_updates_for_commit`); pinning that with a unit test on the
/// helpers directly would require bootstrapping an engine plus raw Lance
/// writes — left as a follow-up.
/// A stronger assertion that captured the sidecar after arming but before the
/// first index effect could inspect the persisted pin set directly. That needs
/// a dedicated pre-effect EnsureIndices rendezvous; the current failpoint is
/// after its effects, so this remains an end-to-end behavioral assertion.
#[tokio::test]
async fn recovery_ensure_indices_handles_empty_tables() {
let dir = tempfile::tempdir().unwrap();

5
crates/omnigraph/tests/scalar_indexes.rs
View file

@ -30,8 +30,8 @@ const DATA: &str = r#"{"type":"Item","data":{"slug":"a","status":"active","publi
{"type":"Item","data":{"slug":"b","status":"archived","published":"2023-01-01T00:00:00Z","rank":2,"title":"beta","note":"n2"}}
{"type":"Item","data":{"slug":"c","status":"active","published":"2025-02-02T00:00:00Z","rank":3,"title":"gamma","note":"n3"}}"#;
// Enums and orderable scalars (DateTime, numeric) get a BTREE from load's
// build-indices pass, so a `=`/range filter on them uses the index. Free-text
// Enums and orderable scalars (DateTime, numeric) get a BTREE from the index
// reconciler, so a `=`/range filter on them uses the index. Free-text
// String `@index` keeps FTS (no BTREE), and an un-annotated column has no
// scalar index — both report `Degraded`, which is the negative control that
// keeps this test from being vacuously green.
@ -41,6 +41,7 @@ async fn node_scalar_and_enum_index_columns_get_btree() {
let uri = dir.path().to_str().unwrap();
let mut db = Omnigraph::init(uri, SCHEMA).await.unwrap();
load_jsonl(&mut db, DATA, LoadMode::Overwrite).await.unwrap();
db.ensure_indices().await.unwrap();
let snap = snapshot_main(&db).await.unwrap();
let ds = snap.open("node:Item").await.unwrap();

381
crates/omnigraph/tests/schema_apply.rs
View file

@ -1,14 +1,17 @@
mod helpers;
use std::fs;
#[cfg(feature = "failpoints")]
use std::sync::Arc;
use omnigraph::db::{Omnigraph, ReadTarget};
use omnigraph::db::{MergeOutcome, Omnigraph, ReadTarget};
use omnigraph::loader::{LoadMode, load_jsonl};
use omnigraph_compiler::{SchemaMigrationStep, SchemaTypeKind};
use helpers::*;
#[tokio::test]
#[cfg_attr(feature = "failpoints", serial_test::parallel)]
async fn plan_schema_reports_supported_additive_change() {
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().to_str().unwrap();
@ -33,6 +36,322 @@ async fn plan_schema_reports_supported_additive_change() {
}
#[tokio::test]
#[cfg_attr(feature = "failpoints", serial_test::parallel)]
async fn long_lived_handle_uses_the_schema_catalog_bound_to_its_write_token() {
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().to_str().unwrap();
let schema_owner = Omnigraph::init(uri, TEST_SCHEMA).await.unwrap();
// Open before the migration: this handle's process-local ArcSwap catalog is
// intentionally stale after `schema_owner` completes the apply.
let stale_handle = Omnigraph::open(uri).await.unwrap();
let desired = format!(
"{}\nnode Project {{\n name: String @key\n}}\n",
TEST_SCHEMA.replace(
" age: I32?\n}",
" age: I32?\n nickname: String?\n}",
)
);
schema_owner.apply_schema(&desired).await.unwrap();
let mutation = r#"
query insert_with_nickname($name: String, $age: I32, $nickname: String) {
insert Person { name: $name, age: $age, nickname: $nickname }
}
"#;
let inserted = stale_handle
.mutate(
"main",
mutation,
"insert_with_nickname",
&mixed_params(
&[("$name", "mutated-after-schema"), ("$nickname", "fresh")],
&[("$age", 31)],
),
)
.await
.expect("mutation must typecheck and build its batch with the token-bound catalog");
assert_eq!(inserted.affected_nodes, 1);
let loaded = stale_handle
.load(
"main",
r#"{"type":"Person","data":{"name":"loaded-after-schema","age":32,"nickname":"fresh"}}"#,
LoadMode::Merge,
)
.await
.expect("load parsing and validation must use the same token-bound catalog");
assert_eq!(loaded.nodes_loaded.get("Person"), Some(&1));
assert_eq!(count_rows(&stale_handle, "node:Person").await, 2);
// The same stale handle must bind branch-merge planning and conservative
// branch-control table gates to the accepted contract captured under the
// schema gate. The warm handle catalog predates Project; consulting it here
// would fail with `unknown node type` (or omit Project's control queue).
stale_handle.branch_create("source").await.unwrap();
stale_handle.branch_create("target").await.unwrap();
let project_mutation = r#"
query insert_project($name: String) {
insert Project { name: $name }
}
"#;
stale_handle
.mutate(
"source",
project_mutation,
"insert_project",
&params(&[("$name", "fresh-catalog-project")]),
)
.await
.expect("source write must use the token-bound post-apply catalog");
assert_eq!(
stale_handle
.branch_merge("source", "target")
.await
.expect("merge planning must use the schema-gated post-apply catalog"),
MergeOutcome::FastForward
);
assert_eq!(
count_rows_branch(&stale_handle, "target", "node:Project").await,
1
);
}
/// Native branch controls must enumerate their conservative table envelope
/// from the accepted catalog captured under the schema gate, not a long-lived
/// handle's pre-apply ArcSwap. Park delete after that envelope is held and prove
/// a legacy Project-only index reconciler cannot cross its table queue.
#[cfg(feature = "failpoints")]
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial_test::serial]
async fn stale_handle_branch_delete_gates_tables_added_by_schema_apply() {
use omnigraph::failpoints::names;
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().to_str().unwrap();
let schema_owner = Omnigraph::init(uri, TEST_SCHEMA).await.unwrap();
let stale_control = Arc::new(Omnigraph::open(uri).await.unwrap());
let desired = format!("{TEST_SCHEMA}\nnode Project {{\n name: String @key\n}}\n");
schema_owner.apply_schema(&desired).await.unwrap();
schema_owner.branch_create("target").await.unwrap();
schema_owner
.load(
"target",
r#"{"type":"Project","data":{"name":"pending-index"}}"#,
LoadMode::Merge,
)
.await
.unwrap();
let index_reconciler = Arc::new(Omnigraph::open(uri).await.unwrap());
let delete_rv =
helpers::failpoint::Rendezvous::park_first(names::BRANCH_DELETE_POST_TABLE_GATES);
let delete_handle = Arc::clone(&stale_control);
let delete_task = tokio::spawn(async move { delete_handle.branch_delete("target").await });
delete_rv.wait_until_reached().await;
let index_handle = Arc::clone(&index_reconciler);
let mut index_task = tokio::spawn(async move { index_handle.ensure_indices_on("target").await });
let index_blocked = tokio::time::timeout(
std::time::Duration::from_millis(250),
&mut index_task,
)
.await
.is_err();
delete_rv.release();
assert!(
index_blocked,
"stale control catalog omitted the newly-added Project table gate"
);
delete_task.await.unwrap().unwrap();
if tokio::time::timeout(std::time::Duration::from_secs(10), &mut index_task)
.await
.is_err()
{
index_task.abort();
let _ = index_task.await;
panic!("index reconciler did not finish after branch delete released its table gate");
}
}
#[cfg(feature = "failpoints")]
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial_test::serial]
async fn mutation_waits_for_mid_apply_schema_gate_then_reprepares() {
use omnigraph::failpoints::names;
let dir = tempfile::tempdir().unwrap();
let db = Arc::new(init_and_load(&dir).await);
let desired = TEST_SCHEMA.replace(
" age: I32?\n}",
" age: I32?\n nickname: String?\n}",
);
// First park the mutation after all validation/staging but before it enters
// the schema→branch→table effect gates. This fixes the otherwise tiny race
// window deterministically.
let mutation_rv =
helpers::failpoint::Rendezvous::park_first(names::MUTATION_POST_STAGE_PRE_EFFECT_GATE);
let mutation_db = Arc::clone(&db);
let mutation_task = tokio::spawn(async move {
mutation_db
.mutate(
"main",
MUTATION_QUERIES,
"insert_person",
&mixed_params(&[("$name", "schema-gated")], &[("$age", 33)]),
)
.await
});
mutation_rv.wait_until_reached().await;
// Start schema apply and park it after its staging files (and any table
// rewrite) exist but before manifest/schema promotion. The outer apply owns
// the schema-control gate throughout this window.
let schema_rv =
helpers::failpoint::Rendezvous::park_first(names::SCHEMA_APPLY_AFTER_STAGING_WRITE);
let schema_db = Arc::clone(&db);
let schema_task = tokio::spawn(async move { schema_db.apply_schema(&desired).await });
schema_rv.wait_until_reached().await;
mutation_rv.release();
// Give the already-runnable mutation repeated scheduler turns. It must stay
// pending on the schema gate; completing here means it either advanced under
// an in-flight migration or returned a spurious post-prepare failure.
for _ in 0..128 {
tokio::task::yield_now().await;
if mutation_task.is_finished() {
break;
}
}
assert!(
!mutation_task.is_finished(),
"mutation must remain behind the schema-control gate while apply is in flight",
);
schema_rv.release();
schema_task.await.unwrap().unwrap();
let result = mutation_task
.await
.unwrap()
.expect("insert-only mutation must reprepare under the promoted schema");
assert_eq!(result.affected_nodes, 1);
assert_eq!(count_rows(&db, "node:Person").await, 5);
}
/// ReadOnly opens participate in the process-local schema publication gate even
/// though they perform no recovery writes. Park an open immediately before its
/// source/IR/state read: schema apply must not reach staging until that coherent
/// catalog capture finishes.
#[cfg(feature = "failpoints")]
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial_test::serial]
async fn read_only_open_holds_schema_gate_through_catalog_capture() {
use omnigraph::failpoints::names;
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().to_str().unwrap().to_string();
let owner = Arc::new(init_and_load(&dir).await);
let desired = TEST_SCHEMA.replace(
" age: I32?\n}",
" age: I32?\n nickname: String?\n}",
);
let open_rv =
helpers::failpoint::Rendezvous::park_first(names::OPEN_BEFORE_SCHEMA_CONTRACT_READ);
let open_uri = uri.clone();
let open_task = tokio::spawn(async move { Omnigraph::open_read_only(&open_uri).await });
open_rv.wait_until_reached().await;
let apply_rv =
helpers::failpoint::Rendezvous::park_first(names::SCHEMA_APPLY_AFTER_STAGING_WRITE);
let apply_owner = Arc::clone(&owner);
let apply_task = tokio::spawn(async move { apply_owner.apply_schema(&desired).await });
assert!(
tokio::time::timeout(
std::time::Duration::from_millis(200),
apply_rv.wait_until_reached(),
)
.await
.is_err(),
"schema apply must remain behind the ReadOnly catalog-capture gate",
);
open_rv.release();
let opened = open_task.await.unwrap().unwrap();
assert!(
!opened.catalog().node_types["Person"]
.properties
.contains_key("nickname"),
"the serialized open must publish the complete pre-apply catalog"
);
apply_rv.wait_until_reached().await;
apply_rv.release();
apply_task.await.unwrap().unwrap();
let current = Omnigraph::open_read_only(&uri).await.unwrap();
assert!(
current.catalog().node_types["Person"]
.properties
.contains_key("nickname"),
"the next open must publish the complete post-apply catalog"
);
}
/// Refresh must reacquire the schema gate after sidecar healing and retain it
/// through the ArcSwap publication. Otherwise a concurrent three-file schema
/// promotion can be interleaved with its contract read.
#[cfg(feature = "failpoints")]
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial_test::serial]
async fn refresh_holds_schema_gate_through_catalog_publication() {
use omnigraph::failpoints::names;
let dir = tempfile::tempdir().unwrap();
let owner = Arc::new(init_and_load(&dir).await);
let stale = Arc::new(Omnigraph::open(dir.path().to_str().unwrap()).await.unwrap());
let desired = TEST_SCHEMA.replace(
" age: I32?\n}",
" age: I32?\n nickname: String?\n}",
);
let reload_rv =
helpers::failpoint::Rendezvous::park_first(names::SCHEMA_RELOAD_BEFORE_CONTRACT_READ);
let refresh_handle = Arc::clone(&stale);
let refresh_task = tokio::spawn(async move { refresh_handle.refresh().await });
reload_rv.wait_until_reached().await;
let apply_rv =
helpers::failpoint::Rendezvous::park_first(names::SCHEMA_APPLY_AFTER_STAGING_WRITE);
let apply_owner = Arc::clone(&owner);
let apply_task = tokio::spawn(async move { apply_owner.apply_schema(&desired).await });
assert!(
tokio::time::timeout(
std::time::Duration::from_millis(200),
apply_rv.wait_until_reached(),
)
.await
.is_err(),
"schema apply must remain behind refresh's catalog-publication gate",
);
reload_rv.release();
refresh_task.await.unwrap().unwrap();
apply_rv.wait_until_reached().await;
apply_rv.release();
apply_task.await.unwrap().unwrap();
stale.refresh().await.unwrap();
assert!(
stale.catalog().node_types["Person"]
.properties
.contains_key("nickname"),
"a post-apply refresh must publish the complete new catalog"
);
}
#[tokio::test]
#[cfg_attr(feature = "failpoints", serial_test::parallel)]
async fn plan_schema_rejects_when_schema_contract_has_drifted() {
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().to_str().unwrap();
@ -49,6 +368,7 @@ async fn plan_schema_rejects_when_schema_contract_has_drifted() {
}
#[tokio::test]
#[cfg_attr(feature = "failpoints", serial_test::parallel)]
async fn apply_schema_noop_returns_not_applied() {
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().to_str().unwrap();
@ -61,6 +381,7 @@ async fn apply_schema_noop_returns_not_applied() {
}
#[tokio::test]
#[cfg_attr(feature = "failpoints", serial_test::parallel)]
async fn apply_schema_rejects_when_non_main_branch_exists() {
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().to_str().unwrap();
@ -79,6 +400,7 @@ async fn apply_schema_rejects_when_non_main_branch_exists() {
}
#[tokio::test]
#[cfg_attr(feature = "failpoints", serial_test::parallel)]
async fn apply_schema_unsupported_plan_does_not_advance_manifest() {
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().to_str().unwrap();
@ -117,6 +439,7 @@ async fn apply_schema_unsupported_plan_does_not_advance_manifest() {
// contract so a regression in the planner can't silently change behavior.
#[tokio::test]
#[cfg_attr(feature = "failpoints", serial_test::parallel)]
async fn apply_schema_drops_a_nullable_property_softly_preserves_prior_version() {
let dir = tempfile::tempdir().unwrap();
let mut db = init_and_load(&dir).await;
@ -221,6 +544,7 @@ async fn apply_schema_drops_a_nullable_property_softly_preserves_prior_version()
}
#[tokio::test]
#[cfg_attr(feature = "failpoints", serial_test::parallel)]
async fn apply_schema_drops_node_and_referencing_edge_softly() {
let dir = tempfile::tempdir().unwrap();
let mut db = init_and_load(&dir).await;
@ -334,6 +658,7 @@ edge Knows: Person -> Person {
}
#[tokio::test]
#[cfg_attr(feature = "failpoints", serial_test::parallel)]
async fn apply_schema_drops_an_edge_type_softly() {
let dir = tempfile::tempdir().unwrap();
let mut db = init_and_load(&dir).await;
@ -391,6 +716,7 @@ async fn apply_schema_drops_an_edge_type_softly() {
}
#[tokio::test]
#[cfg_attr(feature = "failpoints", serial_test::parallel)]
async fn apply_schema_rejects_adding_a_required_property_without_backfill() {
let dir = tempfile::tempdir().unwrap();
let mut db = init_and_load(&dir).await;
@ -419,6 +745,7 @@ async fn apply_schema_rejects_adding_a_required_property_without_backfill() {
}
#[tokio::test]
#[cfg_attr(feature = "failpoints", serial_test::parallel)]
async fn plan_schema_for_property_type_narrowing_is_not_supported() {
// Symmetric companion to `apply_schema_unsupported_plan_does_not_advance_manifest`,
// which exercises widening (I32 -> I64). Narrowing (I64 -> I32) is also
@ -446,6 +773,7 @@ async fn plan_schema_for_property_type_narrowing_is_not_supported() {
}
#[tokio::test]
#[cfg_attr(feature = "failpoints", serial_test::parallel)]
async fn apply_schema_renames_node_type_via_rename_from_and_preserves_rows() {
// Covers the stable-type-id contract: renaming a type preserves the
// underlying Lance dataset (by stable id), so existing rows survive the
@ -535,6 +863,7 @@ edge WorksAt: Human -> Company
// persists. Full orphan-dataset deletion is a separate follow-up.
#[tokio::test]
#[cfg_attr(feature = "failpoints", serial_test::parallel)]
async fn apply_schema_with_allow_data_loss_promotes_drops_to_hard() {
let dir = tempfile::tempdir().unwrap();
let mut db = init_and_load(&dir).await;
@ -601,6 +930,7 @@ async fn apply_schema_with_allow_data_loss_promotes_drops_to_hard() {
}
#[tokio::test]
#[cfg_attr(feature = "failpoints", serial_test::parallel)]
async fn apply_schema_hard_drops_property_makes_prior_version_unreachable() {
let dir = tempfile::tempdir().unwrap();
let mut db = init_and_load(&dir).await;
@ -653,6 +983,7 @@ async fn apply_schema_hard_drops_property_makes_prior_version_unreachable() {
}
#[tokio::test]
#[cfg_attr(feature = "failpoints", serial_test::parallel)]
async fn apply_schema_hard_drops_node_and_edge_with_flag_succeeds() {
let dir = tempfile::tempdir().unwrap();
let mut db = init_and_load(&dir).await;
@ -737,19 +1068,14 @@ edge Knows: Person -> Person {
// (no manifest entry), which is the user-facing guarantee.
}
// Regression (bug 3 / dev-graph iss-848): a `Vector @index` on a 0-row table
// must not abort an otherwise-valid schema apply. A vector (IVF) index trains
// k-means centroids over the column's vectors, so Lance cannot build it on 0
// vectors — it errors with "Creating empty vector indices with train=False is
// not yet implemented". When a *later* migration touches that table (here, an
// unrelated scalar `@index` on `body`), schema apply reconciles the table's
// whole index set, which previously tried to materialize the dormant vector
// index and aborted the entire migration (all-or-nothing). The build is now
// deferred (pending) when the column is untrainable, instead of failing the
// migration. The dormant index is materialized by a later `ensure_indices` /
// `optimize` once the table has rows. Full decoupling — intent recorded at
// apply, an async reconciler converges physical coverage — is iss-848.
// Regression (bug 3 / dev-graph iss-848): schema apply records index intent but
// performs no physical index work. That decoupling is load-bearing for a
// `Vector @index` on a 0-row table: Lance cannot train IVF centroids on no
// vectors, yet the logical migration must still succeed. A later
// `ensure_indices` / `optimize` materializes every buildable declaration once
// data exists and reports an untrainable vector column as pending meanwhile.
#[tokio::test]
#[cfg_attr(feature = "failpoints", serial_test::parallel)]
async fn apply_schema_defers_vector_index_on_empty_table() {
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().to_str().unwrap();
@ -766,9 +1092,8 @@ async fn apply_schema_defers_vector_index_on_empty_table() {
}\n";
let mut db = Omnigraph::init(uri, v1).await.unwrap();
// Add an *unrelated* scalar @index on `body`. This routes Doc through
// schema apply's index reconcile, which must NOT abort on the untrainable
// empty vector index.
// Add an unrelated scalar @index on `body`. Schema apply must record both
// declarations without trying to build either one or train the empty vector.
let v2 = "node Doc {\n \
slug: String @key\n \
body: String? @index\n \
@ -779,11 +1104,8 @@ async fn apply_schema_defers_vector_index_on_empty_table() {
);
assert!(result.applied, "the scalar @index change must apply");
// The deferred vector index is not dropped — once the table has a
// trainable vector, `ensure_indices` materializes it without error. (If
// the guard wrongly skipped a non-empty column, this would still be
// unindexed; if it wrongly tried to build on empty, the apply above would
// have failed.)
// The deferred declarations are not dropped: after data arrives, the
// explicit reconciler materializes every buildable index without error.
load_jsonl(
&mut db,
r#"{"type":"Doc","data":{"slug":"d1","body":"hello","embedding":[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8]}}"#,
@ -803,6 +1125,7 @@ async fn apply_schema_defers_vector_index_on_empty_table() {
// optimize. Pre-iss-848 the indexed_tables block built the index inline and
// bumped the table version.
#[tokio::test]
#[cfg_attr(feature = "failpoints", serial_test::parallel)]
async fn index_only_constraint_apply_touches_no_table_data() {
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().to_str().unwrap();
@ -823,6 +1146,7 @@ async fn index_only_constraint_apply_touches_no_table_data() {
.entry("node:Doc")
.unwrap()
.table_version;
let before_commits = db.list_commits(None).await.unwrap();
// Add an @index on the existing `n` column.
let v2 = "node Doc {\n slug: String @key\n n: I64 @index\n}\n";
@ -840,6 +1164,12 @@ async fn index_only_constraint_apply_touches_no_table_data() {
before, after,
"adding an @index must not bump the table version (no inline index build)"
);
let after_commits = db.list_commits(None).await.unwrap();
assert_eq!(
after_commits.len(),
before_commits.len() + 1,
"metadata-only schema apply must still advance graph_head so it arbitrates concurrent prepared writes"
);
}
// Enum widening (iss-enum-widening-migration): adding variants to an enum is
@ -847,6 +1177,7 @@ async fn index_only_constraint_apply_touches_no_table_data() {
// touched, and the widened set is enforced immediately on writes. Narrowing
// stays OG-MF-106-refused.
#[tokio::test]
#[cfg_attr(feature = "failpoints", serial_test::parallel)]
async fn enum_widening_apply_is_metadata_only_and_accepts_new_variant() {
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().to_str().unwrap();
@ -867,6 +1198,7 @@ async fn enum_widening_apply_is_metadata_only_and_accepts_new_variant() {
.entry("node:Ticket")
.unwrap()
.table_version;
let before_commits = db.list_commits(None).await.unwrap();
let v2 =
"node Ticket {\n slug: String @key\n status: enum(todo, doing, done, blocked)\n}\n";
@ -885,6 +1217,12 @@ async fn enum_widening_apply_is_metadata_only_and_accepts_new_variant() {
before, after,
"enum widening must not bump the table version (metadata-only)"
);
let after_commits = db.list_commits(None).await.unwrap();
assert_eq!(
after_commits.len(),
before_commits.len() + 1,
"metadata-only enum widening must still advance graph_head"
);
// The NEW variant is accepted on the write path...
load_jsonl(
@ -914,6 +1252,7 @@ async fn enum_widening_apply_is_metadata_only_and_accepts_new_variant() {
}
#[tokio::test]
#[cfg_attr(feature = "failpoints", serial_test::parallel)]
async fn enum_narrowing_apply_is_refused() {
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().to_str().unwrap();

51
crates/omnigraph/tests/search.rs
View file

@ -86,6 +86,7 @@ async fn init_search_db(dir: &tempfile::TempDir) -> Omnigraph {
load_jsonl(&mut db, SEARCH_DATA, LoadMode::Overwrite)
.await
.unwrap();
db.ensure_indices().await.unwrap();
db
}
@ -95,6 +96,7 @@ async fn init_mock_embedding_search_db(dir: &tempfile::TempDir) -> Omnigraph {
load_jsonl(&mut db, &mock_embedding_seed_data(), LoadMode::Overwrite)
.await
.unwrap();
db.ensure_indices().await.unwrap();
db
}
@ -104,6 +106,7 @@ async fn init_model_recorded_search_db(dir: &tempfile::TempDir) -> Omnigraph {
load_jsonl(&mut db, &mock_embedding_seed_data(), LoadMode::Overwrite)
.await
.unwrap();
db.ensure_indices().await.unwrap();
db
}
@ -202,6 +205,40 @@ async fn doc_user_index_count(db: &Omnigraph) -> usize {
.count()
}
/// RFC-022 data writes publish only their exact table effects. Declared FTS
/// and vector indexes may therefore still be pending immediately after load;
/// both retrieval modes (and their RRF composition) must remain logically
/// correct through Lance's flat-search paths.
#[tokio::test]
#[serial]
async fn deferred_indexes_do_not_block_hybrid_reads() {
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().to_str().unwrap();
let mut db = Omnigraph::init(uri, MOCK_SEARCH_SCHEMA).await.unwrap();
load_jsonl(
&mut db,
&mock_embedding_seed_data(),
LoadMode::Overwrite,
)
.await
.unwrap();
assert_eq!(
doc_user_index_count(&db).await,
0,
"load must leave declared physical indexes to the reconciler"
);
let result = query_main(
&mut db,
MOCK_SEARCH_QUERIES,
"hybrid_search_vector",
&vector_and_string_params("$vq", &mock_embedding("alpha", 4), "$tq", "alpha"),
)
.await
.expect("pending FTS/vector indexes must degrade to flat search");
assert_eq!(result_slugs(&result)[0], "alpha-doc");
}
struct EnvGuard {
saved: Vec<(&'static str, Option<String>)>,
}
@ -853,7 +890,7 @@ async fn rrf_fuses_two_vector_queries() {
#[tokio::test]
#[serial]
async fn mutation_commit_refreshes_search_indices_without_manual_ensure() {
async fn mutation_with_deferred_index_coverage_remains_searchable() {
let dir = tempfile::tempdir().unwrap();
let mut db = init_search_db(&dir).await;
assert_eq!(doc_user_index_count(&db).await, 4);
@ -879,7 +916,7 @@ async fn mutation_commit_refreshes_search_indices_without_manual_ensure() {
assert_eq!(
doc_user_index_count(&db).await,
4,
"mutation commit should refresh required indices without duplicating them"
"mutation must leave physical index materialization to the reconciler"
);
let result = query_main(
@ -892,7 +929,7 @@ async fn mutation_commit_refreshes_search_indices_without_manual_ensure() {
.unwrap();
assert!(
result_slugs(&result).contains(&"quasar-notes".to_string()),
"newly inserted row should be searchable without an explicit ensure_indices step"
"a row outside current index coverage must remain searchable via fallback scan"
);
}
@ -919,7 +956,7 @@ async fn rrf_fuses_vector_and_text() {
#[tokio::test]
#[serial]
async fn load_commit_creates_vector_index_for_vector_annotations() {
async fn index_reconciler_creates_vector_index_for_vector_annotations() {
let schema = r#"
node Doc {
slug: String @key
@ -935,6 +972,12 @@ node Doc {
load_jsonl(&mut db, data, LoadMode::Overwrite)
.await
.unwrap();
assert_eq!(
doc_user_index_count(&db).await,
0,
"load publishes exact data effects and leaves physical indexes pending"
);
db.ensure_indices().await.unwrap();
let ds = snapshot_main(&db)
.await

86
crates/omnigraph/tests/staged_writes.rs
View file

@ -450,6 +450,56 @@ async fn stage_merge_insert_commit_rebases_over_disjoint_committed_delete() {
assert_eq!(collect_age_for_id(&batches, "p10"), None);
}
#[tokio::test]
async fn exact_commit_exposes_lance_preflight_rebase_version() {
let dir = tempfile::tempdir().unwrap();
let uri = format!("{}/people.lance", dir.path().to_str().unwrap());
let store = TableStore::new(dir.path().to_str().unwrap(), test_session());
let ds = TableStore::write_dataset(&uri, person_batch(&[("alice", Some(30))]))
.await
.unwrap();
let staged = store
.stage_append(&ds, person_batch(&[("bob", Some(25))]), &[])
.await
.unwrap();
let planned = staged.transaction_identity();
assert_eq!(planned.read_version, 1);
// A foreign append lands after staging. Even with max_retries(0), Lance
// performs one preflight conflict-resolution pass and can rebase this
// append before its sole manifest-write attempt. Lance preserves the
// transaction fields, so the exact path must expose the later achieved
// version as the second half of the identity check.
let mut foreign = ds.clone();
lance_append_inline_local(&mut foreign, person_batch(&[("carol", Some(40))])).await;
let (committed, observed) = store
.commit_staged_exact(Arc::new(ds), staged)
.await
.unwrap();
assert_eq!(committed.version().version, 3);
assert_eq!(observed, planned);
assert_ne!(
committed.version().version,
planned.read_version + 1,
"Lance preserves transaction identity during this preflight rebase; \
the achieved version is the second half of exact-effect identity"
);
// Without a concurrent advance, the same one-attempt path lands the exact
// staged transaction identity.
let next = store
.stage_append(&committed, person_batch(&[("dave", Some(50))]), &[])
.await
.unwrap();
let next_planned = next.transaction_identity();
let (_committed, next_observed) = store
.commit_staged_exact(Arc::new(committed), next)
.await
.unwrap();
assert_eq!(next_observed, next_planned);
}
/// **Documented limitation** (see `scan_with_staged` doc): when a filter
/// is supplied, Lance's stats-based pruning drops the staged fragment from
/// the filtered scan because uncommitted fragments produced by
@ -1075,10 +1125,9 @@ async fn lance_restore_appends_one_commit_with_checked_out_content() {
);
}
/// Empirical pin of the `Dataset::restore` concurrency hazard that
/// motivates the recovery sweep's open-time-only invocation strategy
/// and any future continuous-recovery reconciler's queue-acquisition
/// requirement.
/// Empirical pin of the `Dataset::restore` concurrency hazard that requires
/// Full recovery to join the root-scoped writer gates and leaves a real
/// cross-process fencing boundary.
///
/// `Dataset::restore`'s `check_restore_txn` (lance-6.0.1
/// `src/io/commit/conflict_resolver.rs:986`) returns `Ok(())` against
@ -1091,18 +1140,17 @@ async fn lance_restore_appends_one_commit_with_checked_out_content() {
/// rewind commit AFTER the legitimate concurrent Append, silently
/// orphaning that Append's data from the active timeline.
///
/// The recovery sweep sidesteps this by running only at `Omnigraph::open`
/// (before any other writers can race). A future continuous-recovery
/// reconciler must acquire per-(table_key, branch) queues for sidecar
/// tables before invoking restore — otherwise this hazard becomes
/// reachable during in-flight tenant traffic.
/// Full recovery may invoke Restore on a read-write open, but first joins the
/// same root-scoped schema → branch → sorted-table gates as live writers and
/// rechecks that the sidecar still exists after waiting. The in-process healer
/// is roll-forward-only and never restores beneath live traffic. Together those
/// rules keep this substrate asymmetry unreachable against another handle in
/// the same process.
///
/// MR-686 introduces those per-(table_key, branch) writer queues as the
/// application-layer mechanism that closes this hazard once continuous
/// in-process recovery (MR-870) lands. Until MR-686's queue is wired into
/// the recovery path, the open-time-only invocation strategy is the
/// only thing keeping this hazard out of production. See
/// `docs/invariants.md` §VI.30, §VI.32, §VI.33.
/// The gates remain process-local. A recovery pass cannot fence a live writer
/// in another process, so general multi-process write/recovery topologies remain
/// outside the supported boundary until a distributed fence exists. See
/// `docs/dev/invariants.md` and `docs/dev/writes.md`.
///
/// This test is the load-bearing constraint any future reconciler must
/// honor.
@ -1159,10 +1207,10 @@ async fn lance_restore_loses_to_concurrent_append_via_orphaning() {
ids,
vec!["alice".to_string()],
"Concurrent Append's row 'bob' was silently orphaned by the \
Restore. Active-timeline contents == v1's contents. The recovery \
sweep sidesteps this hazard via open-time-only invocation; any \
future continuous-recovery reconciler must guard against it via \
per-(table, branch) queue acquisition. Got: {:?}",
Restore. Active-timeline contents == v1's contents. Full recovery \
must join the root-scoped writer gates before Restore; those gates \
remain process-local, so a distributed fence is required before \
multi-process recovery can be supported. Got: {:?}",
ids,
);

6
crates/omnigraph/tests/traversal_indexed.rs
View file

@ -58,7 +58,8 @@ async fn key_column_index_coverage_detects_btree_presence() {
let db = init_and_load(&dir).await;
let snap = snapshot_main(&db).await.unwrap();
// Edge `src` gets a BTREE from ensure_indices on load → Indexed.
// The shared fixture explicitly reconciles indexes after loading, so the
// edge `src` BTREE is present and fully covered here.
let edge_ds = snap.open("edge:Knows").await.unwrap();
let src_cov = TableStore::key_column_index_coverage(&edge_ds, "src")
.await
@ -86,7 +87,8 @@ async fn coverage_degrades_for_appended_unindexed_fragment() {
let dir = tempfile::tempdir().unwrap();
let mut db = init_and_load(&dir).await;
// Fresh load: the Knows BTREE covers every fragment → Indexed.
// The fixture's explicit post-load `ensure_indices` covers every current
// Knows fragment → Indexed.
let snap = snapshot_main(&db).await.unwrap();
let edge_ds = snap.open("edge:Knows").await.unwrap();
assert_eq!(

37
crates/omnigraph/tests/write_cost.rs
View file

@ -5,8 +5,9 @@
//! boundary. Guards invariant 15 (cost bounded by work, not history) on writes.
//!
//! **Backend split (see docs/dev/testing.md / RFC-013).** This file runs on
//! **local FS** and gates the **internal-table** term (`__manifest`/`_graph_commits`
//! fragment scans, ~+18/depth — O(fragments) on any backend, step 2's target).
//! **local FS** and gates the **internal-table** term (`__manifest` fragment
//! scans, including inline lineage/actor rows — O(fragments) on any backend,
//! step 2's target). The former standalone lineage tables are retired.
//!
//! The **data-table opener** term (step 3a's win) is a per-object-store-RPC
//! phenomenon and is NOT gated here: local-FS latest-resolution is cheap whether
@ -270,14 +271,16 @@ async fn keyed_insert_routes_through_merge_insert_only() {
// ── (D) Step-3b capture-once fitness asserts (RED today → GREEN after WriteTxn) ──
/// A write must validate the schema contract EXACTLY ONCE (3 `read_text` + 2 `exists`).
/// Today the write path re-validates at every resolve point (entry, per-table
/// `resolved_branch_target`, commit-time `fresh_snapshot_for_branch`), so the delta is
/// a multiple of that. Step 3b's `WriteTxn` validates once and threads it. The shape is
/// A write performs one full schema validation while capturing the catalog-bound
/// `WriteTxn`, one trailing state-marker read that fences torn head/schema capture,
/// and one full validation under the pre-effect gates (7 `read_text` + 4 `exists`
/// total). Per-table resolves must not add more validation. The gate read is
/// correctness work: it arbitrates schema identity after preparation and before
/// the recovery sidecar or any Lance HEAD movement. The shape is
/// the write twin of `warm_read_cost.rs::warm_query_validates_schema_contract_once`,
/// built with ZERO production change via the counting storage adapter.
#[tokio::test]
async fn write_validates_schema_contract_once() {
async fn write_schema_io_is_bounded_to_capture_fence_and_effect_gate() {
use omnigraph::instrumentation::CountingStorageAdapter;
use omnigraph::storage::storage_for_uri;
@ -291,6 +294,8 @@ async fn write_validates_schema_contract_once() {
let before_read_text = counts.read_text();
let before_exists = counts.exists();
let before_write_text = counts.write_text();
let before_delete = counts.delete();
db.mutate(
"main",
MUTATION_QUERIES,
@ -302,14 +307,24 @@ async fn write_validates_schema_contract_once() {
let read_text_delta = counts.read_text() - before_read_text;
let exists_delta = counts.exists() - before_exists;
let write_text_delta = counts.write_text() - before_write_text;
let delete_delta = counts.delete() - before_delete;
eprintln!("schema-contract reads on one write: read_text={read_text_delta} exists={exists_delta}");
assert_eq!(
read_text_delta, 3,
"a write must validate the schema contract once (3 reads), not N times",
read_text_delta, 7,
"a write must do capture validation + trailing identity fence + pre-effect validation (7 reads), not per table",
);
assert_eq!(
exists_delta, 2,
"a write must probe contract-file existence once (2 probes), not N times",
exists_delta, 4,
"a write must probe contract-file existence at capture + pre-effect revalidation (4 probes)",
);
assert_eq!(
write_text_delta, 2,
"an enrolled write must write its recovery sidecar exactly twice (arm + exact confirmation)",
);
assert_eq!(
delete_delta, 1,
"a successful enrolled write must delete its confirmed sidecar once",
);
}

157
crates/omnigraph/tests/writes.rs
View file

@ -1,13 +1,17 @@
//! Tests for the direct-publish write path: mutations and loads write
//! directly to target tables and commit once via the publisher's
//! `expected_table_versions` CAS. (History: this replaced the removed Run
//! Tests for the direct-publish write path. Mutations and loads capture one
//! branch-wide authority token, prepare exact per-table transactions, then
//! acquire the root-shared schema → branch → sorted-table gates, revalidate,
//! arm schema-v3 recovery, commit the table effects, and publish once under the
//! same exact-head/table precondition. (History: this replaced the removed Run
//! state machine / `__run__` staging branches / RunRecord — MR-771.)
//!
//! What this file covers:
//! - No `__run__*` branches are created by load or mutate.
//! - Cancellation of a mutation future leaves no graph-level state.
//! - Concurrent non-strict inserts/merges rebase under the per-table queue;
//! strict updates/deletes surface `ExpectedVersionMismatch` on stale state.
//! - A pre-effect branch-authority change makes an insert-only mutation or
//! Append/Merge load discard and fully reprepare with a bounded retry; strict
//! Update/Delete/Overwrite surfaces `ReadSetChanged`. Post-effect failures
//! require recovery.
//! - Failed mutations and loads leave the target unchanged.
//! - Multi-statement mutations are atomic (one commit per query).
//! - actor_id propagates through to the commit graph.
@ -242,11 +246,12 @@ async fn partial_failure_leaves_target_queryable_and_unblocks_next_mutation() {
assert_eq!(frank.num_rows(), 1, "Frank must be visible after publish");
}
/// Stale non-strict writers rebase to the live manifest pin under the
/// per-table queue instead of folding raw drift or returning a false 409.
/// Strict update/delete semantics are covered by the consistency/server tests.
/// Stale non-strict writers discard and reprepare their whole logical attempt
/// from the live branch authority instead of rebasing an already-validated
/// staged transaction or returning a false 409. Strict update/delete semantics
/// are covered by the consistency/server tests.
#[tokio::test]
async fn stale_non_strict_insert_rebases_to_live_manifest_pin() {
async fn stale_non_strict_insert_reprepares_from_live_branch_state() {
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().to_string_lossy().into_owned();
@ -276,9 +281,10 @@ async fn stale_non_strict_insert_rebases_to_live_manifest_pin() {
}
// Writer B's coordinator is still at the pre-A snapshot, but Insert is
// non-strict: commit_all re-reads the live manifest pin under the queue,
// verifies Lance HEAD equals that pin, and then lets Lance rebase the
// staged append.
// retryable: the RFC-022 adapter notices the authority change under the
// branch gate, discards B's prepared/staged attempt, and reruns the full
// operation from A's committed state. Lance never rebases a plan whose
// validation inputs are stale.
db_b.mutate(
"main",
MUTATION_QUERIES,
@ -598,8 +604,16 @@ async fn overlapping_delete_predicates_do_not_double_count_affected() {
);
// The data is correct regardless of the count: Bob + Diana remain.
assert_eq!(count_rows(&db, "node:Person").await, 2, "Bob and Diana remain");
assert_eq!(count_rows(&db, "edge:Knows").await, 1, "only Bob→Diana remains");
assert_eq!(
count_rows(&db, "node:Person").await,
2,
"Bob and Diana remain"
);
assert_eq!(
count_rows(&db, "edge:Knows").await,
1,
"only Bob→Diana remains"
);
assert_eq!(
count_rows(&db, "edge:WorksAt").await,
1,
@ -633,7 +647,9 @@ edge Knows: Person -> Person
{"type":"Person","data":{"name":"Zoe"}}
{"edge":"Knows","from":"Zoe","to":"Charlie"}"#;
let mut db = Omnigraph::init(uri, schema).await.unwrap();
load_jsonl(&mut db, data, LoadMode::Overwrite).await.unwrap();
load_jsonl(&mut db, data, LoadMode::Overwrite)
.await
.unwrap();
let q = r#"
query del_age_then_name($threshold: I32, $name: String) {
@ -668,7 +684,7 @@ query del_age_then_name($threshold: I32, $name: String) {
/// `insert Person 'X'; update Person where name='X' set age=...` — both
/// ops produce content on `node:Person` and coalesce into one
/// `stage_merge_insert` at end-of-query. The accumulator's last-write-wins
/// dedupe (in `MutationStaging::finalize`) ensures the update's value
/// dedupe (during `MutationStaging::stage_all`) ensures the update's value
/// wins. Single Lance commit per table per query.
#[tokio::test]
async fn mixed_insert_and_update_on_same_person_coalesces_to_one_merge() {
@ -692,7 +708,7 @@ async fn mixed_insert_and_update_on_same_person_coalesces_to_one_merge() {
assert_eq!(result.affected_nodes, 2, "1 insert + 1 update reported");
// The end-state row carries the update value (last-write-wins via
// dedupe in finalize), proving the staged merge_insert ran with the
// end-of-query dedupe), proving the staged merge_insert ran with the
// correct source dedupe. Read the underlying Person table directly
// and assert age=99 for the row we just inserted+updated.
let batches = read_table(&db, "node:Person").await;
@ -1063,7 +1079,7 @@ edge WorksAt: Person -> Company @card(0..1)
/// `scan_with_pending`, the second update sees the stale committed value
/// (the first update's row still appears in the Lance scan because the
/// pending side hasn't committed), the predicate matches it, and the
/// dedupe-last-wins step at finalize ends up applying the second update
/// end-of-query dedupe-last-wins step ends up applying the second update
/// to a row whose pending value should have shielded it.
///
/// Concretely: Alice starts at age=30 in TEST_DATA. Op-1 sets Alice to
@ -1347,7 +1363,7 @@ edge WorksAt: Person -> Company @card(0..1)
}
/// A Merge load whose input has TWO rows with the same edge id must be
/// deduped at cardinality-count time, not just at finalize. Without
/// deduped at cardinality-count time, not just during end-of-query staging. Without
/// dedup, two pending rows count twice → spurious `@card` violation.
/// With dedup (last-occurrence-wins, mirroring
/// `dedupe_merge_batches_by_id`), the pending side counts once.
@ -1382,7 +1398,7 @@ edge WorksAt: Person -> Company @card(0..1)
.unwrap();
// Merge load with the SAME edge id twice — the second row supersedes
// the first in the finalize-time dedupe. If pending-counting doesn't
// the first in the end-of-query dedupe. If pending-counting doesn't
// dedupe, Alice has 2 pending edges → @card(0..1) trips → load
// fails. With dedupe, Alice has 1 → load succeeds.
let dup_data = r#"{"edge": "WorksAt", "from": "Alice", "to": "Acme", "data": {"id": "w1"}}
@ -1490,21 +1506,13 @@ async fn scan_with_pending_rejects_key_column_missing_from_projection() {
);
}
/// `PendingTable.schema` is captured from the first `append_batch` call
/// and never updated. On a blob-bearing table, an `insert` produces a
/// full-schema batch (blob columns included) and an `update` that
/// doesn't assign every blob produces a subset-schema batch. Mixed in
/// one query, the second `append_batch` would silently push an
/// incompatible batch — the mismatch surfaced eventually at
/// `concat_batches`/MemTable construction inside finalize, but the
/// failure point was distant from the offending op.
///
/// `append_batch` validates the new batch's schema against the existing
/// accumulator's schema and returns a typed error directing the caller
/// to split the mutation. The error fires at the second op (the
/// update), not at end-of-query.
/// A blob-table insert followed by a non-blob update must remain one
/// full-schema merge stream. The update reads the just-inserted pending row,
/// copies its logical blob column through, and replaces the pending row under
/// the existing last-write-wins shadow semantics. This used to produce a
/// partial update batch and fail schema validation at the second op.
#[tokio::test]
async fn append_batch_rejects_mismatched_schema_in_blob_table_at_offending_op() {
async fn blob_table_insert_then_non_blob_update_preserves_full_schema() {
use omnigraph::loader::{LoadMode, load_jsonl};
const BLOB_SCHEMA: &str = r#"
@ -1527,10 +1535,9 @@ query insert_then_update_note(
let uri = dir.path().to_str().unwrap();
let mut db = Omnigraph::init(uri, BLOB_SCHEMA).await.unwrap();
// Seed with a Document so the update has something to match (the
// mid-query case is the chained-update scenario where the update's
// predicate matches the just-inserted row, exercising the in-memory
// pending union).
// Keep one committed blob row as well as the just-inserted pending row so
// the table has the real blob-v2 physical representation while this query
// exercises the pending union.
load_jsonl(
&mut db,
r#"{"type":"Document","data":{"title":"seed","content":"base64:AQID"}}"#,
@ -1539,7 +1546,7 @@ query insert_then_update_note(
.await
.unwrap();
let err = db
let result = db
.mutate(
"main",
BLOB_QUERIES,
@ -1551,36 +1558,35 @@ query insert_then_update_note(
]),
)
.await
.expect_err("blob-table mixed insert+update with non-fully-assigned blob must error early");
let OmniError::Manifest(manifest_err) = err else {
panic!("expected Manifest error, got {err:?}");
};
assert!(
manifest_err.message.contains("mismatched schemas")
&& manifest_err.message.contains("Split the mutation"),
"error must direct user to split: {}",
manifest_err.message,
.expect("blob-table insert + non-blob update must share a full-schema merge batch");
assert_eq!(
result.affected_nodes, 2,
"one insert plus one pending-row update"
);
// Confirm the manifest didn't advance — early error must be
// before any commit.
let blob = db
.read_blob("Document", "letter", "content")
.await
.expect("inserted blob must remain readable after the update");
assert_eq!(&blob.read().await.unwrap()[..], &[4, 5, 6]);
let qr = db
.query(
ReadTarget::branch("main"),
r#"query get_doc($title: String) {
match { $d: Document { title: $title } }
return { $d.title }
return { $d.title, $d.note }
}"#,
"get_doc",
&params(&[("$title", "letter")]),
)
.await
.unwrap();
assert_eq!(
qr.num_rows(),
0,
"letter must not be visible after early error"
);
assert_eq!(qr.num_rows(), 1);
let json = qr.to_sdk_json();
let row = json.as_array().unwrap().first().unwrap();
assert_eq!(row["d.title"], "letter");
assert_eq!(row["d.note"], "draft 1");
}
/// MR-920 regression: two sequential `update T set {f:v} where x=y`
@ -1794,7 +1800,9 @@ async fn camelcase_mutation_predicate_updates_and_deletes() {
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().to_str().unwrap();
let mut db = Omnigraph::init(uri, CC_SCHEMA).await.unwrap();
load_jsonl(&mut db, CC_DATA, LoadMode::Overwrite).await.unwrap();
load_jsonl(&mut db, CC_DATA, LoadMode::Overwrite)
.await
.unwrap();
let m = r#"
query set_status($repo: String, $st: String) { update Doc set { status: $st } where repoName = $repo }
@ -1802,7 +1810,12 @@ query del($repo: String) { delete Doc where repoName = $repo }
"#;
let upd = db
.mutate("main", m, "set_status", &params(&[("$repo", "acme"), ("$st", "closed")]))
.mutate(
"main",
m,
"set_status",
&params(&[("$repo", "acme"), ("$st", "closed")]),
)
.await
.expect("update with a camelCase predicate must execute");
assert_eq!(upd.affected_nodes, 1, "exactly the acme Doc should update");
@ -1811,9 +1824,16 @@ query del($repo: String) { delete Doc where repoName = $repo }
.mutate("main", m, "del", &params(&[("$repo", "globex")]))
.await
.expect("delete with a camelCase predicate must execute");
assert_eq!(del.affected_nodes, 1, "exactly the globex Doc should delete");
assert_eq!(
del.affected_nodes, 1,
"exactly the globex Doc should delete"
);
assert_eq!(count_rows(&db, "node:Doc").await, 1, "one Doc (acme) should remain");
assert_eq!(
count_rows(&db, "node:Doc").await,
1,
"one Doc (acme) should remain"
);
}
// #283 (pending side): a chained mutation whose 2nd op filters a camelCase
@ -1825,7 +1845,9 @@ async fn camelcase_chained_mutation_reads_pending_by_camelcase() {
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().to_str().unwrap();
let mut db = Omnigraph::init(uri, CC_SCHEMA).await.unwrap();
load_jsonl(&mut db, CC_DATA, LoadMode::Overwrite).await.unwrap();
load_jsonl(&mut db, CC_DATA, LoadMode::Overwrite)
.await
.unwrap();
// op-1 stages a status change to the acme Doc; op-2 re-filters the same
// camelCase column, so it must match op-1's pending row.
@ -1838,8 +1860,13 @@ query chain($repo: String) {
let r = db
.mutate("main", m, "chain", &params(&[("$repo", "acme")]))
.await
.expect("chained camelCase mutation must read the pending row, not fail at the MemTable SELECT");
assert_eq!(r.affected_nodes, 2, "both ops should touch the acme Doc (read-your-writes)");
.expect(
"chained camelCase mutation must read the pending row, not fail at the MemTable SELECT",
);
assert_eq!(
r.affected_nodes, 2,
"both ops should touch the acme Doc (read-your-writes)"
);
}
/// A zero-row cascade delete must not advance an edge table's Lance HEAD past
@ -1992,7 +2019,9 @@ async fn filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent()
)
})
.collect();
load_jsonl(&mut db, &updates, LoadMode::Merge).await.unwrap();
load_jsonl(&mut db, &updates, LoadMode::Merge)
.await
.unwrap();
// The delete adds a deletion vector, so the overlapping region no longer
// densely tiles its id range — the shape lance#7444 choked on.

42
docs/dev/architecture.md
View file

@ -170,30 +170,36 @@ Code paths:
- Graph index: `crates/omnigraph/src/graph_index/`
- Loader: `Omnigraph::ingest` at `crates/omnigraph/src/loader/mod.rs:74`
### Mutation atomicity — in-memory accumulator (MR-794)
### Mutation atomicity — in-memory accumulator + branch-wide OCC (MR-794 / RFC-022)
Inserts and updates inside `mutate_as` and the bulk loader's
Append/Merge modes go through `MutationStaging`
Mutations inside `mutate_as` and every bulk-load mode go through `MutationStaging`
([`crates/omnigraph/src/exec/staging.rs`](../../crates/omnigraph/src/exec/staging.rs)),
a per-query in-memory accumulator. No Lance HEAD advance happens during
op execution; one `stage_*` + `commit_staged` per touched table runs
at end-of-query, then the publisher commits the manifest atomically.
op execution. The attempt captures one native branch identity, exact graph head,
accepted schema identity/catalog, and base snapshot. At end-of-query it stages
one exact Lance transaction per touched table outside the effect gates, then
acquires the root-shared schema → branch → sorted-table gates and revalidates the
complete authority before arming recovery or advancing any table HEAD.
```
op-1 (insert/update) → push RecordBatch → MutationStaging.pending[table]
op-2 (insert/update) → read committed via Lance + pending via DataFusion
MemTable (read-your-writes) → push batch
op-N → push batch
─── end of query ───────────────────────────────────────
finalize: per pending table:
concat batches → stage_append OR stage_merge_insert OR stage_overwrite
→ commit_staged
publisher: ManifestBatchPublisher::publish (one cross-table CAS)
─── end of query ────────────────────────────────────────────────
stage_all: concat batches → one exact stage_* transaction per table (no HEAD move)
commit_all: acquire schema → branch → sorted-table gates
→ reject a new recovery intent → revalidate the complete branch token
→ arm schema-v3 recovery → commit_staged per table
publisher: publish pre-minted lineage under the exact native-branch/head + table precondition
```
A failed op leaves Lance HEAD untouched on the staged tables: the next
mutation proceeds normally with no drift to reconcile. Concrete
contracts:
A failed op leaves Lance HEAD untouched on the staged tables. If authority
changes before effects, insert-only mutations and Append/Merge loads discard the
whole attempt and fully reprepare with a bounded retry; Update/Delete/Overwrite
returns `ReadSetChanged`. After any physical effect, any later error returns
`RecoveryRequired` and leaves the v3 sidecar to converge the fixed outcome.
Concrete contracts:
- `D₂` parse-time rule: a query is either insert/update-only or
delete-only. Mixed → reject. Deletes now stage like inserts/updates
@ -262,7 +268,7 @@ flowchart LR
rec --> diff --> wp
```
Today, indexes are built explicitly via `ensure_indices`. Reads degrade gracefully when index coverage is partial — Lance's scanner unions indexed and scan paths automatically. The roadmap reconciler observes manifest state and converges coverage in the background.
Today, physical indexes are built explicitly via `ensure_indices`/`optimize`; schema apply, mutation, and load only record intent or publish their exact data effects and never build indexes inline. Reads degrade gracefully when index coverage is missing or partial — Lance's scanner unions indexed and scan paths automatically (vector search falls back to brute force). A future background reconciler may automate the same explicit convergence path.
### Server / CLI
@ -276,14 +282,14 @@ flowchart LR
pol[Cedar policy gate<br/>per request]:::l2
wl[WorkloadController<br/>per-actor admission]:::l2
eng[engine API<br/>Arc&lt;Omnigraph&gt;]:::l2
wq[WriteQueueManager<br/>per-(table, branch)]:::l2
wq[WriteQueueManager<br/>root-scoped schema → branch → table gates]:::l2
cli -.-> eng
srv_in --> auth --> pol --> wl --> eng
eng --> wq
```
The server applies Cedar policy at the HTTP boundary today. The roadmap, called out in [docs/dev/invariants.md](invariants.md) as a known gap, is to push policy into the planner as predicates. After Cedar, mutating handlers go through `WorkloadController` (per-actor admission cap + byte budget; PR 2 / MR-686) before reaching the engine. The engine itself holds an `Arc<WriteQueueManager>` so concurrent mutations on the same `(table, branch)` serialize at the queue, while disjoint keys run in parallel — see [docs/user/server.md](../user/operations/server.md) "Per-actor admission control" and [docs/dev/writes.md](writes.md). The CLI bypasses the HTTP layer (and admission) and calls the engine API directly.
The server applies Cedar policy at the HTTP boundary today. The roadmap, called out in [docs/dev/invariants.md](invariants.md) as a known gap, is to push policy into the planner as predicates. After Cedar, mutating handlers go through `WorkloadController` (per-actor admission cap + byte budget; PR 2 / MR-686) before reaching the engine. Every handle for one canonical graph root shares an `Arc<WriteQueueManager>` in-process. RFC-022-enrolled mutation/load attempts prepare outside the effect gates, then acquire the exclusive root schema gate, the target branch-effect gate, and sorted `(table, branch)` gates and hold them through publish. The root schema gate means enrolled effect windows on one graph currently serialize even across different branches; readers remain ungated, and pre-effect parsing, validation, and fragment staging can overlap. The branch gate preserves the complete per-branch validation authority, while table gates protect each concrete Lance effect and legacy adapter. These gates prevent same-process races and impose one deadlock-free order; the exact publisher precondition and durable recovery intent remain the correctness authorities, and the gates are not a cross-process lock. See [docs/user/server.md](../user/operations/server.md) "Per-actor admission control" and [docs/dev/writes.md](writes.md). The CLI bypasses the HTTP layer (and admission) and calls the engine API directly.
Code paths:
@ -303,8 +309,8 @@ Throughout the docs, capabilities are split into:
- **MVCC**: every Lance write bumps a per-dataset version; the OmniGraph manifest version coordinates which sub-table versions are visible together.
- **Snapshot isolation**: a query holds one `Snapshot` for its lifetime; concurrent writes don't leak in.
- **Cross-branch isolation**: copy-on-write means readers and writers on different branches don't block each other.
- **Per-query staging**: `mutate_as` and `load` (Append/Merge) accumulate insert/update batches in an in-memory `MutationStaging`; one `stage_*` + `commit_staged` per touched table runs at end-of-query, then the publisher commits the manifest atomically. A mid-query failure leaves Lance HEAD untouched on staged tables. (MR-794; pre-v0.4.0 used a `__run__<id>` staging branch + Run state machine, removed in MR-771.)
- **Cross-branch isolation**: Lance copy-on-write keeps branch data independent, and readers never acquire write gates. Enrolled writer preparation can overlap across branches, but their effect/publish windows currently share one exclusive root schema gate before the branch/table gates, so those windows serialize in-process even for different branches.
- **Per-query staging**: `mutate_as` and every `load` mode accumulate their work in an in-memory `MutationStaging`; `stage_all` prepares exact per-table transactions without moving HEAD. At commit, the root schema → branch → sorted-table gates protect complete-token revalidation, v3 recovery arming, zero-retry table effects, and one atomic manifest publish. A mid-query failure leaves Lance HEAD untouched on staged tables. (MR-794 / RFC-022; pre-v0.4.0 used a `__run__<id>` staging branch + Run state machine, removed in MR-771.)
- **Schema-apply lock**: `__schema_apply_lock__` system branch serializes schema migrations.
- **Fail-points** (`failpoints` cargo feature): `failpoints::maybe_fail("operation.step")?` in `branch_create`, publish, etc., for deterministic failure injection in tests.

75
docs/dev/execution.md
View file

@ -79,16 +79,16 @@ Hybrid example: `order { rrf(nearest($d.embedding, $q), bm25($d.body, $q_text))
## Mutation execution (`exec/mutation.rs`)
Resolves expression values to literals, converts to typed Arrow arrays (`literal_to_typed_array(lit, DataType, num_rows)`), then writes via Lance's two-phase distributed-write API at end-of-query:
Resolves expression values to literals, converts to typed Arrow arrays (`literal_to_typed_array(lit, DataType, num_rows)`), then writes via Lance's two-phase distributed-write API at end-of-query. Before lowering/execution, one `WriteTxn` captures the target's Lance-native branch identity, exact optional graph head, accepted schema identity/catalog, and base table snapshot; every step in the attempt uses that immutable authority.
- `insert` (no `@key`, edges) → accumulate into `MutationStaging.pending` (Append mode); finalize calls `stage_append` once per touched table.
- `insert` (`@key` node) → accumulate into `pending` (Merge mode); finalize calls `stage_merge_insert` once per touched table.
- `insert` (no `@key`, edges) → accumulate into `MutationStaging.pending` (Append mode); `stage_all` later calls `stage_append` once per touched table.
- `insert` (`@key` node) → accumulate into `pending` (Merge mode); `stage_all` later calls `stage_merge_insert` once per touched table.
- `update` → scan committed via Lance + pending via DataFusion `MemTable` (read-your-writes), apply assignments, accumulate into `pending` (Merge mode).
- `delete` → records a predicate into `MutationStaging.delete_predicates` (count matching committed rows now for `affected_*`); finalize combines a table's predicates into one `stage_delete` (Lance 7.0 `DeleteBuilder::execute_uncommitted`, a deletion-vector transaction) committed via `commit_staged` — no inline HEAD advance (MR-A).
- `delete` → records a predicate into `MutationStaging.delete_predicates` (count matching committed rows now for `affected_*`); `stage_all` combines a table's predicates into one `stage_delete` (Lance 7.0 `DeleteBuilder::execute_uncommitted`, a deletion-vector transaction) — no inline HEAD advance (MR-A).
**D₂ parse-time rule.** A single mutation query is either insert/update-only or delete-only. Mixed → reject before any I/O. The check fires in `enforce_no_mixed_destructive_constructive(&ir)` inside `execute_named_mutation`.
Multi-statement mutations are atomic at the publisher commit boundary: every insert/update batch lives in memory until end-of-query, then exactly one `stage_*` + `commit_staged` runs per touched table, then `ManifestBatchPublisher::publish` commits the manifest atomically with per-table `expected_table_versions` CAS.
Multi-statement mutations are atomic at the publisher commit boundary. Every batch lives in memory until all statements and validation succeed; `stage_all` then prepares one exact transaction per touched table without advancing HEAD. `commit_all` acquires the root-shared schema → branch → sorted-table gates, rechecks for recovery intent, revalidates the complete branch authority, writes the schema-v3 recovery sidecar, and commits the table transactions with zero transparent conflict retries. The guards remain held while `ManifestBatchPublisher` publishes the pre-minted lineage under the same exact native-branch/head and table-version precondition.
### Mutation flow — sequence
@ -100,11 +100,12 @@ sequenceDiagram
participant cmp as omnigraph-compiler
participant stg as MutationStaging<br/>(exec/staging.rs)
participant ts as table_store
participant lance as Lance dataset
participant rec as schema-v3 recovery sidecar
participant pub as ManifestBatchPublisher
client->>og: mutate_as(branch, source, name, params, actor_id)
og->>cmp: parse + typecheck + lower_mutation_query
og->>og: heal/reject recovery intent; open_write_txn
og->>cmp: parse + typecheck + lower using txn catalog
cmp-->>og: MutationIR
og->>og: enforce_no_mixed_destructive_constructive (D₂)
loop for each mutation op
@ -116,23 +117,43 @@ sequenceDiagram
og->>ts: scan_with_pending (Lance + DataFusion MemTable union)
ts-->>og: matched batches
end
else delete (stage, D₂ keeps separate)
else delete (record predicate; D₂ keeps separate)
og->>ts: count_rows (committed match → affected_*)
og->>stg: ensure_path + record_delete (predicate)
end
end
og->>stg: finalize(db, branch)
loop per pending table
stg->>ts: stage_append OR stage_merge_insert (one per table)
ts-->>stg: StagedWrite (transaction + commit metadata + fragments)
stg->>ts: commit_staged (advances Lance HEAD)
ts-->>stg: new Dataset
og->>og: validate complete staged change-set against txn base
og->>stg: stage_all(db, branch)
loop per touched table
stg->>ts: stage_append OR stage_merge_insert OR stage_delete (one per table)
ts-->>stg: exact staged transaction (no HEAD movement)
end
stg->>stg: acquire schema → branch → sorted-table gates
stg->>og: recheck recovery barrier + revalidate complete WriteTxn
alt authority changed before effects
stg-->>og: ReadSetChanged
alt insert-only mutation
og->>og: discard complete attempt; bounded full reprepare
else Update/Delete
og-->>client: ReadSetChanged (409)
end
else authority unchanged
stg->>rec: persist fixed lineage + exact transaction identities
loop per touched table
stg->>ts: commit_staged (zero transparent retries)
ts-->>stg: confirm achieved transaction + table update
end
stg-->>og: updates + expected versions + sidecar + held gates
og->>pub: publish exact graph-head/table precondition
alt publish succeeds
pub-->>og: new manifest version
og->>rec: delete sidecar
og-->>client: MutationResult
else any error after an effect
pub-->>og: error
og-->>client: RecoveryRequired (sidecar remains authoritative)
end
end
stg-->>og: (updates: Vec<SubTableUpdate>, expected_versions)
og->>pub: commit_updates_on_branch_with_expected
pub->>pub: publisher CAS (cross-table OCC on __manifest)
pub-->>og: new manifest version
og-->>client: MutationResult
```
**Code paths:**
@ -143,11 +164,11 @@ sequenceDiagram
- Per-op execution: `execute_insert`, `execute_update`, `execute_delete_node`, `execute_delete_edge`
- Pending-aware reads: `TableStore::scan_with_pending` / `count_rows_with_pending` at `crates/omnigraph/src/table_store.rs`
- Edge cardinality with pending: `validate_edge_cardinality_with_pending` at `crates/omnigraph/src/exec/mutation.rs`
- Per-query accumulator: `crates/omnigraph/src/exec/staging.rs` (`MutationStaging`, `PendingTable`, `PendingMode`, `finalize`)
- End-of-query Lance commit: `TableStore::stage_append`, `stage_merge_insert`, `commit_staged` at `crates/omnigraph/src/table_store.rs`
- Manifest commit primitive: `commit_updates_on_branch_with_expected` at `crates/omnigraph/src/db/omnigraph/table_ops.rs`
- Per-query accumulator and protocol adapter: `crates/omnigraph/src/exec/staging.rs` (`MutationStaging::stage_all`, `StagedMutation::commit_all`)
- End-of-query Lance operations: `TableStore::stage_append`, `stage_merge_insert`, `stage_delete`, `commit_staged` at `crates/omnigraph/src/table_store.rs`
- Manifest commit primitive: `commit_updates_on_branch_with_expected` at `crates/omnigraph/src/db/omnigraph/table_ops.rs` (exact native-branch/head precondition plus expected table versions)
Atomicity guarantee for multi-statement mutations: a mid-query failure leaves Lance HEAD untouched on staged tables (no inline commit happened during op execution), so the next mutation proceeds normally with no `ExpectedVersionMismatch`. The publisher CAS at the very end either succeeds (manifest advances atomically across all touched sub-tables) or fails with a typed `ManifestConflictDetails::ExpectedVersionMismatch` (no partial publish). See [docs/dev/invariants.md](invariants.md) and [docs/dev/writes.md](writes.md).
Atomicity guarantee for multi-statement mutations: a mid-query failure leaves Lance HEAD untouched because no effect occurs during statement execution or staging. A pre-effect authority mismatch discards the complete attempt: insert-only mutations fully reprepare with a bounded retry, while Update/Delete returns typed `ReadSetChanged`. Once any `commit_staged` effect is durable, the attempt either makes every update visible together or returns `RecoveryRequired` with the fixed sidecar left for recovery; the engine never treats that post-effect error as an ordinary rebase. See [docs/dev/invariants.md](invariants.md) and [docs/dev/writes.md](writes.md).
## Bulk loader (`loader/mod.rs`)
@ -162,11 +183,11 @@ Atomicity guarantee for multi-statement mutations: a mid-query failure leaves La
| Mode | Semantics | Path (post-MR-794) |
|---|---|---|
| `Overwrite` | Replace all data in the target tables on the branch | Same accumulator; one `stage_overwrite` + `commit_staged` per touched table at end-of-load (a staged Lance `Operation::Overwrite` transaction — HEAD does not advance until commit; MR-793 Phase 2); publisher CAS. |
| `Append` | Strict insert; duplicates error | In-memory `MutationStaging` accumulator; one `stage_append` + `commit_staged` per touched table at end-of-load; publisher CAS. |
| `Merge` | Upsert by `id` (`merge_insert`) | Same accumulator; one `stage_merge_insert` per touched table at end-of-load (Merge mode dedupes by `id`, last-write-wins); publisher CAS. |
| `Overwrite` | Replace all data in the target tables on the branch | Same accumulator; one staged Lance `Operation::Overwrite` transaction per touched table. A pre-effect authority change is strict `ReadSetChanged`; no automatic replay. |
| `Append` | Strict insert; duplicates error | One `stage_append` transaction per touched table. A pre-effect authority change discards the whole parsed/validated attempt and fully reprepares with a bounded retry. |
| `Merge` | Upsert by `id` (`merge_insert`) | One `stage_merge_insert` transaction per touched table (deduped by `id`, last-write-wins). The same bounded full-reprepare rule as Append applies before effects. |
For all three modes, a mid-load failure (RI / cardinality violation, validation error) leaves Lance HEAD untouched on the staged tables — the next load on the same tables proceeds normally with no `ExpectedVersionMismatch`.
All three modes then use the same schema → branch → sorted-table gate, v3 recovery, zero-retry table commit, and exact publisher-precondition path as mutation. A parse, RI, cardinality, or validation failure leaves Lance HEAD untouched. After any table effect, any later error is `RecoveryRequired`. Load, mutation, and schema apply build no physical indexes inline; explicit `ensure_indices`/`optimize` reconciliation materializes declared intent later.
## `load` and the deprecated `ingest` shims

46
docs/dev/handoff-rfc-013-write-path.md
View file

@ -61,11 +61,14 @@ amplification = PR2's target.)
### A.2 What is LANDED on `main`
- **Step 2a** — `optimize` compacts the internal tables too (`__manifest` / `_graph_commits` /
`_graph_commit_actors`), so a *periodically-compacted* graph keeps Term-1 flat. (Cleanup/version-GC
of them is the still-open PR1.)
- **Step 2a** — #291 originally added compaction for all three then-current
internal tables (`__manifest`, `_graph_commits`, and `_graph_commit_actors`).
After strand-and-retire removed the two lineage datasets, the live behavior is
`__manifest`-only internal compaction. A periodically-compacted graph therefore
keeps the remaining Term-1 flat. (`__manifest` cleanup/version-GC is still open.)
- **Phase 7 / #299** (`1c5cb874`) — graph lineage lives in `__manifest` (`graph_commit` +
`graph_head:<branch>` rows in the same publish merge-insert; `_graph_commits` is now a projection;
`graph_head:<branch>` rows in the same publish merge-insert; `CommitGraph` is
now a projection and the former lineage datasets are gone;
v3→v4 internal-schema migration; schema-version floor). This removed the per-write commit-graph
scan and closed the manifest→commit-graph atomicity + commit-graph-parent-under-concurrency gaps.
**This is the base everything below builds on.**
@ -202,12 +205,14 @@ for the canonical list. Current reality:
- **Step 3a** — opener bypass: write opens go direct (`Dataset::open` by URI + version)
instead of the Lance-namespace builder (#288). **This already banked the dominant
depth win** — see §2 below; it reframes everything.
- **Step 2a** — internal-table compaction: `optimize` now compacts `__manifest` /
`_graph_commits` / `_graph_commit_actors` (#291). Plus the RFC latency-model
correction (#292).
- **Step 2a** — internal-table compaction: #291 originally covered
`__manifest` plus the two then-live lineage datasets. The current
post-retirement implementation compacts only `__manifest`. Plus the RFC
latency-model correction (#292).
- **Step 4 / Phase 7** — graph lineage moved into `__manifest` (#299 `1c5cb874`):
`graph_commit` + mutable `graph_head:<branch>` in the publish merge-insert,
`_graph_commits` now a projection. **The base for the live branch (§A).**
with `CommitGraph` projected directly from those rows and no separate lineage
dataset. **The base for the live branch (§A).**
- **Optimize-vs-write race** — optimize survives a cross-process write race on the
same table (#297, **LANDED** — origin/main `6d4606a8`; see §6 for why it's not
redundant with Design A). Step 3b stacks on top of this.
@ -312,7 +317,7 @@ single staleness feeds **two distinct failure modes**, both surfaced this cycle:
`writes.rs::served_strict_delete_after_external_optimize_advance_auto_refreshes`
(`#[ignore]` on branch `fix/write-path-stale-view-probe`). **The naive "just probe" fix is
proven wrong** — a blanket probe silently refreshes past *logical* advances too, breaking
`consistency::stale_handle_public_mutation_must_refresh_then_retry` (the deliberate
`consistency::stale_handle_strict_mutation_returns_read_set_changed_then_refresh_retry` (the deliberate
cross-process lost-update OCC primitive). The fix must **discriminate by op class**.
**Both fold into Design A (step 5), same as §1c.** `open_txn`'s one warm probe makes the base
@ -425,13 +430,14 @@ reopen/replan **semantics** are permanent. (Noted in RFC §6.6.)
## 4. DONE: Step 3b — capture-once `WriteTxn` (shipped on `rfc-013-step-3b-writetxn-v2`)
**Delivered:** on the **table-touch hot path**, a single `mutate`/`load` validates the schema
contract **once** and opens each touched data table **at most once** — a constant-factor/RTT
win (not a depth-slope win; 1a). Two cost gates in `write_cost.rs` lock it (both on a node
insert): `write_validates_schema_contract_once` (3 `read_text` / 2 `exists`, was 12/9) and
contract once at capture and once at the RFC-022 pre-effect gate, with one cheap trailing
identity-marker read fencing capture (never per table), and opens each touched data table
**at most once** — a constant-factor/RTT win (not a depth-slope win;
1a). Two cost gates in `write_cost.rs` lock it (both on a node insert):
`write_schema_io_is_bounded_to_capture_fence_and_effect_gate` (7 `read_text` / 4 `exists`) and
`keyed_insert_opens_table_at_most_once` (`data_open_count <= 1`, was 4). The carrier is the
minimal `WriteTxn { branch, base }`, threaded as `Option<&WriteTxn>` (`Some` on the hot
mutate/load path, `None` byte-identical everywhere else); it **converges into** step 5's
`PublishPlan`.
operation-local `WriteTxn { branch, base, authority, catalog }`; the catalog is built from the
exact accepted IR named by the authority token. It **converges into** step 5's `PublishPlan`.
**Not "once" everywhere (scope, not regression):** edge endpoint / cardinality RI validation
(`ensure_node_id_exists`, the loader's RI + cardinality) still resolves through
@ -444,8 +450,9 @@ gate yet (only the node gates above).
Commits (off merged-#297 main):
- **Stage 0** — scope `open_count` → `data_open_count`/`internal_open_count` by URI class
(the review fix: `open_dataset_tracked` also opens `__manifest`/`_graph_commits`, so the
raw counter conflated them and the gate was unreachable). Re-baselined RED 4.
(historically, before lineage-table retirement, `open_dataset_tracked` also opened
`__manifest`/`_graph_commits`, so the raw counter conflated them and the gate was
unreachable). Re-baselined RED 4.
- **Commit A (schema-once)** — capture `txn` once at entry (the single validation); the 4
validation sites collapse: S1 (entry `ensure_schema_state_valid`) removed; S3a
(`open_for_mutation_on_branch`) + S3b (`prepare_updates_for_commit`) source `txn.base`;
@ -587,7 +594,8 @@ for #298** (which built none of those constructs) but are **load-bearing constra
residual concurrent TOCTOU is the §7.1 gap (step 4) — un-widen here, don't over-reach.
- **Step 4 / Phase 7** (`iss-991`): **LANDED on `main` as #299 (`1c5cb874`).** Lineage now lives
in `__manifest` (`graph_commit` + mutable `graph_head:<branch>` in the same merge-insert;
`_graph_commits` is a projection). Removed the per-write `commit_graph.refresh`; closed the
`CommitGraph` is projected from those rows and the old lineage dataset is
gone). Removed the per-write `commit_graph.refresh`; closed the
manifest→commit-graph atomicity + commit-graph-parent-under-concurrency gaps. *(Historical note,
kept for the §7.1 framing it carried:)* it
carries the §7.1 *concurrent* write-skew fix (needs the `graph_head` contention row) —
@ -600,7 +608,7 @@ for #298** (which built none of those constructs) but are **load-bearing constra
false-fail** (§1d.2) — the op-class-aware precondition + `open_txn` probe. The contract is
two tests passing *together*: un-ignore
`writes.rs::served_strict_delete_after_external_optimize_advance_auto_refreshes` (goes green)
*while* `consistency::stale_handle_public_mutation_must_refresh_then_retry` stays green
*while* `consistency::stale_handle_strict_mutation_returns_read_set_changed_then_refresh_retry` stays green
(maintenance fast-forwards; logical fails loudly). Self-contained enough to ship standalone
like #297 if prod pain is acute; otherwise fold into the single PublishPlan delta-interpreter.
- **Step 2b** — internal-table cleanup + the Q8 monotonic watermark (a Lance boundary tag).

9
docs/dev/index.md
View file

@ -74,6 +74,15 @@ pattern, not just the outcome.
|---|---|
| camelCase property filters lowercased at runtime (#283) — two engine→Lance boundaries, two different fixes | [bug-case-fix.md](bug-case-fix.md) |
## Active Design Reviews
Review ledgers record open findings against proposed architecture. They remain
as durable disposition history after closure, so RFC backlinks stay valid.
| Area | Read |
|---|---|
| RFC-022–027 split architecture review — open correctness blockers, dependency corrections, and required acceptance evidence | [rfc-022-027-architecture-review.md](rfc-022-027-architecture-review.md) |
## Active Implementation Plans
Working documents for in-flight feature work. Removed when the work lands.

63
docs/dev/invariants.md
View file

@ -146,7 +146,11 @@ converge the physical state.
drifts" and "manifest-derivable reconciler" items, are instances; so is
bounding a read's cost to its working set rather than the commit count. This
is the structural face of "engineering is programming integrated over time":
both failure modes are liabilities that compound as the system grows.
both failure modes are liabilities that compound as the system grows. At a
write/control boundary, schema identity, compiled catalog, and manifest
authority must come from one operation-local accepted view: validating a
fresh on-disk marker while planning or enumerating gates from a stale warm
catalog is not a coherent derivation.
## Current Truth Matrix
@ -155,11 +159,11 @@ converge the physical state.
| Multi-table commit | Manifest CAS plus recovery sidecars; not a single Lance primitive | [writes.md](writes.md), [architecture.md](architecture.md) |
| Constructive mutations | In-memory `MutationStaging`, one end-of-query table commit per touched table, then one manifest publish | [writes.md](writes.md), [execution.md](execution.md) |
| Deletes | Staged like inserts/updates (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, MR-A) — no inline HEAD advance; mixed insert/update/delete in one query rejected by D2 as a deliberate boundary (constructive XOR destructive per query; compose via separate mutations or a branch) | [query-language.md](../user/queries/index.md), [writes.md](writes.md) |
| Branch delete | Manifest is the single authority, flipped atomically first; per-table forks are derived state, reclaimed best-effort (`force_delete_branch`) with the `cleanup` reconciler as the guaranteed backstop. Reusing a name whose reclaim failed before `cleanup` surfaces an actionable error | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md) |
| Branch delete | Manifest is the single authority, flipped atomically first; per-table forks are derived state, reclaimed best-effort (`force_delete_branch`) with the `cleanup` reconciler as the guaranteed backstop. Under schema/target/all-table gates, a target-scoped unresolved sidecar may be made unreachable by deletion and is then audit-discarded by recovery; graph-global SchemaApply still blocks. Reusing a name whose fork reclaim failed before `cleanup` surfaces an actionable error | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) |
| Schema validation | Type checks, required fields, defaults, edge endpoint checks, and edge cardinality are enforced on write paths | [schema-language.md](../user/schema/index.md), [execution.md](execution.md) |
| Unique constraints | Value/enum, uniqueness, edge-RI, and cardinality route through ONE unified, catalog-derived evaluator (`crate::validate`) on ALL THREE write surfaces — branch-merge, mutation, and bulk load: Δ-scoped (checks the delta, not the whole graph) and index-backed (probes committed state through its BTREE rather than full-scanning every catalog table), reusing the leaf checks (`loader::validate_value_constraints`/`validate_enum_constraints`/`composite_unique_key`) so the surfaces cannot drift. This closed the prior merge bug (merge validated `@range`/`@check` but not enum) AND the **cross-version uniqueness gap** on the mutation and load paths (a duplicate of a committed `@unique` value is now rejected; the merge path always enforced it). The committed view is the merge target snapshot (merge), the write's pinned `txn.base` (mutation), or the pinned pre-load base (load — `Overwrite` validates the batch as the whole new image, committed view empty); `@card` reads LIVE HEAD per edge table on the mutation path only (the #298 stale-handle fix). `@key` is id-backed, so it is checked intra-delta only (a committed holder of a key value is always the same row — an upsert), skipping a wasted O(Δ) probe per keyed row; `@unique` (non-key) groups do the committed lookup. | [schema-language.md](../user/schema/index.md) |
| Unique constraints | Value/enum, uniqueness, edge-RI, and cardinality route through ONE unified, catalog-derived evaluator (`crate::validate`) on ALL THREE write surfaces — branch-merge, mutation, and bulk load: Δ-scoped (checks the delta, not the whole graph) and structured-filter-backed (committed probes use a BTREE when reconciled and remain correct by scanning while it is pending), reusing the leaf checks (`loader::validate_value_constraints`/`validate_enum_constraints`/`composite_unique_key`) so the surfaces cannot drift. This closed the prior merge bug (merge validated `@range`/`@check` but not enum) AND the **cross-version uniqueness gap** on the mutation and load paths (a duplicate of a committed `@unique` value is now rejected; the merge path always enforced it). The committed view is the merge target snapshot (merge), the write's pinned `txn.base` (mutation), or the pinned pre-load base (load — `Overwrite` validates the batch as the whole new image, committed view empty); `@card` refreshes the manifest-visible graph-branch snapshot on the mutation path only (the #298 stale-handle fix), then follows each entry's actual inherited/owned Lance ref. `@key` is id-backed, so it is checked intra-delta only (a committed holder of a key value is always the same row — an upsert), skipping a wasted O(Δ) probe per keyed row; `@unique` (non-key) groups do the committed lookup. | [schema-language.md](../user/schema/index.md) |
| Storage trait | `TableStorage` (via `db.storage()`) is staged-only; the sole inline-commit residual (`create_vector_index`) is split onto a separate sealed `InlineCommitResidual` trait reached via `db.storage_inline_residual()` (MR-854), so §1 holds by construction; capability/stat surfaces are roadmap | [writes.md](writes.md), [architecture.md](architecture.md) |
| Index lifecycle | `@index`/`@key` declares *intent*; the physical index is derived state and never fails a logical op. `schema apply` builds no indexes (records intent only; index-only changes touch no table data). `load`/`mutate` build inline through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector) that fault-isolates an untrainable Vector column into a *pending* index instead of aborting. `optimize`/`ensure_indices` is the reconciler: it creates declared-but-missing indexes and folds appended/rewritten fragments into existing ones (`optimize_indices`), reporting still-pending columns. Explicit maintenance call, not yet a background loop | [indexes.md](../user/search/indexes.md), [maintenance.md](../user/operations/maintenance.md) |
| Index lifecycle | `@index`/`@key` declares *intent*; the physical index is derived state and never fails a logical op. `schema apply`, `load`, and `mutate` build no indexes inline: RFC-022 mutation/load sidecars describe only their exact data effects, and index availability must never become a correctness prerequisite. `optimize`/`ensure_indices` is the reconciler: through the single `build_indices_on_dataset_for_catalog` chokepoint it creates declared-but-missing indexes (enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector), folds appended/rewritten fragments into existing ones (`optimize_indices`), and reports untrainable Vector columns as pending. Explicit maintenance call, not yet a background loop | [indexes.md](../user/search/indexes.md), [maintenance.md](../user/operations/maintenance.md) |
| Traversal IDs | Runtime still builds `TypeIndex`; Lance stable row-id based graph IDs are roadmap | [architecture.md](architecture.md), [query-language.md](../user/queries/index.md) |
| Auth | Bearer token hashing and server-side actor resolution are implemented at the HTTP boundary | [server.md](../user/operations/server.md), [policy.md](../user/operations/policy.md) |
| Tests | Tempdir-backed Lance tests are the current substrate; the storage adapter has an in-memory backend for adapter-level contract tests, but Lance datasets bypass it | [testing.md](testing.md) |
@ -215,12 +219,15 @@ them explicit.
`maintenance.rs::optimize_compacts_blob_table_alongside_plain_table` pin the
positive behavior (a red there means blob compaction regressed — restore the
skip machinery from git history).
- **Recovery is serialized against live writers in-process only:** the
write-entry heal (and `refresh`) serialize against a live writer's sidecar
lifetime via the per-`(table, branch)` write queues plus the schema-apply
serialization key — all in-process primitives. A recovery pass in one
process cannot serialize against a live writer in another (the open-time
sweep has the same exposure, and always has): it may roll a live foreign
- **Recovery is serialized against live writers in-process only:** every
`Omnigraph` handle for one canonical local root identity (lexically absolute,
with existing symlink ancestors resolved) shares a root-scoped queue manager;
object-store/custom scheme identities remain their normalized opaque URIs.
The write-entry heal, `refresh`, and Full ReadWrite-open sweep serialize
against a live writer's complete sidecar lifetime in schema → branch → sorted
table order and re-check sidecar existence after waiting. These remain
in-process primitives: a recovery pass cannot serialize against a live writer
in another process. It may roll a live foreign
writer's sidecar forward, which degrades to publisher-CAS contention for
data writes but can race the schema-staging promotion for a foreign live
schema apply. The roll-**forward** CAS contention is now
@ -238,22 +245,26 @@ them explicit.
cross-process serialization primitive (e.g. lease-based use of the
schema-apply lock branch) — design it before promoting multi-process write
topologies.
- **Fork reclaim is in-process-safe only:** the first write to a table on a
branch forks it (a Lance `create_branch` that advances state before the
manifest publish). An interrupted fork (crash, or a cancelled request
future) leaves a manifest-unreferenced branch ref. The next write self-heals
it — `reclaim_orphaned_fork_and_refork` (`force_delete_branch` + re-fork)
— but reclaim is only safe because the writer holds the per-`(table,
branch)` write queue from before the fork through the publish AND re-checks
the live manifest under it, so no *in-process* writer can be mid-fork. A
reclaim cannot serialize against a foreign-*process* in-flight fork: it may
force-delete a peer's just-created ref, which makes that peer's commit fail
and retry — the same one-winner-CAS exposure as above, not corruption. The
reclaim never fires unless in-process-queue + manifest authority both prove
the ref is manifest-unreferenced. `cleanup`'s per-table reconciler
(`reconcile_orphaned_branches`) is the guaranteed backstop for any fork the
write path never revisits. Both degrade to a no-op if Lance ships an atomic
multi-dataset branch op.
- **Fork ownership is durable, but Lance ref deletion is not conditional:** a
first-touch Mutation/Load table no longer creates its target ref while
preparing. Under schema → branch → table gates it revalidates, durably arms a
schema-v3 sidecar naming the target, creates the ref, then stages branch-local
files and commits. Both `reclaim_orphaned_fork_and_refork` and
`reconcile_orphaned_branches` consult pending sidecars before destruction; a
foreign claim is conflict/indeterminate, never permission to delete. Full
recovery accepts sidecar-before-ref crashes and deletes an unpublished fork
only when it is still exactly at the inherited version and no other sidecar
claims `(table_path, target ref)`. A no-effect sidecar with a competitor
discards only itself; the last survivor either cleans the untouched ref or
recovers its owned effect. Partial rollback performs no-effect cleanup before
publishing its fixed outcome. This closes the cross-handle live-ref
deletion bug and keeps cleanup as the backstop for truly unclaimed refs.
The remaining multi-process gap is narrower but real: Lance exposes
`force_delete_branch`, not compare-and-delete by `BranchIdentifier`, so a
foreign process can create intent/ref between the final list/check and the
delete. The documented single-writer-process support boundary remains until
Lance provides a conditional ref primitive (or OmniGraph adds a distributed
fence); process-local queues are not credited as that primitive.
- **Local `write_text_if_match` is not a cross-process CAS:** object-store
backends use a true conditional put (ETag If-Match; the in-memory test
backend too), but upstream `object_store` leaves `PutMode::Update`

2
docs/dev/rfc-004-cluster-graph-schema-apply.md
View file

@ -36,7 +36,7 @@ The implementation spec's hard gate for this phase — failpoint recovery tests
What Phase 4 builds on (all shipped):
- **The engine's recovery discipline.** Writers that can advance Lance HEAD before manifest publish write `__recovery/{ulid}.json` sidecars carrying per-table pins (`expected_version`, `post_commit_pin`); `Omnigraph::open` in read-write mode classifies every pinned table (`NoMovement` / `RolledPastExpected` / `UnexpectedAtP1` / `UnexpectedMultistep` / `InvariantViolation`) and decides all-or-nothing: roll forward via one manifest publish, or roll back via `Dataset::restore`, recording an audit row attributed to `omnigraph:recovery`. The cluster inherits the *vocabulary* of this design but not its mechanics — see the roll-forward-only argument below.
- **The engine's recovery discipline.** Writers that can advance Lance HEAD before manifest publish write `__recovery/{ulid}.json` sidecars carrying per-table pins (`expected_version`, `post_commit_pin`); `Omnigraph::open` in read-write mode classifies every pinned table (`NoMovement` / `RolledPastExpected` / `UnexpectedAtP1` / `UnexpectedMultistep` / `InvariantViolation`) and decides all-or-nothing: roll forward via one manifest publish, or roll back via `Dataset::restore`. Every completed action records an audit row with the interrupted writer's actor in `recovery_for_actor`. A schema-v3 roll-forward publishes the sidecar's pre-minted lineage and therefore preserves that writer's commit id and actor; rollback and legacy recovery commits use `omnigraph:recovery`. The cluster inherits the *vocabulary* of this design but not its mechanics — see the roll-forward-only argument below.
- **The engine's schema-apply surface.** `apply_schema_as(desired_source, SchemaApplyOptions { allow_data_loss }, actor)` returns `SchemaApplyResult { supported, applied, manifest_version, steps }`; `preview_schema_apply_with_options` returns the migration plan plus desired catalog without applying; the `__schema_apply_lock__` branch serializes schema applies graph-wide and refuses to run while user branches exist. Policy enforcement (`enforce(SchemaApply, TargetBranch("main"), actor)`) happens before the lock.
- **Graph init.** `Omnigraph::init(uri, schema_source)` with a strict preflight (errors if schema artifacts exist) and an atomic `_schema.pg` claim. A documented gap: a failed init does not clean up Lance datasets or `__manifest/` it already created.
- **No engine graph-delete primitive.** Deleting a graph today means removing its object-store prefix. This RFC works with that fact rather than waiting on a primitive.

549
docs/dev/rfc-022-027-architecture-review.md Normal file
View file

@ -0,0 +1,549 @@
# Architecture review: RFC-022 through RFC-027
**Status:** Open review findings
**Date:** 2026-07-11
**Audience:** RFC authors, engine/storage maintainers, and release reviewers
**Reviewed against:** OmniGraph 0.8.1; Lance 9.0.0-beta.15 at
`f24e42c11a742581365e1cbe17c906ea2dac1bc6`; full Lance transaction,
branch/tag, MemWAL, row-lineage, read/write, compaction, and cleanup
specifications; pinned Rust implementation where the public specification is
not precise enough
This is a review ledger, not a competing specification. The RFCs remain the
normative proposals. A finding is closed only when the affected RFC either:
1. incorporates the required contract and tests; or
2. explicitly rejects the finding with evidence and records the resulting
support boundary.
Items marked **BLOCKER** must be dispositioned before the RFC that owns them is
accepted. Items marked **TIGHTENING** may be resolved during revision, but they
must not silently disappear from review.
This ledger is durable review history, not a throwaway implementation plan.
After every finding is dispositioned, change its status to closed and record
the resolving RFC sections/commits; keep the RFC backlinks valid.
## Overall architectural judgment
The central architecture remains the right one:
- Lance is the physical storage/versioning truth.
- `__manifest` is the graph-visible authority and one graph publish is the
visibility point for a logical graph commit.
- RFC-022 should define one correctness protocol with explicit physical-effect
adapters, not pretend that Lance offers one universal staged primitive.
- Key fencing, durable heads, retention, MemWAL, and lineage acceleration are
separate irreversible decisions and should keep separate evidence and format
gates.
The current blockers are boundary problems, not a reason to replace that core.
They concern crash classification, actual fencing, foreign-branch authority,
and capability activation.
> 💬 **Second-pass verification (2026-07-11):** I independently re-verified this
> ledger's two sharpest new substrate claims against the pinned checkout before
> commenting: BLOCKER-01's two-phase branch create is confirmed in Lance's own
> doc comment, and BLOCKER-04's lazy-branch exposure is confirmed
> architecturally — and appears to be a live bug in the shipped `cleanup`, not
> only an RFC gap (see the comment there). Overall judgment: concur with this
> ledger; three of its findings supersede corrections I applied to the RFCs on
> 2026-07-11 (noted inline at BLOCKER-07, BLOCKER-11, and tightening 5).
The latest revisions improved several important points and those changes should
remain:
- exact recovery-goal convergence is distinguished from adopting an unrelated
newer version;
- MemWAL is treated as Lance's strategic streaming architecture, with API and
format evolution as the integration risk;
- keyed fast-forward merge now has an explicit embedding-table memory gate;
- the legacy `__manifest` PK representation is preserved rather than
“normalized” into an incompatible form;
- cleanup's required operating cadence and the cost of indefinite deferral are
explicit;
- native branch refs are no longer described as cross-process-safe merely
because a process-local gate exists.
## Dependency correction
Stable table identity and incarnation are consumed by more than durable heads.
The intended dependency shape is therefore:
```text
RFC-022 unified write protocol
|
+-- stable identity/incarnation capability
| +-- RFC-024 durable heads
| +-- RFC-025 checkpoint rows
| `-- RFC-026 stream/reject authority
|
+-- RFC-023 key fencing
| `-- RFC-026 keyed streaming mode
|
+-- RFC-025 retention
+-- RFC-026 MemWAL
`-- RFC-027 merge-delta research
```
The public RFC process accepts whole RFCs, not independently accepted sections.
The identity capability must therefore be extracted into its own RFC, or
RFC-024 must be accepted in full before identity-dependent siblings. An
implementation may phase an accepted RFC internally, but it cannot treat a
sub-contract of an unaccepted RFC as a separately accepted decision.
## Blocking findings
### BLOCKER-01 — native graph-branch control is multi-phase
**Affected:** [RFC-022 §7](../rfcs/rfc-022-unified-write-path.md#7-native-graph-branch-ref-control-protocol)
RFC-022 currently models branch create/delete as one native ref mutation whose
completion is the visibility point. The pinned Lance implementation is more
specific:
- [`Dataset::create_branch`](https://github.com/lance-format/lance/blob/f24e42c11a742581365e1cbe17c906ea2dac1bc6/rust/lance/src/dataset.rs#L488-L535)
first commits a shallow-cloned branch dataset and only then writes
`BranchContents`. Lance explicitly calls this a non-atomic two-phase
operation and calls `BranchContents` the source of truth.
- [branch deletion](https://github.com/lance-format/lance/blob/f24e42c11a742581365e1cbe17c906ea2dac1bc6/rust/lance/src/dataset/refs.rs#L548-L584)
removes `BranchContents` before cleaning the branch directory.
Therefore create can crash with a zombie branch dataset and no authoritative
ref. A retry with the same name can fail until the zombie is reclaimed. Delete
can make the branch logically absent and then return a cleanup error.
**Required disposition:** define an idempotent control-operation classifier for
at least these states:
| Operation | Physical state | Logical result |
|---|---|---|
| create | no clone, no `BranchContents` | not started |
| create | clone only | zombie; reclaim before retry, or wait for an upstream completion API |
| create | clone plus matching `BranchContents` | complete |
| delete | `BranchContents` present | not deleted |
| delete | `BranchContents` absent, directory remains | deleted; reclaim pending |
| delete | both absent | complete |
The pinned public API cannot adopt the clone-only state: `BranchIdentifier` is
generated later by Lance's private `BranchContents` creation phase. The RFC
must therefore say whether a durable operation marker/sidecar is written before
the shallow clone, how retry distinguishes its own zombie from foreign state,
and whether it reclaims and retries or waits for a new upstream completion API.
It must also explain why cleanup failure after authoritative deletion is not
reported as if the branch still existed. Add failpoints at both create phases
and between delete authority removal and directory cleanup.
> 💬 **Verified (2026-07-11):** confirmed verbatim in the pinned source —
> `dataset.rs:488` documents "This is a two-phase operation… These two phases
> are not atomic. We consider `BranchContents` as the source of truth… may
> leave a zombie branch dataset", including the zombie-blocks-retry
> consequence and the cleanup duties this finding transcribes. The 2026-07-11
> RFC-022 §7 revision (single-writer-process boundary + conditional-ref
> upstream ask) addressed the *conditional-put* gap but not this crash
> classification — the two dispositions compose; both are needed.
### BLOCKER-02 — create-if-absent ownership is not a fencing lease
**Affected:** RFC-023 migration claim, RFC-024 migration claim, and
[RFC-025 §2.3](../rfcs/rfc-025-checkpoint-retention.md#23-cross-process-retention-claim)
`PutMode::Create` correctly elects one initial owner. A random token in that
object does not by itself fence a paused owner, because Lance tag, branch,
migration, and cleanup effects do not condition their commits on that token.
RFC-025 proposes a check-then-delete release; RFC-023 does not yet specify a
release protocol at all. For the check-then-delete shape, the race is:
1. old owner reads and verifies token A;
2. old owner pauses;
3. takeover removes A and creates token B;
4. old owner resumes and deletes B, or resumes a destructive physical effect.
“The stale process checks again” does not close the window after its final
check. Calling the random owner token a fencing token overstates the substrate
guarantee.
**Required disposition:** choose one honest contract:
- no takeover while the prior process can resume; operator recovery proves the
process/host is dead before removing the claim;
- a substrate-enforced lease/fencing primitive whose monotonically newer token
is checked by every protected effect, plus conditional compare-and-delete;
- or a non-takeoverable claim that requires explicit offline repair.
The `StorageAdapter` does not currently expose conditional delete, and its local
`write_text_if_match` is not a cross-process CAS. Any selected design must work
on local and S3 or explicitly narrow the supported backend/topology. For a
substrate-enforced lease, tests pause an owner after its last token check,
perform takeover, then resume it; the old owner must be unable to mutate state
or release the new claim. A host-death or non-takeoverable design instead proves
that takeover is refused until external fencing/death proof is established.
> 💬 **Concur (2026-07-11):** the paused-owner race is real and the "fencing
> token" label overstated the guarantee. Given the substrate as it stands (no
> conditional delete on the adapter; the documented local `write_text_if_match`
> gap), the only honest near-term contract is the boring one — no takeover
> while the prior process can resume; operator proves process/host death or
> uses explicit offline repair. The substrate-enforced lease is the right end
> state but should be written as an upstream/adapter work item, not assumed.
### BLOCKER-03 — key-conflict retry must first resolve partial effects
**Affected:** [RFC-022 §9](../rfcs/rfc-022-unified-write-path.md#9-concurrency-and-retry-semantics),
[RFC-023 §6](../rfcs/rfc-023-key-conflict-fencing.md#6-retry-and-validation-semantics),
and [RFC-026 §6](../rfcs/rfc-026-memwal-streaming-ingest.md#6-fold-protocol)
A retryable concurrent inserted-row-filter conflict is detected when a table
transaction attempts to commit. (`WhenMatched::Fail` may instead report a
pre-existing match during merge execution.) In a multi-table graph write or
MemWAL fold, table A may already have advanced under the armed sidecar before
table B reports the concurrent key conflict. Immediately “restart the entire
logical attempt” would replan around unresolved physical state, which RFC-022
forbids.
**Required disposition:** distinguish:
- a conflict before any physical table commit — finalize the already-durable
empty sidecar, then perform a safe full semantic restart;
- a conflict after any physical commit — keep the sidecar and resolve the
RFC-022 recovery outcome first. Normally the partial attempt must roll back
before a new semantic attempt can be prepared. If rollback is not safe, return
a typed recovery-required failure rather than retrying around it.
For strict insert, finalize an empty intent or retain a partial-effect sidecar,
as applicable, and return the terminal `KeyConflict`; RFC-022 permits partial
recovery to finish at the next synchronous barrier. Do not start another
attempt around that sidecar. For non-strict upsert, the barrier must resolve the
sidecar before the bounded automatic whole-operation retry. Add a failpoint/race
in which table N conflicts after table 1 has committed.
### BLOCKER-04 — live graph branches need physical GC protection
**Affected:** [RFC-025 §6](../rfcs/rfc-025-checkpoint-retention.md#6-cleanup-protocol-and-pruned-through-boundary)
The cleanup root set includes live graph branches, but RFC-025 creates Lance
tags only for named checkpoints. This is insufficient for lazy graph branches.
A graph branch may store, in its `__manifest`, a foreign reference to an old
version on a data table's `main` branch. Lance cleanup on that data table cannot
see the foreign OmniGraph row; it protects versions through its own branch and
tag references.
**Required disposition:** for every live graph-branch table reference, either:
- materialize a deterministic Lance tag/native ref that physically protects the
exact `(table branch, version)`; or
- choose a per-dataset cleanup cutoff no newer than the oldest live reference,
accepting that all intervening versions remain retained.
Checkpoint tags alone do not solve this. Add a test where a lazy branch pins an
old main-table version, main advances beyond the retention window, cleanup
runs, and the lazy branch still opens and reads the exact pinned state.
> 💬 **Escalation (2026-07-11):** this is very likely a live bug in the shipped
> `cleanup`, independent of RFC-025. Verified in code: `cleanup_all_tables`
> (`optimize.rs:802`) runs Lance `cleanup_old_versions` per data table with no
> lazy-branch pin logic, and a lazy graph branch creates **no Lance-native ref
> on the data table** until its first write to that table — its pin exists only
> as a foreign `__manifest` row Lance cannot see. Repro shape: create a branch,
> never write table X on it, advance main past the keep window, `cleanup
> --keep N` → the branch's pinned version of X is collected and the branch
> breaks. Per the repo's test-first rule this deserves a red regression test
> and an issue *now*; RFC-025's disposition should then build on that fix
> rather than owning the discovery. (Snapshot/time-travel pins share the
> mechanism but are history-trimming under an operator-confirmed policy; a
> live branch's working state is not history.)
### BLOCKER-05 — durable-head OCC must compare the full head token
**Affected:** [RFC-024 §5](../rfcs/rfc-024-durable-table-heads.md#5-atomic-publisher-contract)
The RFC says expected table versions are compared against table heads. Lance
version numbers can repeat across recreated datasets/branches, so version-only
comparison admits drop/recreate ABA.
**Required disposition:** define the OCC token as the complete logical head
identity, at least:
```text
(state, stable_table_id, incarnation_id, table_path,
table_branch, physical_ref_incarnation, table_version, schema_hash)
```
The publisher may encode that token differently, but retry/revalidation must
not reduce it to `table_version`. `physical_ref_incarnation` means an e_tag when
the backend supplies one, or another proven token that changes when a dataset
or native ref is deleted and recreated at the same path/branch/version. This is
distinct from the logical `incarnation_id`, which RFC-024 deliberately preserves
across an owner handoff. Add stale-writer races for both logical drop/recreate
and physical ref recreation at the same numeric version.
### BLOCKER-06 — MemWAL needs a capability/format activation barrier
**Affected:** [RFC-026 §3](../rfcs/rfc-026-memwal-streaming-ingest.md#3-enrollment-is-a-recoverable-inline-commit),
[§5](../rfcs/rfc-026-memwal-streaming-ingest.md#5-ack-path-validation-and-writer-lifecycle),
[§6](../rfcs/rfc-026-memwal-streaming-ingest.md#6-fold-protocol),
[§8](../rfcs/rfc-026-memwal-streaming-ingest.md#8-epoch-fenced-quiescence-barrier),
and [§12](../rfcs/rfc-026-memwal-streaming-ingest.md#12-phasing)
Enrollment creates persistent MemWAL metadata and `stream_state` changes the
correctness preconditions for schema, branch, maintenance, and data operations.
An older binary that understands key fencing but not stream lifecycle can ignore
`OPEN | DRAINING | SEALED`, mutate the base table, or perform a schema/branch
operation without draining acknowledged rows.
**Required disposition:** define:
- a graph capability/internal-format stamp written only after every enrolled
table and lifecycle authority is valid;
- old-binary/new-format and new-binary/partial-format refusal behavior;
- migration and rollback/roll-forward ordering;
- preservation rules for later heads/retention formats;
- genuine cross-version tests, not a stamp-rewind simulation.
Replica scope must also match RFC-023's recovery support boundary. Multiple
replicas may route acknowledgement traffic to one shard owner, but enrollment,
fold, and sidecar recovery remain single-writer-process operations until
foreign-process sidecar ownership is fenced. Do not advertise general replica
failover for those operations merely because MemWAL has a shard epoch.
### BLOCKER-07 — stable identity ownership contradicts sibling dependencies
**Affected:** [RFC-024 §3.1](../rfcs/rfc-024-durable-table-heads.md#31-identity-and-object-key),
[RFC-025 §2.1](../rfcs/rfc-025-checkpoint-retention.md#21-logical-authority-manifest-rows),
[RFC-026 §7](../rfcs/rfc-026-memwal-streaming-ingest.md#7-fold-time-rejection-is-atomic),
and [RFC-026 §8](../rfcs/rfc-026-memwal-streaming-ingest.md#8-epoch-fenced-quiescence-barrier)
RFC-024 now says it exclusively owns stable table identity and incarnation and
that identity-dependent sibling claims must wait for it. RFC-025 still calls
RFC-024 optional, and RFC-026 persists `stable-table-id` without declaring the
dependency.
**Required disposition:** extract the identity/incarnation format and migration
as a shared prerequisite RFC, or accept RFC-024 in full before RFC-025 and
RFC-026. Durable-head implementation can remain a later phase only after its
owning RFC and format contract are accepted.
The `_ingest_rejects` deterministic key must use stable table ID plus
incarnation rather than mutable `table_key`, or rename/recreate can change reject
identity and break replay idempotence.
> 💬 **Concur; supersedes a 2026-07-11 correction (2026-07-11):** the earlier
> pass placed identity ownership *inside* RFC-024 §3.1, which created exactly
> the contradiction described here (RFC-025 calls heads optional while
> consuming the identity RFC-024 claims to own). The extracted
> identity/incarnation RFC is the cleaner shape and should replace that
> ownership note. The `_ingest_rejects` key observation is an additional catch
> the earlier pass missed — endorsed.
### BLOCKER-08 — retention activation cannot precede migration selection
**Affected:** [RFC-025 §8](../rfcs/rfc-025-checkpoint-retention.md#8-migration-and-compatibility)
and [§11](../rfcs/rfc-025-checkpoint-retention.md#11-phasing)
RFC-025 leaves in-place migration versus export/import undecided. Its phase
table nevertheless puts “format activation” in Phase A and “selected migration
path” in Phase D. Activation cannot be implemented or reviewed before the
upgrade contract is known.
**Required disposition:** select the migration before Phase A, then specify
quiescence, partial-state refusal, main-stamp ordering, branch coverage, crash
recovery, and later capability preservation. If export/import is selected,
state the irreversible loss of branches, commit DAG, snapshots, and historical
checkpoints as part of the acceptance decision.
### BLOCKER-09 — decide whether these are public or internal RFCs
**Affected:** [public RFC process](../rfcs/README.md), RFC-022 through RFC-027
The files currently live in the public RFC track but use internal-style
`rfc-022-*` names, noncanonical `draft`/`research-blocked` statuses, and empty
owner metadata. In the public process, merging an RFC is acceptance; a merged
public RFC cannot simultaneously retain undispositioned acceptance blockers.
**Required disposition:** choose the track before merge:
- for the public track, rename to `NNNN-title.md`, use `Status: Proposed`, fill
the template's author/discussion/implementation metadata, and resolve every
blocker before the RFC PR merges; or
- move the proposals to `docs/dev/` and follow the maintainer-internal process,
leaving the public RFC directory for externally authorable accepted records.
RFC-027 may state “research-blocked” prominently as its technical state while
retaining the public lifecycle status `Proposed`.
> 💬 **Concur (2026-07-11):** same recommendation as the original family
> review — these are maintainer-internal proposals mid-revision; `docs/dev/`
> with the internal process is the low-friction disposition, keeping
> `docs/rfcs/` for its defined merge-equals-acceptance lifecycle. Whichever
> track is chosen, decide it before any of the set merges, since it defines
> what "dispositioned before acceptance" means for every other finding here.
## Protocol clarifications
### BLOCKER-10 — do not put foreign-branch facts in an atomic `ReadSet`
RFC-022 requires every `ReadSet` member to be arbitrated atomically by the
publish CAS. A CAS on reserved main cannot arbitrate a row on a named source
branch; a merge-target CAS cannot arbitrate a source-branch row.
Use the right category for each fact:
- checkpoint creation captures an immutable source version, revalidates it
before tagging, and then relies on the physical tag. A later source-head
advance is intentionally harmless, so the source head is an effect
precondition, not target-CAS authority;
- offline cleanup facts are protected by the selected fleet/retention barrier,
not by pretending one main-branch CAS covers every named branch;
- branch merge should define captured-source-commit semantics. If “latest
source at target publish” is required instead, it needs a real source-branch
fence held through the target CAS.
Keep target-branch values that must remain stable in the atomic `ReadSet`.
### BLOCKER-11 — RFC-022 and no-heads MemWAL need coarse OCC
**Affected:** [RFC-022 §10](../rfcs/rfc-022-unified-write-path.md#10-rollout-and-compatibility)
and [RFC-026 §6](../rfcs/rfc-026-memwal-streaming-ingest.md#6-fold-protocol)
[RFC-022's rollout](../rfcs/rfc-022-unified-write-path.md#10-rollout-and-compatibility)
currently says mutation/load read-set arbitration is likely blocked on
RFC-024 because probed-but-untouched tables lack mutable head rows.
For graph-content writes, `(branch incarnation, optional graph head)` is already
a conservative branch-wide authority token. An established branch changes its
`graph_head:<branch>` row on every graph commit; a fresh branch may initially
have no branch-specific head row, so absence plus incarnation is part of the
captured token. Schema identity needs its own atomically contended authority.
Any concurrent logical graph change forces full revalidation. RFC-024 table
heads can later reduce false contention by narrowing that token to the tables
actually read.
Today's publisher retries head-row contention by re-reading the live head and
reparenting the prepared write. That is insufficient for a
validation-sensitive plan. On every publisher retry, it must compare the
captured token and return control to full revalidation rather than reparenting
and continuing with stale validation.
The lower-liability rollout is therefore:
1. ship correct coarse OCC with `graph_head` plus schema identity;
2. introduce table-head tokens only after RFC-024 passes its independent format
and cost gates.
The no-heads RFC-026 fold path must carry the same captured branch token in its
`ReadSet`. Do not recouple RFC-022 or RFC-026 correctness to RFC-024
performance.
> 💬 **Concur; supersedes a 2026-07-11 correction (2026-07-11):** the premise
> is independently verified — `graph_head:<branch>` is one mutable row updated
> in every graph-content publish (the deliberate contention point pinned by
> the concurrent-disjoint-writers tests), so the coarse token genuinely
> arbitrates probed-but-untouched same-branch tables. This is a better design
> than the earlier RFC-022 §10 wording ("likely blocked on RFC-024"), which
> should be revised to the two-step rollout described here. One cost worth
> stating in the revision: coarse OCC makes *any* concurrent commit on a busy
> branch force full revalidation of in-flight writers — real throughput cost
> on agent-fleet branches, and precisely the honest motivation for RFC-024's
> later narrowing (rather than a correctness argument for it).
> **Implementation disposition (2026-07-11):** mutation/load now capture the
> native Lance `BranchIdentifier`, exact optional `graph_head`, and accepted
> schema identity; revalidate under a branch-then-table gate; and pass the same
> token to every publisher retry and schema-v3 recovery decision. Metadata-only
> schema-apply tests pin the required invariant that supported schema changes
> move `graph_head` even when no data-table version changes. This closes the
> coarse mutation/load cell without claiming a general schema-authority row or
> multi-process native-ref fencing. RFC-024 remains the false-contention
> narrowing step.
### TIGHTENING-01 — symmetric Lance conflicts are not obviously an activation gate
RFC-023 correctly identifies Lance's directional filtered/unfiltered behavior.
However, its own mandatory fleet outage, recovery drain, historical-duplicate
validation, stamp-last activation, and old-binary refusal are intended to make
unfiltered writers unreachable after activation.
The current RFC-023 routing table still permits a direct existing-row `Update`,
whose Lance operation has no inserted-row filter. Therefore unfiltered
transactions do remain reachable after activation. Symmetry can be demoted only
if every insertion-bearing keyed path routes through filtered merge-insert and
the RFC proves a direct update cannot insert a row or change `id`; affected-row
conflict metadata must continue to cover updates to the same existing row.
After that narrowing, upstream symmetry is useful defense in depth and a
valuable Lance surface guard, but it need not block activation for transaction
pairs that cannot violate key insertion. Two old unfiltered writers are not
protected by symmetry in either case.
> 💬 **Concur, with the deciding check named (2026-07-11):** the concrete
> unfiltered-`Update` reachable after activation is the matched-only
> partial-schema update merge (`WhenMatched::UpdateAll` +
> `WhenNotMatched::DoNothing`, the field-level-updates path in PR #342): it
> classifies zero inserts, so whether it emits an **empty** filter (`Some`
> with no keys — symmetric-safe) or **no** filter (`None` — the reachable
> unfiltered transaction) is exactly what decides whether this demotion is
> sound. That is a one-test question against the pinned revision and should be
> pinned as a surface guard before the demotion is accepted.
## Specification and acceptance tightening
These are smaller than the blockers above, but should be resolved before the
RFC set is merged:
1. **Checkpoint-name normalization:** define allowed bytes, Unicode
normalization, case sensitivity, maximum encoded length, reserved prefixes,
and normalization-version compatibility. Test collisions and reuse across
versions.
2. **RFC lifecycle values:** [`docs/rfcs/README.md`](../rfcs/README.md#status-values)
defines `Proposed`, `Accepted`, `Declined`, `Superseded by NNNN`, and
`Implemented`. Either use those values or amend the process before using
`draft` and `research-blocked` as machine-readable statuses. Research-blocked
can remain prominent in RFC-027's body while its lifecycle status is
`Proposed`.
3. **Provisional version naming:** sibling RFCs should say “RFC-024 heads
format” rather than treating `v5` as permanent while RFC-024 says the numeral
will change if another format lands first.
4. **Capability ordering:** RFC-025 must specify retention-first, heads-first,
and co-release preservation rules. RFC-026 is a conditional dependency when
streams are enrolled because cleanup must persistently quiesce them.
5. **Memory measurement:** `helpers::cost` measures I/O, not peak RSS. RFC-023's
adopted-merge and RFC-027's lineage memory gates should use the subprocess
`scenarios.rs` harness or an equivalent `wait4`/`ru_maxrss` instrument and
name dataset sizes, baseline, cap, and pass threshold.
> 💬 **Concur; catches a gap in a 2026-07-11 correction:** the RFC-023
> §11.4 memory gate added that day states the bound ("peak memory bounded by
> batch size") without naming an instrument that can measure it — this item
> is what makes that gate enforceable rather than aspirational.
6. **Derived-index fallbacks:** acceptance tests must force index-absent and
partially covered states for RFC-024 head lookup and RFC-027 candidate
discovery, assert identical logical results, and assert the promised
degraded-cost/fallback telemetry.
## Acceptance order after disposition
The review does not require all RFCs to land together. A safe order is:
1. accept RFC-022 after branch-control recovery, coarse `graph_head` OCC, and
foreign-branch fact classification are explicit;
2. accept and land the stable identity RFC/capability;
3. accept RFC-023 once partial-effect retry and its format/fleet barrier are
complete;
4. evaluate RFC-024 independently on its physical lookup cost gate;
5. accept RFC-025 after physical protection of both checkpoints and live graph
branches plus a selected upgrade path;
6. accept RFC-026 after its capability barrier and writer-ownership scope are
explicit;
7. keep RFC-027 in research until deletion-delta discovery passes its stated
correctness and flat-cost gates.
This ordering preserves the split's main benefit: a blocked performance
optimization or research result does not hold correctness work hostage.
> 💬 **Concur with the order; two sequencing notes (2026-07-11):**
> BLOCKER-04's escalation means step 5 has a prerequisite outside this list —
> the live lazy-branch cleanup exposure needs its red regression test and fix
> in the shipped `cleanup` first, so RFC-025 builds on a correct baseline. And
> step 3's TIGHTENING-01 demotion hinges on the partial-update filter check
> noted there; if that check lands `None`, upstream symmetry returns to the
> activation gate and RFC-023's timeline moves accordingly.

10
docs/dev/testing.md
View file

@ -22,7 +22,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
| `branching.rs` | Branch create / list / delete, lazy fork |
| `merge_truth_table.rs` | Merge-pair truth table (MR-786): all 9×9 `(left_op, right_op)` cells from `{noop, addNode, removeNode, addEdge, removeEdge, setProperty, dropProperty, addLabel, removeLabel}`. Adding a new op to `OpVariant` forces a compile error in `build_case` until the new row + column are dispositioned. 36 executable cells run through real `branch_merge` with a structured oracle (`MergeOutcome` / `MergeConflictKind` + graph-state assert); 45 cells involving `dropProperty`/`addLabel`/`removeLabel` are recorded as `Unsupported` until the mutation grammar grows. |
| `merge_fast_forward.rs` | Fast-forward branch-merge cost + correctness: an append-only adopted-source merge routes *new* rows through `stage_append` instead of one whole-delta `stage_merge_insert` (the full-outer join that exhausted the DataFusion memory pool on embedding-bearing tables). The regression gate is structural — it asserts WHICH staged-write primitive the merge invokes via the task-local write probes (`omnigraph::instrumentation`), not a brittle size threshold; also: merge yields source state, defers vector indexes to the reconciler, streams blob columns |
| `writes.rs` | Direct-publish writes: cancellation, non-strict insert/merge rebase under the per-table queue, strict stale-write conflicts, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery); the lance#7444 row-id-overlap regression (`filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent` — merge-load → same-key merge-load → delete → keyed point lookup, green only under the vendored lance-table patch — plus its append-only control) |
| `writes.rs` | Direct-publish writes: cancellation, RFC-022 non-strict full-attempt reprepare from fresh branch authority, strict stale-write conflicts, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery); the lance#7444 row-id-overlap regression (`filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent` — merge-load → same-key merge-load → delete → keyed point lookup, green only under the vendored lance-table patch — plus its append-only control) |
| `staged_writes.rs` | TableStore staged-write primitives (`stage_append`, `stage_merge_insert`, `commit_staged`, `scan_with_staged`, `count_rows_with_staged`) — primitive-level only; engine code uses the in-memory `MutationStaging` accumulator instead |
| `forbidden_apis.rs` | Defense-in-depth source-walk guard: engine code (`exec/`, `db/omnigraph/`, `loader/`, `changes/`) must not reach around the sealed storage trait to Lance inline-commit APIs, nor open datasets directly (`Dataset::open` / `DatasetBuilder::from_uri`/`from_namespace`) — reads route through `Snapshot::open` and the held-handle cache; `// forbidden-api-allow: <reason>` sentinel exempts reviewed lines |
| `failpoint_names_guard.rs` | Source-walk guard (same defense-in-depth shape as `forbidden_apis.rs`): every failpoint call site (`maybe_fail`, `ScopedFailPoint::new`/`with_callback`, `Rendezvous::park_first`) must reference a `failpoints::names` const, never a bare string literal — a typo'd literal compiles and silently never fires |
@ -36,7 +36,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
| `changes.rs` | `diff_between` / `diff_commits` |
| `consistency.rs` | Cross-table snapshot isolation, atomic publish |
| `lineage_projection.rs` | RFC-013 Phase 7 acceptance gate: graph lineage lives ONLY in `__manifest` — over a realistic history (main commits, a branch, a merge, actors), the production `CommitGraph::open` projection reconstructs the full DAG (commit set, parents, merge parents + merge actor, per-branch heads, inline actors) from the `graph_commit`/`graph_head` rows, and the `_graph_commits.lance` / `_graph_commit_actors.lance` dataset directories are never created at all |
| `schema_apply.rs` | Migration plan + apply, schema-apply lock; index materialization deferred to the reconciler (iss-848): `apply_schema_defers_vector_index_on_empty_table` (an empty-table Vector `@index` never aborts the apply) and `index_only_constraint_apply_touches_no_table_data` (adding an `@index` is metadata-only — no table-version bump); enum widening (iss-enum-widening-migration): `enum_widening_apply_is_metadata_only_and_accepts_new_variant` (no table-version bump; new variant accepted, out-of-set still rejected) + `enum_narrowing_apply_is_refused` (OG-MF-106 with the graph left writable). The planner's widening/narrowing matrix lives in `schema_plan.rs`'s in-source tests |
| `schema_apply.rs` | Migration plan + apply, schema-apply lock; schema-contract publication is pinned by `read_only_open_holds_schema_gate_through_catalog_capture` and `refresh_holds_schema_gate_through_catalog_publication` (source, accepted IR/state, and compiled catalog are captured under one root schema gate). `long_lived_handle_uses_the_schema_catalog_bound_to_its_write_token` covers mutation/load plus a post-apply new node type merged through the pre-apply handle; `stale_handle_branch_delete_gates_tables_added_by_schema_apply` parks delete over that new type while a legacy index reconciler waits, proving merge planning and native-control table envelopes use an operation-local accepted catalog rather than stale ArcSwap state. Index materialization is deferred to the reconciler (iss-848): `apply_schema_defers_vector_index_on_empty_table` (an empty-table Vector `@index` never aborts the apply) and `index_only_constraint_apply_touches_no_table_data` (adding an `@index` is metadata-only — no table-version bump); enum widening (iss-enum-widening-migration): `enum_widening_apply_is_metadata_only_and_accepts_new_variant` (no table-version bump; new variant accepted, out-of-set still rejected) + `enum_narrowing_apply_is_refused` (OG-MF-106 with the graph left writable). The planner's widening/narrowing matrix lives in `schema_plan.rs`'s in-source tests |
| `search.rs` | FTS / vector / hybrid (`bm25`, `nearest`, `rrf`) |
| `scalar_indexes.rs` | Per-property index dispatch of `build_indices_on_dataset_for_catalog`: enums + orderable scalars get a BTREE (so `=`/range/IN/IS NULL are index-accelerated), free-text Strings keep FTS — observed via `TableStore::key_column_index_coverage`, the same helper the traversal chooser uses |
| `traversal.rs` | `Expand`, variable-length hops, anti-join, undirected traversal (`$a <edge> $b`, `Direction::Both` — out ∪ in with set-semantics dedup, both-direction anti-join) (CSR path — `OMNIGRAPH_TRAVERSAL_MODE` unset) |
@ -51,10 +51,10 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
| `validators.rs` | Schema constraint enforcement (enum, range, unique, cardinality) across JSONL load, mutation insert/update. ALL THREE write surfaces — mutation, bulk load, AND merge — route through the unified `crate::validate` evaluator (Δ-scoped, index-backed, reusing these leaf checks). Cross-version-uniqueness closure: `cross_version_unique_rejected_on_mutation_insert` + `reinsert_existing_key_is_upsert_not_unique_violation` (mutation path); `cross_version_unique_rejected_on_append_load` + `merge_load_reupsert_existing_key_is_not_unique_violation` (load path). Per-table `Overwrite`: `overwrite_load_validates_ri_against_new_image` (an edges-only overwrite still resolves RI against retained committed nodes) + `append_load_rejects_orphan_edge`. The evaluator's own unit tests live in `src/validate.rs` (`#[cfg(test)]`); its merge-conflict equivalence is pinned by `merge_truth_table.rs` (OrphanEdge) + `branching.rs` (Unique/Cardinality merge tests). Intra-batch duplicate-`@key` rejection on every load mode is pinned by `consistency.rs::loader_rejects_intra_batch_duplicate_keys`; the mutation-coalesce counterpart (insert+update / chained updates of one id are NOT a self-collision) by `writes.rs`. Non-String `@unique` columns probe committed state with a TYPED literal (not a stringified key): `cross_version_unique_rejected_on_date_column` + `noncolliding_write_to_date_unique_column_succeeds` (a `Date @unique` collision is a proper `@unique` violation, and a distinct value does not raise a Date32-vs-Utf8 coercion error). Cardinality is keyed by edge id, last-wins (matching commit's `dedupe_merge_batches_by_id`): `merge_load_edge_src_move_rechecks_vacated_src_cardinality` (a Merge-load moving an edge recounts the vacated src for `@card` min) + `merge_load_duplicate_edge_id_counts_once_per_card` (a dup edge id under two srcs in one batch counts once, no spurious max violation). Direct deletes capture the ids they remove (from the delete op's own scan) into the change-set's `deleted_ids`, so a delete emptying a src is validated: `mutation_delete_edge_below_card_min_rejected` (a `delete Edge` dropping a src below `@card` min is rejected, not silently committed). |
| `merge_cost.rs` | Cost-budget tests for branch MERGE on the shared `helpers::cost` harness: `merge_validation_is_delta_scoped` (a 1-row-delta merge opens ≤3 data tables — Δ-scoped, not the whole catalog; was ~6 pre-#5) and `merge_manifest_cost_grows_with_history` (the cross-branch `__manifest` open amplification still grows with commit depth — a separate, not-yet-addressed term — while validation `data_open_count` stays flat) |
| `policy_engine_chassis.rs` | Engine-layer Cedar enforcement (MR-722): allow + deny through every `_as` writer via the SDK directly — no HTTP — proving embedded and CLI callers hit the same gate as the server, with action × scope shapes matching `authorize_request` |
| `maintenance.rs` | `optimize` (compaction), `repair` (explicit uncovered-drift publish), and `cleanup` (version GC): empty/idempotent/no-op edges, policy validation, head preservation; `optimize` publishes its own compaction (`optimize_publishes_compaction_to_manifest_so_schema_apply_succeeds`), skips pre-existing uncovered drift (`optimize_skips_preexisting_manifest_head_drift`), and refuses to run while a `__recovery` sidecar is pending (`optimize_defers_when_recovery_sidecar_is_pending`); `repair` previews/heals verified maintenance drift, refuses raw semantic drift without `--force`, and forced repair publishes only by explicit operator choice; the index reconciler (iss-848): `index_build_tolerates_null_vector_rows` (an untrainable Vector column defers instead of aborting the build, sibling indexes still build) and `optimize_materializes_index_declared_but_unbuilt` (optimize creates a declared-but-deferred index) |
| `failpoints.rs` | Failure-injection coverage (gated on `failpoints` feature). Includes the five per-writer Phase B → recovery integration tests (`recovery_rolls_forward_after_finalize_publisher_failure`, `schema_apply_phase_b_failure_recovered_on_next_open`, `branch_merge_phase_b_failure_recovered_on_next_open`, `ensure_indices_phase_b_failure_recovered_on_next_open`, `optimize_phase_b_failure_recovered_on_next_open`) and the write-entry in-process heal contract (the four `*_after_finalize_publisher_failure_heals_without_reopen` tests — load, mutation, schema apply, branch merge: a follow-up write on the same handle rolls a sidecar-covered residual forward without reopen/refresh) and the storage-fault matrix for the sidecar lifecycle (`recovery.sidecar_{write,delete,list}` / `recovery.record_audit` failpoints: Phase A put failure aborts with zero drift, Phase D delete failure is swallowed and healed by the next write, list failures are loud at heal and open, audit-append failures are retried to exactly one audit row; plus the bucket-gated `s3_load_recovers_after_publisher_failure_without_reopen`). And the convergence-idempotent roll-forward regression (`open_sweep_roll_forward_converges_when_manifest_advances_concurrently`: two concurrent open-sweeps race one sidecar at the `recovery.before_roll_forward_publish` rendezvous; the CAS loser must converge, not fail the open — iss-schema-apply-reopen-recovery-race). |
| `maintenance.rs` | `optimize` (compaction), `repair` (explicit uncovered-drift publish), and `cleanup` (version GC): empty/idempotent/no-op edges, policy validation, head preservation; `optimize` publishes its own compaction (`optimize_publishes_compaction_to_manifest_so_schema_apply_succeeds`), skips pre-existing uncovered drift (`optimize_skips_preexisting_manifest_head_drift`), refuses to run while a `__recovery` sidecar is pending (`optimize_defers_when_recovery_sidecar_is_pending`), and compacts blob-v2 tables through the normal path (`optimize_compacts_blob_table_alongside_plain_table`); `repair` previews/heals verified maintenance drift, refuses raw semantic drift without `--force`, and forced repair publishes only by explicit operator choice; the index reconciler (iss-848): `index_build_tolerates_null_vector_rows` (logical load succeeds without inline index work; an untrainable Vector column then defers during reconciliation) and `optimize_materializes_index_declared_but_unbuilt` (optimize creates a declared-but-deferred index) |
| `failpoints.rs` | Failure-injection coverage (gated on `failpoints` feature). RFC-022 includes deterministic post-stage/pre-effect races for mutation/load uniqueness and strict disjoint-head changes, plus the cross-handle post-effect `RecoveryRequired` → read-write-open rollback cell. Control/recovery race cells include `first_touch_post_create_open_error_keeps_recovery_ownership`, `branch_delete_orphans_sidecar_armed_after_initial_barrier`, `branch_merge_fences_target_delete_recreate_aba`, `branch_merge_fences_concurrent_sync_on_same_handle`, `branch_merge_rejects_fresh_target_manifest_change_before_effects`, `branch_merge_rechecks_late_sidecar_after_table_gates`, `cleanup_rechecks_sidecars_under_gc_gates`, and `full_recovery_rereads_sidecar_body_after_discovery`. The suite also includes the five per-writer effect → manifest-CAS recovery tests, write-entry in-process heal contract, storage-fault matrix, S3 recovery twin, and convergence-idempotent roll-forward regression. |
| `failpoint_names_guard.rs` | Source-walk guard: every `maybe_fail(...)` call site (engine + cluster) must reference the compile-checked `failpoints::names` consts, never a bare string literal — a typo'd literal compiles but silently never fires; same defense-in-depth shape as `forbidden_apis.rs` |
| `recovery.rs` | Open-time recovery sweep — sidecar I/O, classifier dispatch (NoMovement / RolledPastExpected / UnexpectedAtP1 / UnexpectedMultistep / InvariantViolation), all-or-nothing decision, roll-forward via `ManifestBatchPublisher::publish`, roll-back via `Dataset::restore`, audit row in `_graph_commit_recoveries.lance`, `OpenMode::ReadOnly` skip path |
| `recovery.rs` | Open-time recovery sweep — legacy sidecars plus schema-v3 Mutation/Load exact transaction identity, fixed logical/rollback outcome IDs, branch-token comparison, fresh under-gate sidecar reread/reparse, all-or-nothing roll-forward/rollback/refusal, audit row in `_graph_commit_recoveries.lance`, and `OpenMode::ReadOnly` skip path |
| `composite_flow.rs` | Compositional/narrative end-to-end stories — multi-step flows that compose mechanics covered by other test files. Catches integration regressions where individual operations all pass their unit tests but their composition breaks (sequential merges, post-merge main writes, time-travel through merge DAG, reopen consistency over multi-merge histories, post-optimize and post-cleanup strict writes). |
## Fixtures

261
docs/dev/writes.md
View file

@ -4,11 +4,14 @@
> removed in MR-771 (shipped v0.4.0). Writes now go directly to the target
> table; this document specifies that direct-publish path.
`mutate_as` and `load` write **directly to the target table**
and call `ManifestBatchPublisher::publish` once at the end with
`expected_table_versions` (the per-table manifest versions captured before
the first write). Cross-table OCC is enforced inside the publisher; the
publisher's row-level CAS on `__manifest` is the single fence.
`mutate_as` and `load` prepare against one immutable branch-authority token,
write **directly to the target table**, and call
`ManifestBatchPublisher::publish` once at the end. The token is
`(Lance branch identifier, exact optional graph_head, accepted schema identity)`;
the exact `graph_head` check protects validation dependencies on tables the
write does not touch. Publisher row-level CAS on `__manifest` is the visibility
fence. Process-local branch/table queues reduce retries but are not distributed
authority.
## What this means in practice
@ -19,15 +22,93 @@ publisher's row-level CAS on `__manifest` is the single fence.
`__run__*` branch on an upgraded graph is swept off `__manifest` by the
v2→v3 internal-schema migration on first read-write open. (The inert
`_graph_runs.lance` bytes remain until a `delete_prefix` primitive lands.)
- Cancelled mutation futures leave **no graph-visible state** — the manifest
is never advanced. They can leave two kinds of unreferenced residue, both
self-healing: orphaned Lance fragments (reclaimed by `omnigraph cleanup`),
and — on the *first* write to a table on a branch, which forks it before the
publish — a manifest-unreferenced branch ref. The next write to that table
reclaims the stale fork and re-forks (`reclaim_orphaned_fork_and_refork`),
and `cleanup`'s per-table reconciler is the guaranteed backstop; see the
- Cancelled mutation futures leave **no graph-visible state** unless the
manifest publish already completed. Before that point they can leave
reclaimable uncommitted Lance files, or sidecar-covered committed table
effects that the next quiesced recovery rolls forward/compensates. A
first-touch named-branch write can also leave a target table ref, but it is
never created without ownership: the schema-v3 sidecar is durable first and
names that `(table_path, target ref)`. Reclaim and `cleanup` treat any
matching pending sidecar as a hard stop. Quiesced full recovery accepts both
crash shapes — sidecar durable with no ref yet, or an exact untouched ref at
the inherited version — and removes the latter before deleting the empty
intent. If several pending intents claim one ref, a no-effect intent discards
only itself while any competitor remains; the last no-effect survivor cleans
an untouched ref, or the effect-owning survivor recovers normally. `cleanup`
remains the backstop for genuinely unclaimed legacy/stale refs; see the
fork-reclaim note in [invariants.md](invariants.md).
## Mutation/load coarse OCC (RFC-022 first adapter)
Mutation and load use a closed prepare → effect → publish attempt:
1. run the branch-aware recovery barrier, then capture the target branch's native Lance
`BranchIdentifier`, exact `graph_head:<branch>` (including absence on a
fresh branch), accepted schema identity, and table snapshot;
2. run the complete validator and prepare every effect outside the effect gate.
Existing-table transactions may stage reclaimable files here. A first-touch
named-branch table retains its batch/predicate and pre-mints the transaction
identity instead: Lance branch-local files cannot be staged until its target
ref exists;
3. acquire the schema gate, branch gate, then sorted table queues; re-check for
a relevant sidecar armed since step 1, then revalidate the token. Any
unresolved relevant intent returns typed `RecoveryRequired` before effects;
4. on a pre-effect mismatch, discard the complete attempt. Append/Insert/Merge
reprepare with a bounded retry; strict Update/Delete/Overwrite return typed
`ReadSetChanged`;
5. arm a schema-v3 recovery sidecar. For each deferred first-touch table, create
its target ref, stage branch-local files on that ref, and bind the staged
transaction to the pre-minted UUID. Then commit every planned transaction
with zero transparent conflict retries, confirm exact transaction UUIDs and
table updates, and publish the pre-minted lineage intent under the same token.
The publisher checks the exact head and native branch identity on every CAS
attempt. It never reparents a validation-sensitive intent after contention. A
mismatch after any physical effect returns `RecoveryRequired` and leaves the
sidecar intact; it is not an ordinary retry loop.
This adapter preserves the documented single-writer-process support boundary.
The native branch identifier detects delete/recreate ABA but is not a Lance
conditional-ref fence, and destructive recovery remains unsafe beside a live
foreign process.
### Branch-merge authority fence (adapter bridge)
Branch merge still uses its writer-specific multi-commit table effects and
confirmation sidecar; it has not yet been converted to the RFC-022 exact-effect
adapter. It does, however, join the closed control boundary needed by this first
slice: after the strict recovery barrier it acquires the root-shared schema gate
and the sorted source/target branch gates, performs the final sidecar check,
loads one operation-local catalog from the accepted contract, captures both graph
heads plus the base/source/target snapshots, and holds those gates through table
effects and manifest publication. Planning stays outside table queues. Before
Phase A, merge acquires the conservative all-catalog table envelope for both
source and target, re-lists sidecars, and compares fresh source/target manifest
versions with the captured snapshots. A stale warm handle catalog or coordinator
snapshot is never accepted as that revalidation.
That fence prevents a same-process target delete/recreate from reusing the branch
name underneath a merge plan. The race test deliberately recreates a target with
the same name and numeric Lance version but a different `BranchIdentifier`, so
version-only checking cannot accidentally satisfy it. This is a process-local
bridge, not a cross-process conditional-ref primitive and not a substitute for
the later full branch-merge read-set/reprepare adapter. `sync_branch` joins the
same root schema gate before replacing a handle's coordinator, so it cannot
overwrite merge's temporary target coordinator or change a native control's
active-branch authority mid-operation.
### Branch-delete orphaning exception
Branch deletion runs the healer first and then holds schema, the target branch,
and every accepted-catalog table gate through the native ref removal. An
unresolved sidecar scoped to that target does not permanently block deletion:
once those gates prove its in-process owner is no longer live, removing the
manifest branch makes its physical effects unreachable. The next write/open
records the orphan-discard recovery audit and deletes the sidecar. A
`SchemaApply` sidecar remains graph-global and blocks deletion. This exception
is specific to removing the authority that made the intent reachable; create,
merge, mutation, and load still reject relevant unresolved ownership.
## Read-your-writes within a multi-statement mutation
A `.gq` query with multiple ops (e.g. `insert Person … insert Knows …`)
@ -47,12 +128,20 @@ shared by both `mutate_as` and the bulk loader:
`TableStore::scan_with_pending`, which scans committed via Lance
and applies the same SQL filter to the pending batches via DataFusion
`MemTable`. Same-query writes are visible to subsequent reads.
- At end-of-query, `MutationStaging::finalize` issues exactly one
`stage_*` + `commit_staged` per touched table (concatenating
accumulated batches; merge-mode dedupes by `id`, last-write-wins),
and the publisher publishes the manifest atomically across all
touched sub-tables. Cross-table conflicts surface as
`ManifestConflictDetails::ExpectedVersionMismatch`.
- Blob-bearing updates use the materializing variant: Lance reads only matched
committed blob payloads as binary, the engine normalizes them back to the
logical blob schema, and the full rows join the same pending-shadow union.
Rewriting matched blob bytes costs I/O proportional to those bytes, but makes
correctness independent of whether a physical index selects a different
Lance merge plan.
- At end-of-query, `MutationStaging::stage_all` prepares exactly one staged
transaction per touched table and `commit_all` commits it (concatenating accumulated
batches; merge-mode dedupes by `id`, last-write-wins), and the publisher
publishes the manifest atomically across all touched sub-tables. Existing
tables stage before gate acquisition; a first-touch named-branch table stages
after sidecar + fork under the gates so its uncommitted files live in the
correct Lance branch tree. Cross-table conflicts surface as typed read-set or
manifest conflicts.
- **Deletes stage too (MR-A).** Lance 7.0's
`DeleteBuilder::execute_uncommitted` (#6658) makes delete a two-phase op,
so deletes no longer inline-commit. Each delete records a predicate in
@ -153,9 +242,9 @@ integrity, and edge cardinality before any Lance HEAD movement, stages
each touched table with Lance `Operation::Overwrite`, then runs
`commit_staged` under the normal `SidecarKind::Load` recovery sidecar
before publishing `__manifest`. `OMNIGRAPH_LOAD_CONCURRENCY` applies to the
fragment-writing stage only; the commit and manifest publish still run
under the per-table write queues. Empty-table overwrite is represented as
a valid zero-fragment Lance `Overwrite` transaction, not as
fragment-writing stage only; the commit and manifest publish run while holding
the root-shared schema → branch → sorted-table gates. Empty-table overwrite is
represented as a valid zero-fragment Lance `Overwrite` transaction, not as
truncate-then-append.
### Open-time recovery sweep
@ -170,8 +259,8 @@ test pins.
A second, narrower drift class — the **finalize → publisher window** —
is closed across one open cycle by the open-time recovery sweep:
`MutationStaging::finalize` runs `stage_*` + `commit_staged` per touched
table sequentially, then the publisher commits the manifest. Lance has
`MutationStaging::stage_all` prepares the table transactions and `commit_all`
runs their independent HEAD advances before the publisher commits the manifest. Lance has
no multi-dataset atomic commit, so the per-table `commit_staged` calls
are independent operations: if commit_staged on table N+1 fails *after*
commit_staged on tables 1..N succeeded, or if the publisher's CAS
@ -179,30 +268,37 @@ pre-check rejects *after* every commit_staged succeeded, tables 1..N
are left at `Lance HEAD = manifest_pinned + 1`.
**Recovery protocol** (lifecycle of every staged-write writer —
`MutationStaging::finalize`, `schema_apply::apply_schema_with_lock`,
`MutationStaging::commit_all`, `schema_apply::apply_schema_with_lock`,
`branch_merge_on_current_target`, `ensure_indices_for_branch`,
`optimize_all_tables`):
1. **Phase A**: writer writes a sidecar JSON to
`__recovery/{ulid}.json` BEFORE its first HEAD-advancing commit
`__recovery/{ulid}.json` BEFORE its first independently durable physical
effect (including a first-touch Lance branch ref) or HEAD-advancing commit
(`commit_staged`, or `compact_files` for `optimize_all_tables`,
which advances the Lance HEAD via a reserve-fragments + rewrite
commit rather than a staged write). The
sidecar names every `(table_key, table_path, expected_version,
post_commit_pin)` it intends to commit + the writer kind +
actor_id.
For a first-touch named-branch Mutation/Load table, Phase A is followed by
target-ref creation and branch-local `stage_*`; the sidecar already carries
its pre-minted transaction identity.
2. **Phase B**: writer's per-table `commit_staged` loop runs.
- **Phase-B confirmation (`BranchMerge` only)**: a `BranchMerge` writer
- **Phase-B confirmation:** a `BranchMerge` writer
advances each table's HEAD by *several* commits (append → upsert →
delete), so a bare "HEAD moved" is ambiguous — it could be a complete
publish or one crashed mid-sequence. After the whole per-table loop
finishes, the writer re-writes the sidecar stamping each pin's
`confirmed_version` with the exact achieved version, then proceeds to
Phase C. This is the commit point of the recovery WAL: a crash *after*
confirmation rolls forward to those versions; a crash *during* Phase B
(sidecar still unconfirmed) rolls back. Other writers don't confirm —
their drift is derived state (index coverage, compaction) that a partial
roll-forward never corrupts.
Phase C. Schema-v3 Mutation/Load sidecars also confirm: each table must
match the staged Lance transaction's `(read_version, uuid)`, and the
sidecar records the exact `SubTableUpdate` plus original lineage intent.
This is the commit point of the recovery WAL: a crash *after* confirmation
rolls forward only when the captured branch token still matches; a crash
*during* Phase B (sidecar still unconfirmed) rolls back. Remaining legacy
writers don't confirm — their drift is derived state (index coverage,
compaction) that a partial roll-forward never corrupts.
3. **Phase C**: publisher commits the manifest.
4. **Phase D**: writer deletes the sidecar.
@ -226,6 +322,18 @@ recovery sweep in `crates/omnigraph/src/db/manifest/recovery.rs`:
`BranchMerge` sidecar, a moved HEAD with no `confirmed_version` classifies
as `IncompletePhaseB` (a partial multi-commit publish) and forces roll-back;
with a `confirmed_version`, roll-forward targets exactly that version.
Schema-v3 Mutation/Load additionally requires `EffectsConfirmed`, the exact
Lance transaction identity at the confirmed version, the original immutable
manifest delta, and a matching captured authority token. A changed token is
rollback-only; an unknown/foreign effect is refused rather than adopted.
An Armed first-touch intent with no owned transaction is deferred by live
roll-forward-only healing because another handle may still own it. Quiesced
full recovery tolerates an absent target ref (crash before fork), or removes
an exact unchanged unpublished ref after proving no other pending sidecar
claims it. With competing claims, the current no-effect sidecar discards
itself without touching the ref; the final survivor owns cleanup/recovery.
During partial rollback, no-effect refs are removed before the rollback
outcome is published so a retry cannot strand them.
- If any table is `InvariantViolation` (Lance HEAD < manifest pinned —
should be impossible), **abort** with a loud error and leave the
sidecar on disk for operator review.
@ -243,14 +351,14 @@ recovery sweep in `crates/omnigraph/src/db/manifest/recovery.rs`:
each iteration. The audit row's `to_version` records the logical
rolled-back-to version (`manifest_pinned`); the manifest is published at the
restore commit (`manifest_pinned + 1`, same content).
- After a successful roll-forward or roll-back, an audit row is
recorded — the graph commit lineage (the `graph_commit` rows in `__manifest`
since RFC-013 Phase 7) carries a commit tagged
`actor_id = "omnigraph:recovery"`, and a sibling
`_graph_commit_recoveries.lance` row carries `recovery_kind`,
`recovery_for_actor` (the original sidecar's actor), `operation_id`,
per-table outcomes. Operators run `omnigraph commit list --filter
actor=omnigraph:recovery` to find recoveries.
- After a successful roll-forward or roll-back, an internal
`_graph_commit_recoveries.lance` row records `recovery_kind`,
`recovery_for_actor` (the original sidecar's actor), `operation_id`, and
exact per-table outcomes. A v3 roll-forward publishes the interrupted
writer's fixed lineage intent, including its original actor; rollback and
legacy recovery commits use `actor_id = "omnigraph:recovery"`. Ordinary
commit history is therefore not a complete recovery enumeration, and the
CLI currently has no public query for the recovery-audit table.
- Sidecar deleted as the final step.
Triggers for the residual: transient Lance write errors during finalize
@ -264,13 +372,24 @@ contention exceeding `PUBLISHER_RETRY_BUDGET = 5` retries.
Phase B → Phase C residual closes on the next write, without a
restart and without an explicit refresh. The heal lists `__recovery/`
(one `list_dir`; empty in the steady state) and, per sidecar, acquires
the same per-`(table_key, table_branch)` write queues every sidecar
writer holds from before `write_sidecar` until after `delete_sidecar` —
so it serializes against a live writer instead of rolling its
schema → branch → sorted-table gates that overlap the writer's guarded
sidecar lifetime. RFC-022 mutation/load writers hold the complete order. Branch
merge now holds schema plus source/target branch authority for its whole attempt
and then the all-catalog source/target table envelope; other legacy adapters
serialize through their existing table or schema gate until their own adapter
slices land. The
manager is shared by every
`Omnigraph` handle for one canonical local root identity (relative, absolute,
and symlink aliases converge; object-store/custom schemes stay opaque), so this
also serializes a refresh or separately-opened handle against a live writer instead of rolling its
in-flight sidecar forward from under it (a sidecar whose queues can be
acquired belongs to a writer that finished or died; an existence
re-check after the wait skips the finished case). Lock order is
queues → coordinator, matching every writer's commit→publish path.
schema → branch → sorted tables → coordinator, matching the writer effect path.
Enrolled mutation/load attempts and branch merge perform one additional
`list_dir` after acquiring their authority gates; that final check closes the
pre-gate recovery TOCTOU without moving mutation/load validation or staged-file
construction under the gate.
Pinned by the four
`tests/failpoints.rs::*_after_finalize_publisher_failure_heals_without_reopen`
tests (load, mutation, schema apply, branch merge). The maintenance
@ -280,8 +399,11 @@ Phase-B commit (dropping its rows), and a branch merge publishes the
drift as an unattributed side effect — both while the stale sidecar
lingers to misclassify later.
Sidecars that would require a `Dataset::restore` (mixed / unexpected
state) are deferred to the next `OpenMode::ReadWrite` open: restore is
unsafe under concurrency because Lance's `check_restore_txn` accepts
state) are deferred to the next `OpenMode::ReadWrite` open. Full open-time
recovery uses the same root-scoped ordered gates and post-wait existence check,
so it cannot Restore/delete under a live writer owned by another handle in the
same process. Restore remains unsafe across processes because Lance's
`check_restore_txn` accepts
the restore against in-flight Append/Update/Delete commits and
silently orphans them (pinned by
`tests/staged_writes.rs::lance_restore_loses_to_concurrent_append_via_orphaning`).
@ -289,17 +411,24 @@ When such a deferred sidecar blocks a write, the commit-time drift
guard says so explicitly ("a pending recovery sidecar requires
rollback — reopen the graph read-write") instead of pointing at
`omnigraph repair`, which refuses while a sidecar is pending.
`cleanup` refuses pending sidecars at entry as well, before orphan reconciliation
or version GC: v3 ownership and compensation recovery may need the retained
Lance transaction/version history, so garbage collection cannot outrun the
recovery barrier.
Continuous in-process recovery for the rollback path is the goal of a
future background reconciler. `ensure_indices` does not heal at entry
itself — it runs inside the load / schema-apply flows after their
entry heal, and its strict preconditions still fail loudly on drift
when invoked directly.
future background reconciler. `ensure_indices` does not heal at entry itself;
it is an explicit maintenance/reconciliation call, separate from mutation,
load, and schema apply, and its strict preconditions fail loudly on drift.
The publisher-CAS contract is unchanged: a *concurrent writer* that
advances any of our touched tables between snapshot capture and
publisher commit produces exactly one winner. The residual above is
about *our* abandoned commits in the failure path, not about
concurrency races.
For enrolled mutation/load, the publisher rechecks the attempt's exact native
branch identity and `graph_head` as well as the touched-table versions. A
concurrent graph commit anywhere on the target branch therefore invalidates the
prepared authority instead of silently reparenting it. Before effects, an
insert-only mutation or Append/Merge load fully reprepares with a bounded retry; strict
Update/Delete/Overwrite returns `ReadSetChanged`; after any effect, any later
error returns `RecoveryRequired` and leaves the fixed v3 intent durable. Legacy
writers still arbitrate only their explicit touched-table expectations until
their adapters are enrolled.
**Sidecar I/O failure semantics** (all sidecar I/O goes through the
backend-generic `StorageAdapter`; the contracts below are pinned by the
@ -341,19 +470,25 @@ error, not a silent `false`.
## Conflict shape
Concurrent writers to the same `(table, branch)` produce exactly one
success and one failure. The losing writer's error is
`OmniError::Manifest` with kind `Conflict` and details
`ManifestConflictDetails::ExpectedVersionMismatch { table_key, expected,
actual }`. The HTTP server maps this to **409 Conflict** with body
`{"error": "...", "code": "conflict", "manifest_conflict": { "table_key":
"...", "expected": N, "actual": M }}` — see [docs/user/server.md](../user/operations/server.md).
For mutation/load, a changed authority detected before effects is
`ManifestConflictDetails::ReadSetChanged { member, expected, actual }`.
Retryable Insert/Merge/Append attempts handle this internally by fully
repreparing; strict writes surface **409 Conflict** with structured
`read_set_conflict` details. A changed authority discovered after a physical
effect, or any unresolved overlapping intent found at the synchronous recovery
barrier, is `OmniError::RecoveryRequired { operation_id, … }`, mapped to **503
Service Unavailable** with structured `recovery_required`; retry only after the
sidecar has been resolved. Legacy, not-yet-enrolled writers may still surface
`ExpectedVersionMismatch` and `manifest_conflict`.
## Audit
## Commit actor history
`actor_id` lands in the graph commit lineage — the `graph_commit` rows in
`__manifest`, written in the publish CAS (RFC-013 Phase 7; previously
`_graph_commits.lance`). Audit history is queried via `omnigraph commit list`.
`_graph_commits.lance`). Ordinary commit/actor history is queried via
`omnigraph commit list`. Crash-recovery actions additionally live in the internal
`_graph_commit_recoveries.lance` table described above; that exact recovery log
does not yet have a public CLI query.
## Storage versioning (no in-place migration)

26
docs/rfcs/rfc-0018-ingest-wal.md
View file

@ -2,7 +2,7 @@
type: spec
title: "RFC-018 — Streaming-ingest WAL on Lance MemWAL"
description: Adds a durability-first streaming ingest path (ack on WAL durability, asynchronous fold into the graph commit chain) built entirely on Lance's MemWAL primitive; reconciled against Lance v8.0.0 and the v9 beta line; analyzed for composition with the upstream multi-table-commit RFCs.
status: draft
status: superseded
tags: [eng, rfc, wal, ingest, lance, omnigraph]
timestamp: 2026-07-02
owner:
@ -10,12 +10,19 @@ owner:
# RFC-018 — Streaming-ingest WAL on Lance MemWAL
**Status:** Draft / for discussion
**Status:** Superseded by [RFC-026](rfc-026-memwal-streaming-ingest.md)
**Date:** 2026-07-02
**Surveyed version:** 0.7.2 (branch `dst-extract-crate`); Lance pinned at 7.0.0
**Upstream surveyed:** Lance v8.0.0 (released; RC votes closed 2026-07-01), v9.0.0-beta.10; MemWAL spec (`lance.org/format/table/mem_wal/`, fetched in full 2026-07-02); discussions #7260, #7264, #7222, #7176
**Audience:** OmniGraph maintainers
> **Supersession note (2026-07-10):** RFC-026 carries the streaming-ingest
> design forward under RFC-022's unified graph-write protocol. It also corrects
> this draft's characterization of MemWAL: MemWAL is a strategic Lance
> architecture and a major substrate investment, not an experimental direction.
> The integration risk is its evolving API and format surface across Lance
> releases.
---
## 0. TL;DR
@ -125,8 +132,10 @@ fencing primitive.
## 3. Substrate inventory — what Lance provides and how we use all of it
Per the lance.md protocol the MemWAL spec and adjacent pages were fetched in
full (2026-07-02). The spec is **experimental** upstream — risk register in
§10. Inventory, mapped to consumption:
full (2026-07-02). MemWAL is a strategic Lance architecture with substantial
upstream investment. Its API and format surface are still evolving across
releases; §10 treats that maturity boundary as an integration risk. Inventory,
mapped to consumption:
| Lance tooling | Spec/PR | How this RFC uses it |
|---|---|---|
@ -323,8 +332,8 @@ this RFC phases on:
matures it may fit *small in-place updates* better than WAL-upsert-fold;
watch-listed as a possible Phase 4 refinement, not a dependency.
- MemWAL fixes keep landing on v9 betas (#7489 cross-generation block-list on
in-memory scan arms) — confirming the experimental-spec churn risk (§10)
and the value of keeping our exposure transient-state-only.
in-memory scan arms) — confirming that its API/format integration surface is
still moving (§10) and the value of keeping our exposure transient-state-only.
## 7. Composition with upcoming Lance multi-table commits
@ -464,8 +473,9 @@ participates in publication authority.
## 10. Risks
- **MemWAL is experimental upstream** (spec banner; live format votes —
#7418 Status field mid-2026-06). Mitigation is structural: WAL state is
- **MemWAL's API and format surface continues to evolve.** This risk concerns beta-era
API churn, not architectural commitment: Lance has made MemWAL a strategic
streaming-write investment. Mitigation is structural: WAL state is
*transient* (folded then GC'd), so a format change between Lance versions
can be handled by fold-to-quiescent before the bump; no long-lived on-disk
state depends on the MemWAL format. This must stay true — resist any

13
docs/rfcs/rfc-0019-heads-and-fences.md
View file

@ -2,7 +2,7 @@
type: spec
title: "RFC-019 — Heads and Fences: structural O(1) writes without a warm-cache truth fork"
description: Replaces the warm-publish/pinned-open machinery of PR #318 with two structural changes — durable per-table head rows ("refs, not replay") and substrate-native key-conflict fencing (Lance's unenforced-PK KeyExistenceFilter) — landing together as internal schema v5; composes with RFC-018 (WAL ingest) and the upstream multi-table-commit direction.
status: draft
status: superseded
tags: [eng, rfc, write-path, manifest, lance, omnigraph]
timestamp: 2026-07-04
owner:
@ -10,12 +10,21 @@ owner:
# RFC-019 — Heads and Fences
**Status:** Draft / for discussion
**Status:** Superseded by [RFC-023](rfc-023-key-conflict-fencing.md) and [RFC-024](rfc-024-durable-table-heads.md)
**Date:** 2026-07-04
**Surveyed:** omnigraph `main` @ 98530a0e (0.8.0); Lance pinned 7.0.0 (+ vendored lance-table carrying lance#7480); upstream Lance v8.0.0 (released 2026-07-01), v9.0.0-beta.15; PR #318 at `2aab48ba` (reviewed 2026-07-04, 8 verified findings)
**Companion docs:** RFC-018 (streaming-ingest WAL), PR #318's plan doc (`unlimited-history-latency-plan.md`, whose §9 "U2" this RFC promotes from follow-up to prerequisite)
**Audience:** OmniGraph maintainers
> **Supersession note (2026-07-10):** key-conflict fencing and durable table
> heads are separate irreversible decisions with different substrate gates and
> rollout requirements. RFC-023 and RFC-024 review them independently under
> RFC-022's shared graph-write protocol. RFC-023 also corrects this draft's
> load-bearing symmetry claim: on the surveyed Lance revision, an unfiltered
> current transaction (including bare `Append`) can rebase after a filtered
> update. Fencing therefore requires both transaction orders plus a fleet
> compatibility barrier; it is not “unblocked for new tables” in a mixed graph.
---
## 0. TL;DR

773
docs/rfcs/rfc-022-unified-write-path.md Normal file
View file

@ -0,0 +1,773 @@
---
type: spec
title: "RFC-022 — Unified graph-write protocol"
description: One correctness protocol for graph-visible writes, with synchronous recovery, complete read-set arbitration, writer-specific physical-effect adapters, and explicit control-plane exceptions.
status: draft
tags: [eng, rfc, write-path, manifest, recovery, concurrency, lance, omnigraph]
timestamp: 2026-07-10
owner:
---
# RFC-022: Unified graph-write protocol
**Status:** Draft / for team review
**Date:** 2026-07-10
**Surveyed:** OmniGraph 0.8.1 (`main`); Lance 9.0.0-beta.15, git rev `f24e42c1`
**Audience:** engine and storage maintainers
**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md).
Findings marked **BLOCKER** must be dispositioned before acceptance.
---
## 0. Summary
OmniGraph will have one **correctness protocol** for every operation that changes
manifest-resolved graph state. The protocol does not require every writer to use
the same Lance primitive. A mutation can commit one staged transaction per table,
a branch merge can make several commits to a table, and optimize can use Lance
operations that have no staged API. Writer-specific **effect adapters** describe
those physical operations; one coordinator enforces the common safety rules.
A graph-visible write follows this state machine:
```text
recovery barrier
→ prepare pinned base + complete ReadSet + Effects
→ acquire ordered process-local gates
→ revalidate the complete ReadSet, or restart
→ durably arm recovery
→ apply writer-specific physical effects
→ publish exactly one graph-visible __manifest CAS
→ finalize derived state
```
The protocol has three hard boundaries:
1. Recovery is a synchronous pre-write safety barrier. It may also run in the
background, but a writer never proceeds merely because recovery was scheduled.
2. Validation and merge classification are valid only for their complete read set.
A changed probed table causes revalidation or a full restart; the writer never
refreshes expected versions underneath a plan computed from an older base.
3. “One `__manifest` CAS” applies only to graph-visible commits. Native graph-branch
ref creation/deletion and physical-only maintenance have explicit, smaller
control protocols and do not manufacture graph commits.
This RFC deliberately does not combine key fencing, durable table heads,
checkpoint retention, MemWAL ingest, or lineage-based merge-delta discovery into
one format and rollout. They are focused follow-ups:
- [RFC-023 — Key-conflict fencing](rfc-023-key-conflict-fencing.md)
- [RFC-024 — Durable table heads](rfc-024-durable-table-heads.md)
- [RFC-025 — Checkpoint retention](rfc-025-checkpoint-retention.md)
- [RFC-026 — MemWAL streaming ingest](rfc-026-memwal-streaming-ingest.md)
- [RFC-027 — Lineage merge deltas](rfc-027-lineage-merge-deltas.md)
## 1. Scope and authority
### 1.1 Graph-visible writes
A **graph-visible write** changes state resolved through a graph manifest snapshot:
- a node or edge table version;
- a registered or tombstoned table;
- accepted schema identity or schema-visible table metadata;
- a graph commit or graph head;
- any future logical marker that changes query or time-travel semantics.
For such an operation, the only visibility point is one successful `__manifest`
commit containing the entire graph delta. Per-table Lance commits before that point
are physical effects covered by recovery; they are not independently graph-visible.
### 1.2 Control operations
Two classes are not graph-visible commits:
1. Native graph-branch ref create/delete. Lance stores these refs outside the
dataset-version chain and creates no new `__manifest` version for either action.
2. Physical-only maintenance whose result is content-equivalent and is not selected
through a data-table pointer in `__manifest`, such as compacting `__manifest`
itself or reclaiming unreachable files.
Sections 7 and 8 define their control protocols. Branch **merge** and a data-table
optimize whose new version must be published are graph-visible writes and remain in
the main protocol.
### 1.3 What “unified” means
Unified means one set of safety obligations, one coordinator state machine, and one
closed registry of effect adapters. It does **not** mean:
- one storage trait method for every Lance operation;
- exactly one physical commit per table;
- pretending native refs are manifest rows;
- treating process-local queues as a distributed transaction manager;
- moving commit recovery out of the correctness path.
## 2. Protocol objects
The names below are conceptual. Implementations may choose different Rust names,
but they must preserve the represented information and transitions.
```rust
struct PreparedWrite {
operation_id: OperationId,
writer_kind: WriterKind,
target: BranchTarget,
base: BaseView,
read_set: ReadSet,
effects: Effects,
manifest_delta: ManifestDelta,
lineage_intent: Option<LineageIntent>,
recovery: RecoveryPlan,
}
```
### 2.1 `BaseView`
`BaseView` is the immutable state against which the operation was computed. It
contains at least:
- target manifest branch and its incarnation/freshness token;
- pinned manifest version;
- accepted schema identity;
- pinned table entries used by the operation;
- graph head when parentage, merge base, or branch semantics depend on it.
The base is captured only after the recovery barrier completes. A recovery pass may
advance the manifest or promote schema state, so a base captured before recovery is
not valid write input.
### 2.2 `ReadSet`
`ReadSet` is every authority value whose stability is required for the prepared
result to remain correct. It includes, as applicable:
- target branch incarnation;
- accepted schema identity;
- graph head;
- the table head for every table written;
- the table head for every table probed by uniqueness, referential-integrity,
cardinality, policy-independent structural validation, or merge classification;
- writer-specific authority, such as an enrolled stream's configuration and merge
generation.
A table belongs in the read set because its value affected the decision, not because
the writer happens to update it. Read-only dependencies are load-bearing.
The publisher must arbitrate every read-set member atomically with the manifest
commit. A fresh pre-check followed by an unconditional write is not a CAS. The
implementation must ensure that a concurrent change after the check contends on a
stable authority row or equivalent substrate token. If the current representation
cannot arbitrate a read-only dependency, that writer has not completed this RFC and
must not ship the outside-gate validation optimization.
### 2.3 `Effects`
`Effects` is an adapter-owned physical plan. Each effect declares:
- physical dataset and Lance branch/ref it targets;
- expected pre-state;
- whether it is staged, inline, ref-only, or zero-commit;
- the possible post-state shape: exact version, bounded range, or adapter-confirmed
version;
- whether rollback is safe;
- how recovery distinguishes no movement, partial movement, completed movement,
and an already-published result.
The generic coordinator never assumes `expected_version + 1`. That assumption is
valid for some mutation/load effects and false for branch merge, compaction, index
work, schema metadata changes, and no-op plans.
### 2.4 `ManifestDelta`
`ManifestDelta` is the complete logical result to publish. It contains table-version
journal entries, registrations/tombstones, mutable logical heads when present, schema
identity, and lineage rows as required by the operation. Its logical contents are
immutable for one prepared attempt. Physical version fields may be declared output
slots that the adapter binds to an allowed, confirmed effect result; binding such a
slot cannot widen or otherwise change the logical plan. A revalidation mismatch
discards the delta and restarts preparation.
### 2.5 `RecoveryPlan`
`RecoveryPlan` supplies the writer-specific classifier and compensation/roll-forward
rules. It must be serializable into a recovery sidecar before the first independently
durable physical effect. It is part of the commit protocol, not a best-effort
maintenance hint.
## 3. Normative invariants
1. **Recovery before base capture.** Every graph-visible writer runs and awaits the
recovery barrier before pinning `BaseView`.
2. **No durable effect before durable intent.** On an existing physical ref,
reclaimable uncommitted files may be staged first. A first-touch named-table
transaction is different: Lance writes its uncommitted files into the opened
branch tree, so Prepare retains the logical batch/predicate and pre-mints its
transaction identity; after revalidation the recovery sidecar becomes durable,
then the writer creates the target ref and stages those branch-local files under
the held gates. In every case the sidecar precedes the first independently
persistent physical effect, including a native table ref, data-table HEAD
advance, or native tag/index mutation needed by the graph-visible result.
3. **Complete read-set validation.** Physical effects may start only while the fresh
authority state equals the complete `ReadSet` used to prepare the operation.
4. **Recompute, do not patch.** On a pre-effect mismatch, discard the prepared plan
and rerun validation/classification. Never refresh expected versions beneath an
old plan.
5. **One graph visibility point.** A graph-visible write publishes one manifest CAS;
no subset of its table effects becomes graph-visible independently.
6. **Adapters own effect truth.** Every effect-producing writer uses a registered
adapter. There is no generic fallback that guesses version movement or recovery
safety.
7. **Queues are local optimization.** Ordered process-local gates prevent avoidable
in-process races and deadlocks. Cross-process correctness comes from Lance
conflicts, manifest arbitration, and recovery.
8. **Recovery safety is synchronous.** Background sweeping is permitted, but a later
overlapping writer waits for, performs, or fails on unresolved recovery before it
can advance state.
9. **Derived work follows visibility.** Expensive index reconciliation, cache warming,
and orphan reclaim may run asynchronously only when logical correctness does not
depend on their completion.
These refine, and do not weaken, invariants 2, 3, 4, 5, 7, 9, 13, and 15 in
[`docs/dev/invariants.md`](../dev/invariants.md).
## 4. The graph-write state machine
### 4.1 Stage A — recovery barrier
Before accepting a base, the coordinator discovers pending recovery intents that can
overlap this operation. It must reach one of three outcomes:
1. all relevant intents are fully resolved;
2. each remaining intent is proven already satisfied and can be finalized safely;
3. the write fails with a typed recovery-required or live-writer-contention error.
“Spawn recovery and continue” is not an outcome. Listing sidecars may run concurrently
with non-authoritative I/O, but preparation cannot accept its base until recovery has
completed and the post-recovery schema/manifest state is available.
Recovery must not destructively act on a sidecar that may belong to a live writer
without ownership/fencing proof. It waits, returns contention, or uses a protocol that
proves the prior owner cannot continue.
Recovery classifiers use exact effect identity or confirmed post-state. A numeric
test such as `manifest_version >= observed_lance_head` is sufficient only when the
adapter independently proves lineage containment; version ordering alone is not that
proof. Exact identity cuts both ways: a roll-forward that loses its manifest CAS to
a concurrent writer which already published this intent's exact goal state is
**convergence, not failure** — the barrier records the audit outcome and removes the
intent rather than failing the write (preserving the concurrent-advance convergence
behavior fixed in #296). "Exact" forbids adopting an unrelated newer version; it does
not forbid recognizing one's own goal already achieved.
The barrier is a deliberate availability trade, stated plainly: an unresolvable
overlapping recovery intent — including a live foreign writer's sidecar that cannot
be fenced — fails every overlapping write with a typed error until resolved. Safety
outranks availability here by design; operators observe the condition through the
typed error and recovery telemetry rather than through silently degraded writes.
### 4.2 Stage B — prepare
Prepare runs without writer gates. It may:
- capture `BaseView`;
- evaluate the complete constraint set;
- classify a merge;
- compute embeddings or other deterministic payloads;
- build staged Lance transactions and reclaimable uncommitted objects for targets
whose physical refs already exist;
- for a first-touch named-table target, retain the complete logical stage input and
pre-mint its recovery transaction identity without creating the ref;
- construct the complete `ReadSet`, `Effects`, `ManifestDelta`, and `RecoveryPlan`.
Prepare must not advance a Lance HEAD or the graph's native branch refs. Staged files
that are not referenced by a committed Lance manifest are permitted and are
reclaimable if the attempt restarts.
After Stage E arms recovery, a first-touch adapter may create its declared target ref
under the held gates, stage branch-local files on that ref, bind the resulting Lance
transaction to the pre-minted identity, and only then advance HEAD. The ref is itself
an independently durable effect covered by the sidecar; recovery must classify both
sidecar-before-ref and ref-before-HEAD crash states.
Every validator registers its actual committed-state probes in `ReadSet`. Key fences
are an additional same-key conflict signal; they do not replace read-set arbitration
for non-key uniqueness, RI, cardinality, or merge-target stability.
### 4.3 Stage C — acquire ordered gates
The prepared effects declare their gate keys. One attempt acquires the complete set in
this total order:
1. durable global operation claims (migration, retention, or future claims) by
deterministic claim key;
2. graph/schema-control key, if required;
3. target branch-control key, if required;
4. `(physical_branch, table_key)` keys in deterministic sorted order.
The gates are held through revalidation, recovery arming, physical effects, and the
manifest visibility decision. They may be released after a successful manifest CAS or
after a failed post-effect attempt has safely left its sidecar for recovery.
Acquiring a global claim is coordination, not a graph effect for §3's sidecar-order
rule: the claim record must itself contain an owner/fencing token and an explicit
crash-release/takeover contract. No data HEAD, tag, index, or logical authority may
move merely because the claim was acquired.
The merge-exclusive mutex may protect a coordinator swap, but it is not a semantic
cross-process lock and must not substitute for a target read set.
### 4.4 Stage D — revalidate or restart
With all gates held, the coordinator loads fresh authority state and compares every
member of `ReadSet`.
- If all members match, the attempt may arm recovery.
- If any member differs, no physical effect may run. The attempt releases its gates,
discards staged state, and restarts from Prepare.
- A strict API may return a typed conflict instead of retrying, but it may not publish
the stale plan.
Retries are bounded and observable. Retrying a merge means recomputing the merge base
and reclassifying against the new target. Retrying a validation-sensitive mutation
means rerunning the validators, including probes of tables the mutation does not
write.
### 4.5 Stage E — arm recovery
After successful revalidation, write the recovery sidecar durably before the first
independently durable physical effect. An effect-free or authority-first workflow
described in §4.9 may omit the sidecar. Otherwise the sidecar contains at least:
- operation id, writer kind, actor, and target branch/incarnation;
- pinned schema identity and complete read set;
- every physical target and expected pre-state;
- adapter recovery strategy;
- intended manifest delta and lineage intent, or a durable reference to them;
- confirmed post-state once the adapter reaches its all-effects-complete boundary.
A multi-step adapter first records its pre-state plan. After all its physical effects
finish, it durably records the exact confirmed post-state before manifest publish.
Until that confirmation exists, recovery treats the effect set as possibly partial.
### 4.6 Stage F — apply effects
The adapter applies its declared physical effects. A failure after recovery is armed
leaves the sidecar intact. The request must not delete it, silently adopt live HEAD, or
start a fresh plan around it.
The adapter returns exact achieved state to bind the physical output slots declared by
`ManifestDelta`. If achieved state differs from the prepared effect envelope, the
operation fails into recovery rather than widening its plan in place.
### 4.7 Stage G — manifest CAS
A graph-visible operation performs exactly one `__manifest` CAS carrying its complete
logical delta and lineage when the plan has `LineageIntent`. Metadata-only plans carry
their explicit authority/operation rows without manufacturing graph lineage. On every
CAS attempt, the commit authority re-reads and arbitrates the complete `ReadSet`.
The following cases are distinct:
- **Conflict before any physical effect:** safe bounded restart from Prepare.
- **Conflict after physical effects:** recovery case. Keep the sidecar; do not simply
re-stage or point the manifest at whatever HEAD is now live.
- **CAS success:** the graph commit is visible atomically.
When durable table heads land under RFC-024, tombstoning a table must update its mutable
head to an explicit deleted state in this same CAS. A stale live head plus an immutable
tombstone history is not a valid O(tables) current-state representation.
### 4.8 Stage H — finalize
After CAS success:
- delete the sidecar best-effort when the workflow has one;
- refresh/invalidate process-local views;
- enqueue derived index reconciliation and orphan reclaim;
- record recovery/audit completion when applicable.
Sidecar deletion failure does not turn a durable successful graph commit into a user
error. The next recovery barrier proves the exact intent satisfied and removes the
artifact.
### 4.9 Authority-first control workflows
Some metadata workflows have no independently durable physical effect before their
manifest CAS. They may use an **authority-first** subtype:
1. run the recovery barrier, prepare, acquire gates/claims, and revalidate exactly as
above;
2. publish the metadata transition as the first durable effect;
3. perform only idempotent work whose desired target is fully encoded by that
transition and whose interruption cannot expose incorrect graph data.
No generic sidecar is required before step 2 because there is no pre-authority effect
to recover. The authority row itself is the durable recovery cursor. Checkpoint
deletion, a GC-boundary publish, and `OPEN -> DRAINING` stream intent are candidate
examples; each follow-up RFC must prove its post-CAS work is convergent and safe.
A long control workflow is not one giant RFC-022 attempt. Each lifecycle transition,
fold, schema/branch operation, and resume transition is a separate prepared write or
native-ref control step with its own read set and visibility point. If a later phase
needs a non-idempotent or independently visible effect not fully described by the
authority row, that phase uses a normal sidecar before the effect.
## 5. Crash contract
| Crash point | Required result |
|---|---|
| Before sidecar | No independently durable effect occurred; uncommitted objects are reclaimable. |
| Sidecar durable, no effect | Recovery aborts/finalizes the empty intent. |
| Some effects applied, not confirmed | The adapter rolls back, completes, or refuses safely according to its declared strategy. |
| All effects confirmed, before manifest CAS | Recovery rolls forward the exact confirmed manifest delta, or applies the adapter's explicit all-or-nothing rule. |
| Manifest CAS succeeded, sidecar remains | Recovery proves the exact intent visible, audits it, and removes the sidecar. |
Rollback is not assumed safe. Lance `Restore`, schema-file promotion, native refs, and
content-replacing operations have different concurrency properties; each adapter must
state which recovery direction is legal and under what fencing.
## 6. Writer-effect adapters
The adapter registry is closed by default: adding a graph-visible writer requires a
new adapter or an explicit use of an existing adapter. Code review and tests must be
able to enumerate every adapter and every entry point that invokes it.
### 6.1 Mutation and load
- Construct one staged effect per touched table where the Lance API permits it.
Existing-table effects may stage in Prepare; first-touch named-table effects use
the sidecar → target ref → branch-local stage ordering above.
- Put every uniqueness, RI, and cardinality probe in `ReadSet`.
- Revalidate or restart when a probed-but-untouched table changes.
- Preserve strict replacement semantics for overwrite/delete.
- Treat key-conflict fencing and strict keyed Append semantics as RFC-023 concerns;
no fence is credited as protection until that RFC's rollout gates pass.
### 6.2 Branch merge
- Compute row classification outside the gates.
- Include the target graph head and every target table used by classification or
validation in `ReadSet`.
- Any target change before effects forces a complete reclassification; publishing a
result computed against an old target and parenting it to a new live head is
forbidden.
- The adapter supports zero, one, or several physical commits per table and records
exact confirmed post-state before manifest publish.
- Lineage-based candidate discovery may replace the classifier only under RFC-027;
this protocol does not assume it is O(delta).
### 6.3 Schema apply and storage migration
- Acquire the schema-control gate before effect application.
- Include accepted schema identity and every affected table in `ReadSet`.
- Cover schema staging-file promotion, data-table schema/field-metadata commits,
registrations, tombstones, and final schema identity with one recovery intent.
- Write the sidecar before the first table HEAD advance, including unenforced-PK
metadata backfill or other inline metadata commits.
- A branch-wide or graph-wide migration must enumerate every physical manifest/data
branch it changes; updating main does not implicitly migrate older branch manifests.
### 6.4 Data-table optimize and index work
- The adapter may describe zero or multiple inline, content-preserving Lance commits.
- It records the exact achieved version rather than assuming one version of movement.
- If the new data-table version is selected through `__manifest`, publishing that
pointer is a graph-visible commit and uses this protocol.
- Logical operations never fail because a derived index is absent or behind.
- Physical-only internal-table maintenance remains the exception in Section 8.
### 6.5 MemWAL fold
RFC-026 owns enrollment, acknowledgement, quiescence, fresh-read semantics, and the
public ingest surface. Any fold that becomes graph-visible is an adapter here:
- fold-time validation contributes its complete read set;
- Lance `merged_generations` changes atomically with the base-table data commit;
- the sidecar covers the data-commit-to-manifest gap;
- one successful manifest CAS makes the folded graph state visible.
## 7. Native graph-branch ref control protocol
Creating or deleting a graph branch mutates a native Lance branch ref for
`__manifest`. Lance specifies that these operations do not generate a dataset version.
There is no target branch on which to publish before create, and no target remains on
which to publish after delete. They therefore cannot truthfully be instances of the
graph-visible manifest-CAS protocol.
Their control protocol is:
1. run and await the recovery barrier;
2. quiesce enrolled streams as required by RFC-026;
3. acquire any active global claim, such as RFC-025's retention claim, and then
the graph/branch-control gate in §4.3 order;
4. freshly revalidate source ref, target existence, and branch incarnation;
5. perform one native Lance ref mutation, which is the visibility point;
6. release the gate and reclaim orphaned per-table forks asynchronously.
Delete has one recovery disposition that create does not: after the complete
schema/target-branch/all-table gate set has waited out any live in-process owner,
an unresolved sidecar scoped to the branch being removed may be rendered
unreachable by the native ref deletion. A later heal records the orphan-discard
audit and retires it. Graph-global schema recovery still blocks the control, and
create/merge may not adopt this exception.
The native ref operation itself should enforce the freshly checked precondition or
surface concurrent ref mutation as a conflict — but at the pinned Lance revision it
does not: branch-ref creation is an existence check followed by an unconditional
put, not a conditional primitive (the same fact for which RFC-025 §2.3 rejects a
branch ref as a claim mechanism). Until Lance ships a conditional/CAS ref mutation,
graph-branch create/delete therefore inherit the documented single-writer-process
support boundary — the same disposition RFC-023 §10 applies to recovery ownership —
and multi-process branch operations are not advertised. The upstream ask for a
conditional ref primitive is filed alongside this RFC; a process-local branch gate
remains a local optimization, not the missing cross-process guarantee.
These operations do not emit a synthetic graph commit. If a future product contract
requires a native ref mutation and manifest/audit rows to become atomic together, it
needs a separate multi-authority recovery protocol; this RFC does not claim an
atomicity the substrate does not provide.
This exception applies only to graph-level create/delete. Branch merge is a
graph-visible write. Lazy per-table forks created while preparing a branch write are
declared physical effects of that writer and remain subject to its recovery/reclaim
contract.
## 8. Physical-maintenance control protocol
Physical work that does not change manifest-resolved logical graph state does not
create graph lineage merely to fit the main protocol. Examples include:
- compacting `__manifest`, which is itself the authority and is read at its Lance HEAD;
- deleting versions/files already proven unreachable under the active retention
contract;
- reclaiming orphaned branch refs or uncommitted objects;
- rebuilding derived physical state when no graph-visible data-table pointer changes.
Such work must still be idempotent, bounded, observable, and safe under concurrent
native Lance commits. It uses substrate conflict/retry semantics appropriate to the
operation. It must never expose partial logical graph state.
The exception is from graph-lineage publication, not from recovery safety. Maintenance
must run the recovery barrier or refuse before it can replace or delete an artifact
named by an unresolved sidecar. It must also acquire any relevant process-local gates;
as elsewhere, those gates are an optimization rather than cross-process authority.
Data-table compaction/index work that advances a version which graph reads must select
through `__manifest` is not exempt; Section 6.4 applies. Checkpoint reachability and
the mapping from graph checkpoint rows to Lance-native GC pins belong to RFC-025.
## 9. Concurrency and retry semantics
Process-local gates reduce same-process races and establish one deadlock-free order.
They do not coordinate two servers or CLIs. A conforming implementation remains safe
if every process has its own gate manager.
Cross-process safety comes from:
- complete read-set arbitration at the manifest authority;
- Lance transaction conflicts for physical table effects;
- durable recovery before physical HEAD movement;
- refusal to continue past unresolved overlapping recovery.
Retry rules are phase-specific:
- before effects, a read-set mismatch discards and recomputes the whole attempt;
- while applying an effect, the adapter may retry only when Lance guarantees the
operation can be safely replanned from fresh physical state;
- after any effect, manifest contention is resolved through the armed recovery intent,
not by silently rebasing the logical plan;
- retry exhaustion returns a typed, observable conflict.
## 10. Rollout and compatibility
This RFC authorizes a protocol refactor, not a manifest v5 format moment. RFC-023
through RFC-027 own their respective format and public-surface changes.
It also does not require a mutable-tip `GraphState` singleton. Three measured,
local latency fixes can land independently of the adapter conversion:
1. make the graph's shared Lance `Session` a required parameter of every
manifest open/publisher path, so remote opens do not rebuild clients and cold
metadata state;
2. capture one immutable operation-local manifest/lineage view and pass it down
the call stack instead of reopening the same state repeatedly; and
3. remove the verified-redundant branch-idle refresh and the back-to-back second
`branch_delete_as` refresh once their existing coverage asserts unchanged
behavior.
These are narrow access-shape fixes, not a second commit-input authority. They
must preserve snapshot pinning and still cross the recovery/read-set barriers
defined above.
Implementation proceeds in this order:
1. Introduce `PreparedWrite`, `ReadSet`, effect-adapter, and recovery-plan concepts
while preserving existing behavior.
2. Ship conservative branch-wide arbitration first. Mutation/load captures
`(Lance BranchIdentifier, exact optional graph_head, accepted schema identity)`;
every publisher retry compares that token instead of reparenting. Because every
supported graph-content and schema apply advances `graph_head:<branch>` before
schema promotion, the shared head row atomically arbitrates probed-but-untouched
same-branch dependencies. The native branch identifier detects delete/recreate
ABA under the documented single-writer-process branch-control boundary. RFC-024
later narrows false contention with table heads; it is not a correctness
prerequisite for this coarse step. Existing live committed-state validation
probes remain until the narrowed read set replaces them.
> **Implementation note (2026-07-11):** mutation/load now use this coarse
> token, schema-v3 exact-effect sidecars, fixed lineage/rollback outcome ids,
> zero transparent Lance commit retries, and bounded full reprepare before
> effects. Branch merge remains on its writer-specific multi-commit path, but
> now holds the root-shared schema plus source/target branch gates from its
> strict recovery barrier and authority capture through publication. It plans
> with an accepted-contract catalog captured under that schema gate, then
> acquires all catalog table gates for source and target, re-lists recovery,
> and compares fresh manifest versions before Phase A. This closes
> same-process delete/recreate ABA and legacy table-only-writer races while
> its full exact-effect adapter remains future work. Schema apply,
> optimize/index, and MemWAL fold remain on their writer-specific paths until
> their adapter slices land.
3. Convert mutation/load, branch merge, schema apply/migration, data-table optimize,
and graph-visible index work one adapter at a time.
4. Add static or runtime enumeration proving no graph-visible entry point bypasses the
coordinator.
5. Delete superseded writer-specific orchestration only after its crash and
concurrency cells pass through the adapter.
6. Optimize background recovery latency only after the synchronous barrier and all
recovery classifications remain intact.
Mixed writer binaries are not made safe by process-local gates. A deployment may
enable the new protocol only when every writer that can reach the graph obeys the same
sidecar and manifest-arbitration contract, or when a compatibility gate rejects older
writers.
## 11. Required tests and cost gates
### 11.1 Protocol conformance
- Enumerate every graph-visible entry point and its adapter.
- Assert no sidecar-backed adapter can create an independently durable physical
effect before its sidecar is durable.
- Enumerate authority-first workflows and assert their CAS is the first durable
effect and every post-CAS action is idempotently derivable from its authority row.
- Assert one graph-visible operation produces exactly one manifest visibility commit.
- Assert branch create/delete and physical-maintenance exceptions produce no synthetic
graph lineage.
### 11.2 Read-set races
- Two distinct ids racing on the same non-key `@unique` value cannot both publish.
- An edge insert racing deletion of an endpoint must revalidate or conflict.
- Cardinality probes of an untouched table participate in arbitration.
- A target advance after merge classification forces complete reclassification.
- Run the same cells with separate `Omnigraph` handles sharing one root-scoped
process-local gate manager, then with separate processes that do not share it.
### 11.3 Recovery
- Fail before sidecar, after sidecar, after each physical effect, after confirmation,
after manifest CAS, and during sidecar deletion for every adapter.
- A later overlapping writer blocks, heals, or returns recovery-required; it never
advances around the sidecar.
- A live foreign writer's sidecar is not destructively recovered without fencing.
- Recovery proves exact effect identity; a numerically newer unrelated version is not
accepted as proof of ancestry.
### 11.4 Adapter-specific truth tables
- Mutation/load: zero/one table, multi-table, strict, non-strict, and
probed-but-untouched dependency cases.
- Merge: adopt, rewrite, multi-commit, no-op, target advance, and partial-Phase-B crash.
- Schema: staging files, registration-only, metadata HEAD advance, partial multi-table
migration, and branch-local state.
- Optimize/index: zero, one, and several Lance commits, including monotonic publish and
retryable physical contention.
- MemWAL fold, when RFC-026 lands: merged-generation conflict and every fold crash
boundary.
### 11.5 Cost gates
The protocol must not move validation, classification, embedding computation, or
staged-file construction under process-local gates. Measure gate hold time separately
from Prepare. Correctness gates precede latency optimization: a cost regression can
delay rollout, but it cannot justify touched-table-only validation or asynchronous
recovery safety.
## 12. Invariants and deny-list check
This design reinforces the existing architecture:
- Lance and `__manifest` remain the sources of truth; `PreparedWrite` is immutable
attempt-local state, not a shadow mutable tip.
- Graph visibility remains manifest-atomic.
- Recovery remains part of the commit protocol.
- Logical constraints fail loudly and are revalidated when their inputs change.
- Derived indexes and reclaim work converge without becoming logical commit points.
- No custom WAL, transaction manager, buffer pool, or distributed lock is introduced.
The design rejects two tempting deny-list violations: treating process-local queues as
distributed correctness, and treating recovery as derivable background work that a
new writer may outrun.
Acceptance should also add one clarification to invariant 15: a view of immutable,
version-pinned state may be cached, while an in-memory view of the mutable tip is only
a hint. Every use of mutable-tip state as write input must be re-arbitrated by the
commit authority. Durable heads under RFC-024 are one possible authoritative
representation; this protocol does not require or bless a warm parallel truth path.
## 13. Drawbacks and rejected alternatives
### 13.1 One generic staged-storage method
Rejected. Lance does not expose staged forms for every operation, and existing writers
have materially different version movement and rollback safety. A generic method would
either lie about those differences or accumulate writer-kind conditionals. One
protocol plus explicit adapters has lower long-run liability.
### 13.2 Asynchronous heal with optimistic continuation
Rejected. Current recovery classification relies on later writers not advancing past
an unresolved sidecar. Scheduling a sweep does not establish that fact. Recovery may
be proactively asynchronous, but the next writer still crosses a synchronous barrier.
### 13.3 Touched-table-only CAS
Rejected. Non-key uniqueness, RI, cardinality, and merge classification read tables
the operation may not write. Ignoring those dependencies admits commits that were never
valid against one serial graph state.
### 13.4 Treat every state change as a manifest commit
Rejected. Native branch refs and physical maintenance have different substrate
visibility points. Manufacturing manifest commits would add coordination without
making the native mutation atomic with them.
## 14. Reversibility
The in-memory coordinator and adapter refactor is reversible. Recovery-sidecar schema
changes must be versioned and backward-compatible during rollout. This RFC alone does
not authorize a `__manifest` schema-stamp bump, a public wire change, or a new storage
substrate.
The correctness contract is intentionally difficult to reverse: after writers rely on
complete read-set arbitration and recovery-before-write, weakening either would
reintroduce silent integrity or recovery races. Focused irreversible changes are
reviewed in RFC-023 through RFC-027.
## 15. Follow-up RFC boundaries
- **RFC-023** owns PK annotation, fenced merge routing, strict keyed Append behavior,
mixed fenced/unfenced rollout, conflict mapping, and both commit-order tests.
- **RFC-024** owns mutable table-head rows, explicit live/tombstoned head state,
current-state read shape, migration, and cost gates.
- **RFC-025** owns checkpoint rows, Lance-native physical pins, cleanup reachability,
pruned-through semantics, and checkpoint/cleanup crash ordering.
- **RFC-026** owns MemWAL enrollment, durable acknowledgements, fold/dead-letter
behavior, stream quiescence, fresh reads, public surfaces, and upgrade fencing.
- **RFC-027** owns candidate discovery from row lineage, deletion discovery, fallback
semantics, and evidence for any O(delta) merge claim.
Those RFCs call this protocol when they produce a graph-visible write. None may weaken
the recovery barrier, omit read dependencies from `ReadSet`, or create a second graph
visibility point.

480
docs/rfcs/rfc-023-key-conflict-fencing.md Normal file
View file

@ -0,0 +1,480 @@
---
type: spec
title: "RFC-023 — Substrate-native key-conflict fencing"
description: Make concurrent keyed writes fail or retry through Lance's transaction conflict filters, forbid keyed Append, and activate the contract only behind an all-branch fleet/format barrier.
status: draft
tags: [eng, rfc, write-path, concurrency, lance, primary-key]
timestamp: 2026-07-10
owner:
---
# RFC-023: Substrate-native key-conflict fencing
**Status:** Draft / for team review
**Date:** 2026-07-10
**Surveyed:** omnigraph 0.8.1 (`main`); Lance 9.0.0-beta.15 at git rev
`f24e42c1`; full Lance transaction, table-schema, read/write, branching, and
MemWAL specifications; pinned Rust conflict-resolver and merge-insert sources
**Relationship to RFC-022:** this RFC is the fencing decision split from the
earlier monolithic RFC-022 draft. [RFC-022](rfc-022-unified-write-path.md)
defines the shared write/recovery protocol; this RFC owns the substrate,
compatibility stamp, and rollout requirements for key conflicts. It may share a
release with [RFC-024](rfc-024-durable-table-heads.md), but neither RFC depends
on the other's storage decision.
**Audience:** engine, storage, migration, and release maintainers
**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md).
Findings marked **BLOCKER** must be dispositioned before acceptance.
---
## 0. Decision summary
OmniGraph will use Lance's unenforced-primary-key merge-insert filter as the
key-conflict primitive. It will not emulate the filter in the engine and will
not add a lock table or custom transaction manager.
The guarantee is deliberately narrow and strong:
> For a keyed table, two concurrent operations that may insert the same `id`
> MUST NOT both commit silently. They either serialize, retry from a new graph
> base, or fail loudly.
That guarantee cannot be rolled out by annotating a few new tables in an
otherwise v4 graph. Lance's current mixed filtered/unfiltered conflict handling
is directional, and a bare `Append` can commit after a filtered update. The
contract therefore activates only after all supported writers and every table
state reachable from every graph branch have crossed a fencing-compatible
format barrier.
Normative decisions:
1. Node and edge table PK = `id`.
2. Bare Lance `Append` is forbidden for keyed tables.
3. Every keyed insert/upsert path produces the same Lance key filter.
4. Mixed-version serving is forbidden during activation; the fencing-compatible
format stamp is written last and older binaries refuse it.
5. PK metadata is permanent and preserved by every later schema/data rewrite.
6. Existing-table migration covers every graph branch, including lazy-inherited
table states, and is recoverable by rolling forward.
7. Fencing does not replace read-set OCC for `@unique`, RI, or cardinality.
## 1. Problem
OmniGraph validates duplicate `@key` values inside one delta, but today two
processes can both read a base where `id = K` is absent, stage disjoint Lance
fragments containing `K`, and let Lance rebase both commits. The graph manifest
CAS orders graph publication; it does not tell Lance that the two data-table
transactions inserted the same logical key.
The result is worse than an ordinary write conflict: both callers can receive
success while a keyed table contains duplicate IDs. Every subsequent upsert,
edge lookup, uniqueness check, and merge then operates on a broken identity
relation.
The desired primitive already exists in Lance. The missing work is to use it on
every keyed path and to define a rollout that never admits an unfenced writer.
## 2. Substrate facts and the directional asymmetry
This section is load-bearing. Tests pin every statement before implementation.
### 2.1 Filter attachment
On the pinned Lance revision, a v2 merge-insert whose ON field IDs exactly equal
the schema's unenforced PK field IDs attaches an `inserted_rows_filter` to its
`Operation::Update`. The filter contains keys for rows classified as inserts;
updates of existing rows continue to use Lance's affected-row / fragment
conflict machinery.
The legacy indexed merge path does not attach this filter. Therefore a BTREE on
`id` can route an otherwise correct merge onto an unfenced path unless the
caller disables that path or Lance wires the filter into it.
### 2.2 Conflict compatibility is directional
Lance transaction compatibility is evaluated from the transaction currently
attempting to commit against transactions that committed after its read
version. It is not implicitly symmetric.
At `f24e42c1`, `check_update_txn` behaves as follows:
- current `Some(filter)` vs committed `Some(filter)` — compare field IDs and
filter intersection; overlap or incompatible filter configuration retries;
- current `Some(filter)` vs committed `None` — retry conservatively;
- current `None` vs committed `Some(filter)` — no corresponding conservative
arm; it may rebase;
- current bare `Append` vs committed filtered `Update` — `Append` treats the
`Update` as compatible.
Consequently, "filtered vs unfiltered conflicts" is not a sufficient rollout
argument. Commit order matters. A filtered writer can win first and a stale
unfiltered writer or bare keyed append can still land second.
### 2.3 What a PK annotation does not do
The key is explicitly *unenforced*. Merely setting the metadata:
- does not validate historical uniqueness;
- does not make bare appends unique;
- does not protect a merge whose ON set differs from the PK;
- does not repair an existing duplicate;
- does not replace OmniGraph's semantic validators.
## 3. Scope and non-goals
In scope:
- node and edge data tables;
- mutation, load, branch merge, recovery replay, and WAL fold paths;
- new-table creation and all-branch existing-table activation;
- schema/overwrite preservation of PK metadata;
- typed retry behavior and coverage gates.
Out of scope:
- using a composite edge PK (`src`, `dst`);
- enforcing arbitrary `@unique` groups through the Lance PK;
- keyless streaming-table deduplication;
- a custom OmniGraph WAL, lock table, or transaction manager;
- declaring general multi-process writes supported before foreign-process
recovery-sidecar ownership is solved.
## 4. Table classes and PK contract
### 4.1 Keyed graph tables
Every normal node and edge table has a non-null `id` field. Its Lance schema
MUST mark exactly that field as the unenforced PK. For edges, `src` and `dst`
remain ordinary fields governed by referential-integrity and cardinality
validation. An edge endpoint move is an update of the row identified by `id`.
The PK field is addressed by stable field ID, not column position or mutable
name. Until rename-stable OmniGraph type/property identity is closed, the
fencing migration cannot claim rename safety.
### 4.2 Keyless append-only tables
A table explicitly declared append-only may omit a PK. Such a table may use
Lance `Append`, including MemWAL append-only operation. It receives no
same-logical-key guarantee because it has no logical key.
Current node and edge types are not in this class: both have graph identity on
`id`. The class is reserved for an explicitly designed internal or future
non-graph table surface.
The distinction is catalog-derived and first-class. Callers do not choose it
with an ad-hoc flag.
## 5. Normative write routing
| Logical operation | Keyed table | Keyless append-only table |
|---|---|---|
| Strict insert / load append | merge-insert ON exactly `id`, `WhenMatched::Fail`, filtered path | `Append` allowed |
| Upsert / load merge | merge-insert ON exactly `id`, `WhenMatched::UpdateAll`, filtered path | workload-specific |
| Fast-forward branch merge of new rows | filtered merge-insert with `WhenMatched::Fail`, even when every row was classified new | `Append` allowed |
| WAL upsert fold | filtered merge-insert with `merged_generations` | append transaction allowed |
| Update existing row | merge-insert/update with affected-row conflict metadata | workload-specific |
| Delete | staged Lance delete; PK filter is not the delete-conflict primitive | staged Lance delete |
| Overwrite | staged overwrite whose output schema preserves the exact PK | staged overwrite |
### 5.1 No keyed `Append`
The prohibition includes internal optimization paths. A caller may not infer
"all rows are new" and switch a keyed table to `stage_append`: that inference
was made against a snapshot and is exactly what a concurrent same-key writer can
invalidate.
Routing through merge-insert does not collapse strict and upsert semantics.
Strict `load --mode append` and other insert-only surfaces use
`WhenMatched::Fail`; a row already present at the pinned base or discovered at
execution remains an error. Only declared upsert surfaces use `UpdateAll`.
The storage trait and `forbidden_apis` guard MUST make a keyed append difficult
to express accidentally. The fast-forward merge optimization is retained only
for keyless append-only tables unless Lance ships a key-filtered append
transaction.
This prohibition knowingly retires a measured fix. The fast-forward append
path exists because a whole-delta merge-insert join exhausted the query memory
pool on embedding-bearing tables (#277); routing adopted rows back through a
filtered merge-insert re-exposes that workload to join memory behavior. The
regression class is therefore a named ship gate: the fenced bulk adopt-merge
must pass the §11.4 memory/cost gate on embedding-bearing tables — via bounded
batched fenced merges inside one staged transaction, a pool-bounded execution
mode, or an upstream key-filtered append — before the keyed fast-forward path
is removed. Correctness wins the ordering, but the memory bound is not
optional.
### 5.2 Routing choice
There are two acceptable implementations:
1. use the v2 merge path (`use_index(false)`) and pass its scale gate; or
2. consume a pinned Lance revision whose indexed path emits the identical
filter and passes the same surface guards.
If the v2 hash-join cost scales unacceptably at the Phase-B workload, fencing
waits for option 2. Correctness is not traded for the old indexed-path speed.
### 5.3 Symmetric mixed-transaction behavior
Before activation, the pinned Lance revision MUST conservatively reject both
orders of:
- filtered Update vs unfiltered Update;
- filtered Update vs bare Append.
The preferred fix is upstream conflict-resolver symmetry. A workspace-only
fork is not an accepted permanent design. The fleet barrier remains necessary
even after that fix because two old, unfiltered writers still have no filters
to compare.
## 6. Retry and validation semantics
A Lance retryable key conflict restarts the entire logical attempt:
1. gather a new graph snapshot and schema identity;
2. rerun all delta and committed-state validation;
3. restage from that base;
4. commit and publish through the normal recovery-covered pipeline.
It is incorrect to retry only `commit_staged`: an insert may have become an
update, defaults or checks may now differ, and cross-table validation may have
changed.
Upsert surfaces may perform the bounded semantic retry. Strict insert surfaces,
including `load --mode append`, do not change meaning under contention: both an
already-present match from `WhenMatched::Fail` and a concurrent same-key commit
normalize to typed `KeyConflict` / HTTP 409 for the whole strict operation. They
do not switch to `UpdateAll`; callers decide whether to resubmit. Other strict
read-modify-write surfaces retain their typed write conflict. Retry exhaustion
on a non-strict upsert remains a retryable 409.
Fencing covers the PK insertion race only. `@unique` values on different IDs,
edge RI, and cardinality depend on a read set. Their correctness requires the
read-set-in-CAS design or equivalent revalidation before HEAD movement; this RFC
does not claim that fences close those races.
## 7. Version and fleet barrier
### 7.1 No partial activation in the old format
OmniGraph MUST NOT annotate a new data table and advertise fencing while the
graph remains generally writable by older binaries. An older process can select
the legacy merge path or keyed append and bypass the guarantee.
This RFC owns its activation boundary:
- operators quiesce every server, CLI writer, and embedded writer for the
graph;
- one migration claimant holds an atomic create-if-absent claim with a random
owner/fencing token; a native Lance branch sentinel is not accepted as CAS;
- only the dedicated migration binary may open the old graph for writes;
- the fencing-compatible stamp is written after every branch/table verification;
- normal serving begins only after the stamp; older binaries then refuse.
The migration claim uses the storage adapter's `PutMode::Create` contract,
records operation/owner token, and has no time-only takeover. Recovery under the
fleet outage must classify the migration ledger/sidecars before replacing a
stale token.
The stamp is graph-wide and read from the reserved main manifest before any
named-branch open; selecting a named branch cannot bypass the compatibility
check.
An in-process mutex is not a fleet barrier. A marker unknown to v4 binaries is
also not a fleet barrier. The operator procedure and cluster control plane must
keep old writers stopped until finalization.
The exact format number is assigned when this RFC is accepted. If RFC-024 is
also accepted and ready, the two migrations may deliberately share its v5
release after a combined failure-matrix review. If durable heads fail their
cost gate, fencing still proceeds with its own next compatible stamp; if
fencing is blocked upstream, durable heads need not wait.
If durable heads are already active when fencing migrates, every PK metadata
version repoint also emits the identity-bearing journal event and matching
`table_head` transition in the same manifest CAS. If fencing lands first, its
format/stamp becomes an explicit predecessor that a later heads migration must
preserve. Acceptance covers heads-first, fencing-first, and co-release orders.
### 7.2 New graphs
A graph created directly at the fencing-compatible format creates every keyed
table with the PK metadata already present and enables only the write routing
in §5. There is no post-create annotation window.
## 8. All-branch PK migration
Migration operates on graph states, not merely table roots. The unit is every
reachable tuple:
```
(graph_branch, table_key, table_path, pinned_table_branch, pinned_table_version)
```
This matters for lazy branches: a graph branch may still point at an old main
table version whose schema predates the PK, even after main HEAD is annotated.
Under the fleet barrier, the migration:
1. enumerates and incarnation-pins every live graph branch;
2. folds each branch manifest and enumerates its live keyed tables;
3. validates that every pinned table image has non-null, unique `id` values;
4. acquires branch/table gates in RFC-022 order and freshly revalidates the
pinned tuple, schema identity, and migration claim;
5. writes a per-unit RFC-022 sidecar declaring expected branch/table state, the
optional native fork effect, PK metadata effect, and intended manifest delta
before either effect can persist;
6. for an owned table branch, commits a set-if-absent PK metadata update;
7. for a lazy-inherited state, forks an owned table branch from the *exact
pinned version*, applies the PK metadata there, and leaves row contents
unchanged;
8. records exact achieved fork identity and table version in the sidecar;
9. publishes that graph branch's manifest to the annotated physical version
with an audited migration marker but no graph-content commit or graph-head
movement, including a table-head transition when heads are active;
10. records a branch/table completion digest;
11. re-enumerates branches and verifies every live branch before writing the
fencing-compatible stamp.
PK installation advances a Lance table version before the graph manifest can
publish it, and a lazy fork creates native ref state first. The sidecar covers
both gaps and lets recovery reclaim or adopt the exact fork rather than infer
from a branch name. Because Lance forbids clearing/changing a set PK,
migration is roll-forward-only:
- an already-correct PK is an idempotent success and is not rewritten;
- an absent PK resumes installation;
- a different PK is a loud, operator-visible refusal;
- recovery never attempts to "undo" PK metadata.
Branch create/delete, schema apply, and normal data writes remain blocked for
the whole enumeration/install/verify interval. The migration ledger makes a
crash resumable without treating partial annotation as a served graph.
## 9. Preservation after activation
Once set, the following are storage invariants:
- a table overwrite carries the same PK field IDs and positions;
- schema apply cannot remove, replace, reorder semantically, or make nullable a
PK field;
- rename preserves the PK field ID and metadata;
- branch fork/clone preserves it;
- import/rebuild creates it before accepting data;
- recovery restore may select an older data image only if that image is already
fencing/PK-compatible;
- a table recreation uses a new table incarnation but installs the same
catalog-derived PK contract at creation;
- `__manifest`'s existing legacy PK key form is preserved exactly as stored;
the migration never rewrites or "normalizes" it. Lance forbids changing a
set PK, and the native-namespace decoupling documented in the Lance
alignment audit depends on that legacy form remaining in place.
Every open-for-write path verifies the physical schema matches the catalog PK
contract. The check is against the pinned physical schema and is not a
maintained parallel registry.
## 10. Recovery and multi-process scope
All data writes retain the existing Phase A-D sidecar protocol. The key filter
does not close the table-HEAD-before-graph-manifest window.
The fenced data-table transaction is cross-process safe in its failure-free
commit path. OmniGraph's current recovery sweep, however, serializes with live
writers only in-process; a foreign recovery process can still inspect a live
sidecar, and destructive `Restore` cannot be made convergence-idempotent.
Therefore this RFC MUST use one of two honest dispositions:
1. retain the documented single-writer-process support boundary and describe
fences as closing the silent key-race primitive only; or
2. land a cross-process sidecar claim/lease before advertising general
multi-process writes.
Fences alone are not evidence for disposition 2.
## 11. Tests and acceptance gates
### 11.1 Lance surface guards
- exact PK ON set + v2 path produces a non-empty filter for inserts;
- `WhenMatched::Fail` preserves that filter and reports an existing match
without writing;
- mismatched ON set produces no filter;
- legacy/indexed path behavior is pinned until replaced;
- filtered/filtered overlapping keys retry;
- filtered/filtered disjoint keys may rebase;
- filtered/unfiltered retries in **both** commit orders;
- filtered Update/bare Append retries in **both** commit orders;
- PK metadata cannot be changed or removed once installed.
### 11.2 Engine concurrency tests
- the same-key DST cell becomes a hard assertion with N concurrent writers;
- different keys remain concurrently writable;
- every keyed load/mutation/merge/fold path is observed to use the filtered
primitive;
- strict append of an existing `id` still fails and never updates the row;
- strict pre-existing-match and concurrent-insert cases normalize to the same
external `KeyConflict` while preserving `WhenMatched::Fail`;
- a source-walk guard rejects keyed `stage_append`, including the former
fast-forward path;
- a retry reruns validation rather than committing the stale staged batch.
### 11.3 Migration and recovery tests
- main plus owned and lazy-inherited graph branches all emerge PK-annotated;
- duplicate historical IDs abort before the fencing-compatible stamp;
- crash after each table annotation and before each manifest repoint resumes
without data change;
- crash before/after lazy fork and PK metadata commit recovers the exact
sidecar-recorded ref/version;
- branch enumeration is incarnation-safe;
- old binary/new graph and new binary/partially migrated graph refuse loudly;
- heads-first, fencing-first, and co-release upgrades preserve every active
format capability and produce identical logical rows;
- overwrite, schema apply, branch fork, restore, and import preserve PK metadata.
### 11.4 Cost gate
Measure a small upsert into 10K, 100K, and 1M-row indexed tables using the
shared cost harness. If `use_index(false)` makes work scale with table size
beyond the accepted budget, the indexed-path upstream work is a ship blocker.
Additionally measure the bulk adopt-merge shape that motivated the keyed
fast-forward path (#277): a many-row, all-new-rows fenced merge into an
embedding-bearing table, asserting peak memory bounded by batch size rather
than table or delta width. If the fenced path cannot meet that bound, the
keyed fast-forward removal waits for the mitigation named in §5.1.
> 💬 **Instrument required (tightening 5 in the
> [review ledger](../dev/rfc-022-027-architecture-review.md)):**
> `helpers::cost` measures I/O, not peak RSS, so this memory bound is
> unenforceable as written. Use the subprocess `scenarios.rs` harness or an
> equivalent `wait4`/`ru_maxrss` instrument, and name dataset sizes, baseline,
> cap, and pass threshold.
## 12. Decisions and open gates
### Decided
- `id`, not `src`+`dst`, is the edge PK.
- No keyed append, including optimization-only append.
- No mixed-fleet or new-table-only v4 rollout.
- PK migration is all-branch, offline, idempotent, and roll-forward-only.
- A retryable upsert conflict retries the logical operation; strict insert maps
both existing and concurrent matches to `KeyConflict` without changing mode.
- Read-set validation remains a separate required concurrency design.
### Open ship gates
1. Upstream symmetric filtered/unfiltered and filtered/Append conflict behavior.
2. v2-path scale result versus indexed-path filter availability.
3. Operator repair procedure for pre-existing duplicate IDs.
4. Rename-stable field/type identity.
5. Cross-process recovery ownership before any broadened topology claim.
6. Final format-number/release sequencing and the all-branch fleet stamp.
7. The fenced bulk adopt-merge memory/cost gate on embedding-bearing tables
(the #277 regression class) — see §5.1 and §11.4.

558
docs/rfcs/rfc-024-durable-table-heads.md Normal file
View file

@ -0,0 +1,558 @@
---
type: spec
title: "RFC-024 — Durable table heads and the v5 manifest"
description: Materialize one live-or-tombstoned current-state row per table inside each manifest branch, prove bounded physical lookup, and migrate every branch atomically before stamping v5.
status: draft
tags: [eng, rfc, manifest, write-path, versioning, migration, lance]
timestamp: 2026-07-10
owner:
---
# RFC-024: Durable table heads and the v5 manifest
**Status:** Draft / for team review
**Date:** 2026-07-10
**Surveyed:** omnigraph 0.8.1 (`main`); Lance 9.0.0-beta.15 at git rev
`f24e42c1`; full Lance table layout, transaction, branching, indexing,
compaction, cleanup, and object-store specifications
**Relationship to RFC-022:** this RFC is the durable-heads decision split from
the earlier monolithic RFC-022 draft. [RFC-022](rfc-022-unified-write-path.md)
defines the shared publisher/recovery protocol; this RFC owns the v5 format and
migration boundary. It deliberately excludes checkpoint retention, which
[RFC-025](rfc-025-checkpoint-retention.md) reviews separately. Key fencing in
[RFC-023](rfc-023-key-conflict-fencing.md) is also independently reviewable;
the two may share a release but do not block one another's evidence gates.
**Audience:** engine, manifest, migration, branch, and release maintainers
**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md).
Findings marked **BLOCKER** must be dispositioned before acceptance.
---
Throughout this draft, **v5** means the next internal schema after today's v4.
If another independently accepted format change lands first, the release number
is reassigned; none of the head-row semantics depend on the numeral.
## 0. Decision summary
The current manifest is both an immutable graph journal and the place writers
ask "what is current?" Current-state resolution folds history, so its physical
cost grows with commit count even though the answer contains only one value per
table.
v5 adds one mutable, durable `table_head` row per stable table identity inside
each native `__manifest` branch. The publisher updates the head in the same
Lance merge-insert transaction as immutable table-version rows, tombstone
events, graph lineage, and `graph_head`. The journal remains the history source;
heads become the current-state source.
The format does **not** ship merely because the logical result has O(tables)
rows. A filtered scan over a history-sized Lance table is still O(history)
physical work. v5 is gated on a Lance-native indexed lookup whose measured I/O
is flat at history depth and whose uncovered tail is bounded and observable.
Normative decisions:
1. Heads live in `__manifest`; there is no second heads dataset and no warm
mutable-tip authority.
2. Head state is explicitly `live` or `tombstoned` and carries an incarnation.
3. Every publish and recovery outcome updates journal and head atomically.
4. Current-state reads use heads; history reads use the journal.
5. Missing or duplicate heads in v5 are corruption, not a reason to silently
return to the history fold.
6. Migration covers every live manifest branch, then stamps main v5 last.
7. If RFC-023 is co-released, the combined migration also verifies PK fencing;
durable heads do not depend on that decision.
8. Checkpoint/retention markers are deferred to a separate RFC.
## 1. Problem
`__manifest` currently stores immutable table-version and tombstone rows. To
resolve a branch tip, readers scan those rows, select the greatest version per
table, and apply tombstones. This makes a write's coordination cost depend on
the total graph history. Compaction reduces fragment count but cannot remove
the semantic journal rows, so even a compacted manifest retains a row-volume
slope.
Caching that fold as mutable in-process state is the wrong authority shape for
writes: invalidation becomes a correctness condition across processes and
branch incarnations. Storing the folded answer durably in the same transaction
as the journal removes both liabilities:
- no parallel authority can drift; and
- current-state work is proportional to catalog width, provided physical
lookup is also bounded.
## 2. Scope and non-goals
In scope:
- v5 table-head schema and state transitions;
- atomic publisher, recovery, and current-read semantics;
- bounded physical access proof;
- all-branch predecessor→heads migration and compatibility refusal;
- optional co-release integration with RFC-023's all-branch PK activation.
Out of scope:
- a `GraphState` singleton or warm publish input;
- a separate `__heads` Lance dataset;
- deleting immutable journal rows;
- commit-graph ancestry acceleration;
- checkpoint retention and arbitrary-version GC;
- using a physical index as a correctness precondition.
Checkpoint retention is excluded because a row in `__manifest` does not by
itself pin a version in another Lance dataset. Lance cleanup recognizes its own
versions, tags, and branch references, not foreign references. A later RFC must
define substrate-native per-table pins and their crash-safe lifecycle.
## 3. v5 table-head schema
Each native manifest branch contains exactly one current head row for each
stable table identity known to that branch.
### 3.1 Identity and object key
```
object_id = "table_head:<stable_table_id>"
object_type = "table_head"
```
`stable_table_id` survives a type rename. It is not the display name and is not
derived anew from `kind:name`. Closing the current rename-stable identity gap is
a v5 ship gate, and **this RFC owns it**: the stable-ID encoding, the compiler
contract that mints and preserves it, and the incarnation baseline are defined
here. RFC-023 §4.1 (rename-safe PK claims) and RFC-025 §2.1 (checkpoint table
rows) consume this identity and must not ship identity-dependent claims before
it lands. No other RFC in the family defines a competing identity scheme.
> 💬 **Superseded by review (BLOCKER-07 in the
> [review ledger](../dev/rfc-022-027-architecture-review.md)):** ownership
> *inside* this RFC contradicts siblings that consume the identity while
> calling this RFC optional (RFC-025) or without declaring the dependency
> (RFC-026). Disposition: extract identity/incarnation into its own
> prerequisite RFC, or accept this RFC in full before the identity-dependent
> siblings. Replace this ownership note accordingly.
The head row is mutable under `WhenMatched::UpdateAll`, just like
`graph_head:<branch>`. There is one object ID per stable identity; recreation
does not leave multiple candidate heads.
### 3.2 Payload
v5 reuses the manifest's typed columns where they already fit and stores the
remaining versioned payload in a typed JSON structure:
```text
TableHeadMetadata {
state: "live" | "tombstoned",
stable_table_id: String,
incarnation_id: ULID,
schema_hash: String,
head_graph_commit_id: Option<ULID>,
}
```
The row columns have these meanings:
| Column | `live` | `tombstoned` |
|---|---|---|
| `table_key` | current public table key/name | last public key/name |
| `location` | physical table location | last physical location, diagnostic only |
| `table_version` | current visible Lance version | last live Lance version |
| `table_branch` | physical owner branch, nullable for main | last owner branch |
| `row_count` | current row count | null |
| `metadata` | `TableHeadMetadata` | `TableHeadMetadata`; `head_graph_commit_id` names the tombstoning graph commit |
The `state` field is authoritative. A tombstoned row MUST NOT become live merely
because an older journal version has a greater data-table version than some
other row.
### 3.3 Incarnations
`incarnation_id` distinguishes drop/recreate ABA:
- rename preserves stable ID and incarnation;
- ordinary writes preserve both;
- physical owner handoff preserves both;
- dropping the type transitions the one head to `tombstoned`;
- recreating a logically new table mints a new incarnation and updates the same
stable-identity head to `live` only through an explicit schema operation.
If recreation is assigned a new stable table identity by schema semantics, it
gets a new head object and the old identity remains tombstoned. The schema
planner, not a name comparison, chooses this outcome.
### 3.4 Journal identity
The mutable head is not the only place that records identity. Every v5
table-version, registration, rename, and tombstone journal event carries
`stable_table_id` and `incarnation_id`; otherwise drop/recreate followed by a
new physical dataset whose Lance versions restart cannot be replayed
unambiguously.
Migration writes one immutable v5 incarnation-baseline row per known identity,
bound to the source-tip digest. New registrations/recreations write an immutable
incarnation transition. Head repair starts from that baseline and replays only
identity-bearing v5 events. It does not infer identity from mutable names or
compare unrelated Lance version numbers. If pre-v5 history cannot be mapped to
one baseline identity deterministically, migration refuses before the stamp.
## 4. State-transition rules
| Event | Required head transition |
|---|---|
| Register table | absent → live, new incarnation |
| Data write / optimize / index publish | live version N → live version M |
| Owner-branch handoff | live owner A → live owner B, even if version is equal |
| Rename | live key/name A → live key/name B; identity/incarnation unchanged |
| Drop table | live → tombstoned in the same graph publish as the tombstone journal row |
| Recreate | tombstoned → live with the schema-planned identity/incarnation outcome |
| Recovery roll-forward | apply the failed writer's intended live/tombstone transition |
| Recovery rollback | publish a head matching the restored physical version and logical pre-write state |
No path may append a table-version or tombstone journal row without including
its corresponding head mutation in the same publisher source batch.
## 5. Atomic publisher contract
The v5 publisher constructs one merge-insert source containing:
- immutable, identity-bearing table-version rows;
- immutable, identity-bearing table-tombstone/transition rows;
- mutable table-head rows;
- when the RFC-022 plan carries `LineageIntent`, the immutable `graph_commit`
and mutable `graph_head:<branch>` rows;
- for metadata-only plans, their specific CAS authority/operation rows without
manufacturing graph lineage.
One Lance manifest commit makes the entire set visible. The publisher still
resolves the graph parent and re-reads commit authority inside every CAS retry.
Expected table versions are compared against table heads, not reconstructed by
folding the journal.
Two graph-content writers touching disjoint data tables still contend on
`graph_head:<branch>`, form one linear graph history, and re-parent on retry.
Writers touching the same table also contend on its one `table_head` object.
Metadata-only CASes contend on the stable authority rows named by their complete
`ReadSet`; they do not update `graph_head` merely to create contention.
The immutable journal remains necessary for snapshots, diffs, audit, migration
verification, and head repair. Head rows do not replace or truncate it.
## 6. Read contract
### 6.1 Current state
A v5 current-state read:
1. derives the expected live stable table IDs from the pinned catalog and fixed
system-table registry;
2. issues a structured lookup for the exact head object IDs;
3. requires exactly one valid row per expected identity;
4. includes only rows whose authoritative state is `live`;
5. validates schema identity from the head payload;
6. returns one immutable `Snapshot` used for the operation's lifetime.
Missing, duplicate, unknown-state, or schema-mismatched **live** heads fail
loudly. The hot path does not enumerate every identity ever dropped merely to
prove all tombstone heads exist; the branch completion digest plus explicit
`heads verify`/repair owns bounded tombstone-set validation. A missing or
duplicate tombstone is still corruption, but normal reads do not regain an
O(history) scan to discover it.
### 6.2 History
`snapshot_at_version`, commit resolution, change feeds, and audit continue to
read immutable journal/lineage state at the requested manifest version. A v5
manifest version contains the heads as they stood at that version, but the
journal remains the normative explanation of how the state arose.
### 6.3 Diagnostic repair
An explicit offline repair loads the branch's v5 incarnation baseline, replays
identity-bearing journal transitions from that baseline, compares the result to
its heads/marker digest, and publishes corrected heads with an audited system
actor. Repair is not part of the read hot path and never silently runs from a
query.
## 7. Bounded physical lookup is a ship gate
Logical O(tables) output does not prove physical O(tables) work. Without an
index, `object_id IN (...)` still scans the journal-bearing manifest fragments;
compaction reduces files but not semantic row count.
### 7.1 Required property
At fixed catalog width, a reconciled v5 current-state lookup MUST have zero
positive slope in:
- manifest object-store reads;
- bytes read;
- fragments/pages scanned; and
- rows decoded
as commit history grows.
The bound must hold on a real object store as well as local FS and must be shown
for compacted and uncompacted histories. The test uses the shared IO-tracking
harness and installs the tracker before the manifest handle opens.
### 7.2 Candidate access shape
The primary candidate is a structured exact lookup on `object_id` backed by a
Lance scalar index. It is acceptable only if measurement proves:
- indexed head lookup avoids journal-fragment scans;
- newly committed head rows leave at most a bounded uncovered tail;
- reconciliation restores coverage without synchronous expensive work in the
logical write path;
- index absence or partial coverage remains logically correct and is surfaced
as an observable degraded-cost mode.
The index is derived state. Queries MUST remain correct if it is missing, and a
missing index cannot block a logical write. The performance promise applies to
the reconciled serving state and includes an explicit bound on uncovered work;
it is not inferred merely from the existence of an index declaration.
### 7.3 Rejected access shape
A separate heads dataset is rejected. Lance commits are per dataset, so it
would reintroduce a journal→heads crash gap and require another sidecar protocol
for the very pointer whose purpose is to remove drift.
If no in-manifest Lance-native access shape passes the gate, v5 does not ship
with head rows. The fallback is to retain v4 plus the local session/view-passing
improvements, not to waive the cost claim.
## 8. Recovery protocol
Data-table writers still use the existing four phases:
1. write sidecar before a Lance HEAD advance;
2. commit staged/inline table work;
3. publish `__manifest`;
4. delete sidecar.
The sidecar's logical intent in v5 includes the expected and desired table-head
payload. Recovery behavior is therefore complete:
- roll-forward publishes the data version, journal row, table head, lineage,
and graph head together;
- rollback restores the physical version, then publishes journal/audit state
and a table head matching the restored logical state;
- a stale sidecar whose goal is already represented by the exact head
incarnation/version converges idempotently;
- a partially matching head is not treated as success.
Recovery remains a synchronous barrier before any later writer advances a
touched table. Index reconciliation may be asynchronous; unresolved commit
protocol state may not.
## 9. v5 boundary and compatibility
v5 comprises:
1. table-head rows and their publish/read semantics;
2. branch-local v5 completion markers; and
3. the graph-level internal-schema stamp written after verification.
It does **not** comprise checkpoint/retention markers.
If RFC-023 is independently accepted and ready for the same release, v5 may
also carry its PK annotation and fencing-compatible marker after a combined
migration review. That is release coordination, not a prerequisite of heads.
Format sequencing is explicit:
| Order | Result |
|---|---|
| Heads first | proposed v5 contains heads; later fencing migration preserves and atomically updates heads for every PK-version repoint |
| Fencing first | fencing takes the next format number; this heads migration takes the following number, accepts that exact predecessor, and preserves its PK/stamp invariant |
| Co-release | one format maps to both independently accepted capabilities and runs the combined failure matrix |
Migration code dispatches from an exact supported predecessor feature set; it
never assumes that “pre-heads” means pristine v4 or drops a capability it does
not own.
After upgrade, serving is strict-single-version:
- v5 binaries refuse unstamped or partially migrated graphs in normal mode;
- older binaries refuse v5 with the existing upgrade message;
- every open reads the graph-wide main stamp before selecting a named branch;
- only the dedicated offline migration command may open the exact supported
predecessor for conversion;
- there is no mixed v4/v5 serving period.
## 10. All-branch predecessor→heads migration
### 10.1 Preconditions
The operator stops every graph writer and acquires an atomic
create-if-absent migration claim (with exact owner/fencing token; not a Lance
branch sentinel). The barrier covers server, CLI, embedded, maintenance,
branch, and schema writes. Because predecessor binaries do not understand the
new in-graph migration marker, the offline fleet barrier remains mandatory
until the final stamp.
The claim uses `PutMode::Create`, records the migration operation and owner
token, and permits no time-only takeover. Recovery classifies the durable
ledger/sidecars under the fleet outage before replacing a stale token.
After acquiring the claim, migration completes RFC-022's recovery barrier
before pinning any branch source tip.
If fencing is co-released, the migration first executes RFC-023's all-branch
table-PK plan. Otherwise head migration neither adds nor changes PK metadata.
### 10.2 Per-branch conversion
For each live native `__manifest` branch:
1. capture branch name, ref incarnation, manifest version, and e_tag/timestamp;
2. run the predecessor journal+tombstone fold once at that pinned tip;
3. construct exactly one live or tombstoned head plus one immutable incarnation
baseline per stable table identity;
4. validate table schema hashes and, only in a combined release, RFC-023 PK state;
5. publish all heads/baselines, an audited migration record
(`actor = omnigraph:migration/v5`), and a branch-local completion marker in
one manifest CAS that revalidates the captured ref incarnation/source tip;
this physical representation migration does not create a graph-content
commit or move `graph_head`;
6. store in the marker the source tip, head count, identity-baseline digest,
head-set digest, and heads-format version; the marker is content-scoped and
deliberately does not embed the source branch incarnation; source tip is
provenance, while digest/format determine inherited-marker validity;
7. verify by reopening the produced branch version through the v5 head reader.
The completion marker has a deterministic object ID within each manifest
branch. A retry that finds a matching source/digest is complete; a mismatching
marker is a loud migration conflict.
### 10.3 Finalization
After all branches report completion, migration re-enumerates native manifest
refs and verifies the same incarnation set. Branch create/delete is blocked, so
any difference indicates out-of-band modification and aborts finalization.
Only then does it stamp main as internal schema v5. The stamp is the fleet
commit point: before it, no serving process may start; after it, only v5 code may
open the graph.
New branches created after v5 fork a source manifest that already contains
complete heads and the content-scoped v5 marker. The inherited marker remains
valid for the identical snapshot; branch open separately binds that content to
the new native ref incarnation and validates the graph-wide stamp. No
post-create marker rewrite is required.
## 11. Migration recovery
The migration keeps a durable ledger outside the ordinary read path with one
record per branch and, in a combined release, per RFC-023 table conversion. It
records expected source incarnations, achieved physical versions, produced
manifest versions, and digests.
Recovery is idempotent and roll-forward-only:
- completed, digest-matching branch conversions are skipped;
- a table HEAD advance not yet represented in its graph branch is recovered by
the normal sidecar and then branch conversion resumes;
- an uncommitted head-row source leaves no visible state and is rebuilt;
- a committed branch marker is authoritative for that branch conversion;
- the main v5 stamp is never written while any ledger unit is incomplete;
- co-released PK metadata is never cleared to simulate rollback.
If migration crashes, the graph remains offline. Restarting an old serving
binary is not a recovery procedure; the operator resumes the v5 migration.
## 12. Tests and acceptance gates
### 12.1 Head semantics
- current state from heads is byte-equivalent to the predecessor fold across realistic
histories;
- live→tombstoned never resurrects an older live version;
- drop/recreate distinguishes incarnations;
- identity-bearing journal replay remains unambiguous when a recreated physical
dataset restarts Lance version numbering;
- rename preserves identity/incarnation and changes the public key only;
- owner-branch handoff at an equal table version updates the head;
- missing, duplicate, malformed, and schema-mismatched heads fail loudly.
- `heads verify` detects a missing/duplicate tombstone against the baseline and
completion digest without adding that enumeration to every current read.
### 12.2 Publisher and recovery
- concurrent disjoint writers produce one linear graph chain and correct heads;
- same-table writers contend on one head row;
- failpoints after every table commit but before manifest publish recover to
matching physical version, journal, table head, and graph head;
- rollback and roll-forward assertions include head payloads, not only table
versions;
- a stale sidecar converges exactly once with one audit record.
### 12.3 All-branch migration
- main, owned named branches, and lazy-inherited branches all convert;
- tombstoned types are represented on every relevant branch;
- crashes before/after every per-branch CAS resume from the ledger;
- branch ref deletion/recreation is caught by incarnation verification;
- final stamp is impossible while any table or branch marker is absent;
- each branch's graph head and content lineage are unchanged by head migration;
- old/new binary refusal matrix covers every supported predecessor capability
set, partial heads migration, and complete heads format;
- a post-v5 branch inherits and validates complete heads.
- the inherited completion marker is content-scoped while ref-incarnation
validation is performed separately.
### 12.4 Cost gates
At fixed table count and increasing commit depth, assert flat curves for
manifest reads, bytes, fragments/pages, and decoded rows:
- local FS, compacted and uncompacted;
- S3/RustFS with real e_tags;
- warm repeated read;
- cold operation-local open with shared Session;
- one uncovered-head update before reconciliation;
- reconciled steady state.
The test must fail if the lookup silently falls back to scanning journal
history in the claimed steady state.
The decode term is part of the gate: parsing head rows — including the typed
JSON `TableHeadMetadata` payload — must be bounded by catalog width. A
per-read parse cost that grows with anything other than table count fails the
gate even when I/O is flat.
### 12.5 Format guards
- exact v5 head metadata schema and object IDs;
- one head row per stable identity;
- incarnation-baseline and identity-bearing journal event schemas;
- content-scoped branch completion marker schema/digest;
- RFC-023 PK metadata on node and edge tables when the release combines them;
- v5 publisher source always pairs a journal/tombstone event with a head row.
## 13. Decisions and open gates
### Decided
- Heads and journal share one `__manifest` transaction.
- Current reads use heads; historical reads keep the journal.
- Heads represent `live | tombstoned` plus incarnation explicitly.
- A separate heads dataset and a mutable in-process tip authority are rejected.
- Migration is offline, all-branch, resumable, and stamps v5 last.
- RFC-023 PK activation is verified by v5 only when deliberately co-released.
- Checkpoint retention is deferred.
### Open ship gates
1. Rename-stable table/type identity and final stable-ID encoding — owned by
this RFC; consumed by RFC-023 and RFC-025 (§3.1).
2. The in-manifest indexed lookup implementation and bounded uncovered-tail
proof.
3. Passing local and object-store depth-slope cost gates.
4. Atomic cross-process migration-claim implementation and operator runbook.
5. Final v5 metadata JSON/typed-column compatibility review.
6. Full all-branch migration/failpoint matrix.

444
docs/rfcs/rfc-025-checkpoint-retention.md Normal file
View file

@ -0,0 +1,444 @@
---
type: spec
title: "RFC-025 — Checkpoint-pinned retention"
description: Makes named graph checkpoints authoritative retention roots, materializes them as Lance-native manifest and data-table tags, and defines crash-safe reconciliation and cleanup ordering on the RFC-022 unified write path.
status: draft
tags: [eng, rfc, retention, checkpoint, cleanup, manifest, lance, omnigraph]
timestamp: 2026-07-10
owner:
---
# RFC-025 — Checkpoint-pinned retention
**Status:** Draft / for team review
**Date:** 2026-07-10
**Depends on:** [RFC-022](rfc-022-unified-write-path.md)'s publisher and
recovery-sidecar protocol. It composes with
[RFC-024](rfc-024-durable-table-heads.md), but is not part of v5 by implication.
**Surveyed:** omnigraph 0.8.1; Lance 9.0.0-beta.15 (`f24e42c1`)
**Audience:** engine, storage, CLI, and operations maintainers
**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md).
Findings marked **BLOCKER** must be dispositioned before acceptance.
---
## 0. Decision
OmniGraph adopts checkpoint-pinned retention. A checkpoint is a durable, named
graph snapshot: one reference set in the reserved main-manifest registry
containing every physical manifest/table lineage and version needed to
reconstruct that graph state.
The checkpoint rows are the **logical authority**. Lance tags are the
**physical pins** that make that authority effective. This distinction is
load-bearing: Lance `cleanup_old_versions` protects Lance tags and branches; it
does not inspect OmniGraph rows in `__manifest`. A checkpoint row without the
corresponding tags would therefore promise time travel to versions that Lance
is free to delete.
The safe ordering is asymmetric:
- create physical tags first, publish checkpoint authority last;
- tombstone checkpoint authority first, reclaim physical tags last.
Every crash window consequently over-retains. None under-retains.
## 1. Scope and non-goals
This RFC specifies:
1. checkpoint and GC-boundary manifest rows plus their own format activation;
2. deterministic Lance tags for the pinned manifest and every pinned table version;
3. create, delete, and reconciliation protocols;
4. how `cleanup` computes and protects its root set;
5. CLI, policy, audit, observability, migration, and acceptance contracts.
It does not change snapshot isolation, invent a second transaction log, or
replace Lance cleanup. It does not make every historical graph commit a
checkpoint. History outside branch heads, named checkpoints, and the explicit
retention window remains eligible for collection after operator confirmation.
## 2. Sources of truth
### 2.1 Logical authority: manifest rows
A checkpoint is immutable after creation. Retargeting a name means deleting the
old checkpoint and creating a new checkpoint ID; a name never silently moves.
The stable name-reservation row carries `state = live | tombstoned` and a
monotonic `generation`. Reuse is a CAS transition from the exact tombstoned
generation to `live` with `generation + 1` and a fresh checkpoint ID. Thus a
deleted name may be deliberately reused, but two concurrent re-creations cannot
both win.
All checkpoint authority rows live on the reserved **main** branch of
`__manifest`, regardless of the logical branch being checkpointed. This is a
global registry, not branch-local graph state: deleting the source branch must
not delete the authority that protects its snapshot. One main-manifest publish
writes:
- `checkpoint_name:<normalized-name>` — unique name reservation pointing at the
immutable checkpoint ID;
- `checkpoint:<checkpoint_id>` — header containing `name`, source logical
branch, `graph_commit_id`, source physical manifest branch and version,
manifest schema stamp, `created_at`, and `actor`;
- `checkpoint_table:<checkpoint_id>:<stable-table-id>:<incarnation-hash>` — one
row per pinned table containing stable table identity, table key,
`table_path`, `table_branch`, `table_version`, and physical incarnation.
The name reservation, header, and every table row land in one RFC-022 publisher
CAS on main. First creation is insert-only; reuse requires the exact tombstoned
reservation generation in the `ReadSet`. Two concurrent attempts for the same
normalized name therefore conflict even when they captured different source
branches. A missing or duplicate table row is an invalid checkpoint, not a
partial checkpoint.
Stable table identity/incarnation is a ship gate even when RFC-024 has not
landed; checkpoint rows cannot key retention to mutable names. A deployment may
reuse RFC-024's identity baseline, but durable heads themselves are not a
dependency.
Deletion changes the name reservation from `live` to `tombstoned` and writes
manifest tombstones for the checkpoint header and all table rows in one CAS.
The reservation and every tombstone carry the same fresh
`delete_operation_id`; that CAS is the durable delete/recovery marker. The
retention claim may repeat the operation ID while the writer is live, but it is
only a fence, never recovery authority. Checkpoint IDs are never reused.
Checkpoint create/delete are manifest metadata transactions. They do not create
a graph-content commit or move `graph_head`: taking a checkpoint must not change
the graph state it names. They still use RFC-022's read-set arbitration,
manifest CAS, actor, and audit contracts. Creation uses a recovery sidecar for
its pre-CAS tag effects; deletion is authority-first and embeds its idempotent
operation marker in the tombstone CAS instead of writing a generic sidecar.
The named snapshot is the source branch version and graph commit pinned in step
2 of creation, not whatever is current when the later main-registry CAS lands.
That immutable version may remain a valid checkpoint if the source branch
advances after capture. The response always returns the exact captured commit
and manifest version; it never implies that the checkpoint tracks a moving tip.
### 2.2 Physical protection: Lance tags
Every checkpoint owns deterministic Lance tags of two kinds:
```text
ogcp_<checkpoint-id-base32>_manifest_<branch-hash>
ogcp_<checkpoint-id-base32>_<table-and-lineage-hash>
```
The manifest tag targets the exact `(manifest_branch, manifest_version)` named
by the header. Each table tag targets the exact `(table_branch, table_version)`
recorded by its table row. The spelling stays within Lance tag-name constraints
and includes enough physical-lineage identity to avoid collisions across lazy
forks. Pinning `__manifest` itself is mandatory: data-table tags alone cannot
preserve the schema, registrations, and lineage needed to reconstruct the graph
snapshot.
Internal `ogcp_` tags are reserved. User-supplied tooling must not create,
move, or delete them. If an existing deterministic tag points anywhere else,
checkpoint creation and cleanup fail loudly with a typed corruption error.
Tags are derived state. A live checkpoint row with a missing tag is repaired
from the row. An internal tag with neither a live checkpoint nor an in-flight
checkpoint sidecar is orphaned and may be removed.
Tags protect versions from `cleanup_old_versions`; they do **not** protect a
named branch from Lance branch deletion, which removes `tree/<branch>`. The
initial contract therefore refuses physical deletion of any manifest or data
branch named by a live checkpoint row. Graph branch deletion and orphan-branch
reclamation read the main checkpoint registry under the retention claim and
return `CheckpointPinsBranch` with the blocking checkpoint IDs. They never call
`force_delete_branch` on a pinned lineage. Operators delete the checkpoints
first; hidden checkpoint branches or full snapshot copies require a separate
design.
### 2.3 Cross-process retention claim
Checkpoint create/delete, pin reconciliation, destructive cleanup, and graph
branch deletion serialize through one graph-wide, substrate-backed retention
claim. Acquisition uses the storage adapter's atomic create-if-absent primitive
(`PutMode::Create`) on `__leases/retention.json`; Lance branch creation is
explicitly rejected because its branch-metadata step is an existence check
followed by an unconditional put.
The claim payload records operation ID, action, actor, creation time, and a
random fencing/owner token. Every protected phase re-reads and verifies that
token. Normal release verifies the exact token before deleting the claim; no
other owner can replace it while create-if-absent observes the object. There is
no time-based lease stealing. Crash takeover requires an explicit fleet write
outage, classification of the sidecar/manifest operation marker, removal of the
stale claim, and acquisition with a new token. A resumed stale process must
fail its next token check.
A process-local mutex is only a contention optimization. The retention claim is
held across tag creation or physical cleanup, preventing this race: cleanup
computes roots, another process captures an untagged old version, then cleanup
deletes it before the tag lands. A crash leaves the claim and therefore
over-retains. Takeover first classifies the prior operation from its recovery
sidecar and/or exact manifest operation marker, then may release the claim
only after that operation is completed or safely aborted. It never steals on
elapsed wall time alone.
The claim is not presented as a distributed reader/writer lock for arbitrary
graph writes. Initial destructive cleanup is an **offline** operation: operators
quiesce every server, CLI, embedded writer, and MemWAL stream before acquiring
the claim. New binaries also refuse to start a write while the claim exists,
but that check is defense in depth rather than proof that an already-running
foreign writer drained. Online cleanup requires a separate fleet writer-epoch
or substrate lease design and is out of scope.
## 3. Checkpoint create protocol
Checkpoint creation is a writer on the RFC-022 pipeline:
1. Run and await RFC-022's synchronous recovery barrier.
2. Authorize, capture one immutable source-branch manifest snapshot and graph
commit, and prepare the complete header/table/tag set plus a `ReadSet` that
includes source branch incarnation, source manifest version, format/schema
identity, applicable GC boundaries, and the main-registry name reservation;
a source at or behind a pruned-through boundary is refused even if its files
happen to remain.
3. Acquire the retention claim, checkpoint/cleanup process-local serialization,
source branch-control gate, and adapter-declared metadata gates in RFC-022's
canonical order. Ordinary data queues are not held merely because immutable
versions are being tagged.
4. Freshly revalidate the complete `ReadSet`. A changed branch incarnation,
missing tag target, conflicting name, or format change releases the gates and
restarts before any tag exists.
5. Write a generic recovery sidecar containing the intended rows and tags.
6. Create the manifest tag and every table tag idempotently. Existing correct
tags are no-ops; conflicting tags fail.
7. Verify every tag, then release source branch control. A source-head advance
is harmless because the tags already pin the exact captured version;
source-branch deletion takes the same retention claim, classifies overlapping
create sidecars, and then refuses if the published checkpoint pins the ref.
8. Publish the name/header/table rows in one CAS on the main manifest registry.
9. Delete the recovery sidecar. Sidecar deletion remains best-effort after the
authority publish, as in the existing write protocol.
10. Release the retention claim after the outcome is durably classifiable.
A crash before step 8 leaves tags but no checkpoint. Recovery may complete the
publish when every precondition still holds or remove the orphan tags. A crash
after step 8 leaves a valid checkpoint even if sidecar cleanup did not finish.
## 4. Checkpoint delete protocol
Deletion deliberately reverses the create order:
1. run the recovery barrier, authorize, and prepare the complete checkpoint plus
name/format/registry `ReadSet`;
2. acquire the retention claim and local gates in canonical order, then freshly
revalidate the complete checkpoint;
3. publish the tombstoned name reservation plus header/table tombstones, all
carrying one fresh `delete_operation_id`, in one manifest CAS;
4. release the claim and acknowledge once the authority change is durable;
5. delete the deterministic Lance tags asynchronously and idempotently.
A failed tag deletion only retains extra data. The checkpoint is already gone
from the user-visible namespace. The reconciler and the next `cleanup` retry
the physical reclaim.
Delete is an RFC-022 authority-first metadata workflow: no independently durable
effect precedes the atomic tombstone CAS, and later tag deletion is derived,
over-retaining cleanup. It therefore needs no generic write sidecar. The
`delete_operation_id` persisted by the tombstone CAS lets crash recovery
distinguish pre-CAS from post-CAS state exactly; the retention claim does not.
## 5. Pin reconciler
Read-write open and every destructive `cleanup` run reconcile checkpoint pins
from the main-manifest checkpoint registry.
The reconciler:
1. reads the live checkpoint rows once;
2. classifies checkpoint sidecars before touching apparently orphaned tags;
3. creates or verifies every required tag;
4. deletes internal tags proven to have no live or in-flight authority;
5. emits a typed result for repaired pins, reclaimed pins, and failures.
`cleanup` is fail-closed: any missing, conflicting, unreadable, or ambiguous
pin aborts deletion for the affected graph. It never treats reconciliation
failure as permission to collect data.
The reconciler runs under the retention claim and is serialized with
checkpoint creation and table maintenance.
It may run concurrently across independent graphs, but it must not delete a
tag belonging to a foreign process's live create sidecar.
## 6. Cleanup protocol and pruned-through boundary
The cleanup root set is:
- every live graph branch head;
- every live checkpoint manifest and table row;
- every version inside the operator-selected time/count retention window;
- versions Lance itself protects through non-OmniGraph tags or branch refs.
Read-only preview may run without a claim, but it is explicitly provisional.
Confirmed execution uses this order:
1. establish the operator write outage, persistently seal/fold streams through
RFC-026 **before** taking the retention claim, and complete the RFC-022
recovery barrier;
2. acquire the retention claim, then revalidate the fleet outage, every stream's
`SEALED` cut, and absence of recovery sidecars;
3. run pin reconciliation and recompute the exact root set and per-dataset
cutoffs under the claim;
4. prepare and revalidate a complete RFC-022 `ReadSet` containing format/schema
identity, branch incarnations, current manifest table entries/heads, prior GC
boundaries, and the root digest;
5. publish mutable `gc_boundary:<dataset-lineage-hash>` rows containing cutoff,
root digest, cleanup operation ID, and timestamp in one reserved-main
manifest CAS;
6. invoke Lance `cleanup_old_versions`, relying on the verified tags to retain
sparse checkpoint versions;
7. record per-table removal statistics/failures and release the claim only when
the operation is durably resumable or complete.
Step 5 is an RFC-022 authority-first metadata transaction: it changes which
versions recovery may select, carries no graph-content lineage, and needs no
sidecar because no physical effect precedes its CAS. Step 6 is RFC-022 §8
physical maintenance under the already-published boundary. The two are not one
indistinguishable “cleanup commit.”
The boundary advances before delete. Any later recovery, restore, or publisher path
that could make an older physical version current must include the boundary in
its RFC-022 `ReadSet` and revalidate it before the physical effect. A position
at or behind the boundary is retried from current state or refused. Cleanup
never starts with an armed recovery that may still need an older version.
Per-table cleanup remains fault-isolated, but partial operational success is
reported explicitly. A successful table does not hide another table's error.
The claim metadata plus `gc_boundary` operation ID is the cleanup recovery
marker: takeover resumes or reports the remaining table units under the same
root digest before releasing the claim; it does not silently recompute a
broader destructive plan after a partial run.
### 6.1 Operational cadence — the cost of never running this
Offline-only destructive cleanup has a predictable failure mode: operators
defer it indefinitely, and the graph silently reacquires the unbounded-history
cost class this program measured (per-version chains that grow one entry per
commit; latest-version listings whose page size grows with history; on hosted
deployments, seconds of added latency per operation). The docs and `cleanup`
preview therefore state the deferral cost explicitly, and deployments are
expected to schedule a periodic maintenance window — for continuously written
graphs, the same cadence discipline as `optimize` — rather than treating
cleanup as exceptional. Read-only preview and pin reconciliation remain online
so drift is visible between windows.
Online destructive cleanup — concurrent with live writers under a fleet
writer-epoch or substrate lease — is the named successor design, deliberately
out of scope here (§2.3). This RFC's offline barrier is the correct first
delivery, not the end state; a deployment for which write outages are
unacceptable should treat the successor design as the gating requirement for
adopting aggressive retention policies.
## 7. Public and policy surface
Initial delivery is engine plus direct-storage CLI, matching `cleanup`:
```text
omnigraph checkpoint create <name> [--branch <branch>] <store>
omnigraph checkpoint list <store>
omnigraph checkpoint show <name-or-id> <store>
omnigraph checkpoint delete <name-or-id> <store>
```
JSON output includes checkpoint ID, name, actor, branch, graph commit, creation
time, table count, and pin-health status. `cleanup` preview reports which branch,
checkpoint, or window protects each retained version and states that execution
requires a graph-wide write outage.
Checkpoint creation and deletion use dedicated Cedar actions. Embedded callers
hit the same engine gate as the CLI. HTTP management endpoints are out of scope
until server-side maintenance has a general design; no transport-only bypass is
introduced here.
## 8. Migration and compatibility
This RFC owns its own internal-format activation. RFC-022 authorizes no format
bump, and RFC-024 explicitly excludes checkpoint rows from its heads format.
The retention format may receive the next schema version after heads, or the
two may share one release only if both RFCs are independently accepted and
RFC-024 is amended before implementation. A fresh activated graph starts with
no named checkpoints and a zero pruned-through boundary. No older commit is
silently promoted into a checkpoint.
The upgrade mechanism must be chosen before implementation:
- an in-place upgrade preserves branches and history and must define quiescence,
crash recovery, stamp ordering, and rollback;
- export/import creates a fresh graph at the new format but loses the old commit
DAG, branches, snapshots, and time-travel history. Those losses must be shown
in the plan and confirmation output; there is nothing left to backfill as a
checkpoint.
Mixed-version writers are unsupported. An old binary must not write after the
retention format stamp or tag protocol becomes authoritative.
## 9. Observability and bounds
Expose:
- live checkpoint count and pinned table-version count;
- missing, conflicting, repaired, and orphan-tag counts;
- oldest checkpoint age and retained bytes when Lance reports them;
- current GC boundary and root digest per dataset lineage;
- cleanup versions/bytes removed and per-table failures;
- checkpoint create/delete/reconcile latency and retry counts.
Checkpoint enumeration and cleanup planning must be bounded by live checkpoints
times catalog width, not total commit history. Operators can limit checkpoint
count by policy; the engine refuses names or payloads beyond configured bounds
before creating tags.
That is a physical-I/O claim, not just a logical row-count claim. Registry
lookup must use a structured Lance access path with a measured bound on
uncovered fragments, reusing RFC-024's in-manifest scalar-index work when it is
available or proving an equivalent access shape here. A filtered scan that
still reads history-sized manifest fragments does not pass this RFC's cost gate;
a separate checkpoint dataset is rejected because its authority rows could not
share the main-manifest CAS.
## 10. Acceptance gates
- A checkpoint on an old sparse version survives cleanup while adjacent
unpinned versions are removed.
- Branch delete and orphan reclamation refuse a non-main manifest/data lineage
while a live checkpoint references it; after checkpoint deletion and tag
reconciliation, branch deletion succeeds. `force_delete_branch` is never used
to bypass the pin.
- The source `__manifest` version survives cleanup; data-table tags without the
manifest tag fail the checkpoint-validity test.
- Concurrent creation of the same normalized name yields exactly one authority
record; the losing attempt leaves only safely reclaimable tags.
- After deletion, concurrent attempts to reuse the tombstoned name generation
yield exactly one fresh checkpoint ID; the old checkpoint ID never revives.
- Create failpoints cover sidecar, tag, and manifest boundaries; delete
failpoints cover authority CAS, claim release, and tag reclaim. Every outcome
converges to valid pins or safe over-retention.
- Missing live tags are repaired before cleanup; conflicting tags block cleanup.
- The retention claim blocks checkpoint create/delete/reconcile and branch delete
during cleanup on local FS and S3; destructive cleanup refuses active streams
or recovery sidecars.
- Two-process races prove the atomic retention claim, not a process-local gate,
closes checkpoint-vs-branch-delete windows; crash takeover never relies on
wall-clock expiry alone.
- Local and S3 claim tests prove `PutMode::Create` admits exactly one owner and
that mismatched owner tokens cannot release another operation's claim.
- A documented fleet write-outage test runs writers before and after cleanup;
online writer-vs-cleanup concurrency is explicitly not advertised.
- Genuine cross-version tests cover the selected upgrade path.
- Cost tests use `helpers::cost` at realistic commit depth and prove that a
steady checkpoint list/lookup and cleanup plan do not grow with commit count
once physical layout is held constant, including a bounded uncovered tail.
## 11. Phasing
| Phase | Content | Gate |
|---|---|---|
| A | row schemas, engine DTOs, format activation, deterministic tag encoding | schema and tag surface guards |
| B | create sidecar, delete-operation fields in the authority CAS, and reconciler | crash matrix; sparse-pin correctness |
| C | offline cleanup integration and GC boundary enforcement | quiescence/refusal tests; cost budgets |
| D | CLI, policy, audit, docs, and selected migration path | CLI outputs; genuine upgrade test |

415
docs/rfcs/rfc-026-memwal-streaming-ingest.md Normal file
View file

@ -0,0 +1,415 @@
---
type: spec
title: "RFC-026 — MemWAL streaming ingest"
description: Adopts Lance MemWAL as OmniGraph's strategic streaming-write architecture, with durable per-row acknowledgement, graph-atomic folds, epoch-fenced quiescence, and explicit fresh-read cuts on the RFC-022 unified write path.
status: draft
tags: [eng, rfc, streaming, ingest, wal, memwal, lance, omnigraph]
timestamp: 2026-07-10
owner:
---
# RFC-026 — MemWAL streaming ingest
**Status:** Draft / for team review
**Date:** 2026-07-10
**Depends on:** [RFC-022](rfc-022-unified-write-path.md)'s unified write and
generic recovery-sidecar protocol, plus
[RFC-023](rfc-023-key-conflict-fencing.md) for the initial keyed graph-stream
mode. Durable heads from
[RFC-024](rfc-024-durable-table-heads.md) are compatible but not required.
**Surveyed:** omnigraph 0.8.1; Lance 9.0.0-beta.15 (`f24e42c1`); complete MemWAL format specification
**Audience:** engine, server, CLI, policy, and operations maintainers
**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md).
Findings marked **BLOCKER** must be dispositioned before acceptance.
---
## 0. Decision and risk posture
OmniGraph adopts Lance MemWAL as its strategic streaming-write architecture.
MemWAL is a major Lance architectural bet: a sharded LSM write path with durable
WAL entries, flushed Lance generations, merge progress committed with base-table
data, maintained indexes, and epoch-fenced writers. OmniGraph consumes that
architecture rather than building a WAL, shard protocol, or LSM reader.
This RFC does **not** characterize the architecture as experimental. The risk is
narrower: Rust API names, some format details, and operational helpers are still
maturing across Lance releases. We manage that API/format-maturity risk with a
small adapter, compile/runtime surface guards, a quiescence requirement before
Lance upgrades, and a fresh full-spec alignment audit on every bump. It is not a
reason to fork or reimplement MemWAL.
The contract is:
- stream acknowledgement means the row's WAL entry is durable;
- acknowledgement does not mean graph visibility;
- default queries see only the manifest-committed graph;
- a fold is an ordinary RFC-022 graph writer and is the sole visibility point;
- fresh reads are explicit and never claim cross-table atomicity.
## 1. Scope and non-goals
This RFC specifies enrollment, the public stream API, acknowledgement semantics,
folding, fold-time integrity, dead-letter atomicity, branch/schema quiescence,
fresh-read cuts, resource bounds, observability, testing, and upgrade posture.
It does not replace `load` or `mutate`, provide cross-query transactions, store
manifest mutations in MemWAL, create a second metadata authority, or weaken
default snapshot isolation. Stream-mode
deletes remain out of the first delivery and require the Lance tombstone surface
plus a separate acceptance pass.
## 2. Stream mode and key semantics
Initial delivery exposes
`@stream(mode="upsert", on_reject="strict")` on a node or edge type;
`on_reject` accepts `strict` or `dead_letter`.
It requires the table's immutable unenforced primary key to equal OmniGraph's
merge key: `id` for nodes and edges. All occurrences of one key map to one shard
and MemWAL applies last-write-wins ordering.
Public append mode is deliberately out of scope. Nodes and edges always have
logical identity; allowing a retry to append the same `id` twice would violate
that contract. A future explicitly keyless, non-graph append-only table class
may consume MemWAL append semantics under its own schema/API decision.
Stream ordering intentionally differs from the interactive fence:
- concurrent interactive same-key writes serialize or fail/retry loudly;
- same-key stream entries resolve by MemWAL generation/position order;
- duplicate keys inside one bulk-load input retain the existing load error.
The schema and user docs state all three together.
## 3. Enrollment is a recoverable inline commit
Schema apply records `@stream` intent only. First stream use enrolls the physical
table by creating the singleton `__lance_mem_wal` system index and its sharding
configuration.
Enrollment advances Lance HEAD inline. It therefore uses the RFC-022 generic
recovery protocol, not an ad-hoc state machine:
1. run and await RFC-022's synchronous recovery barrier;
2. authorize, pin the manifest/schema/table state, and prepare a complete
`ReadSet` containing schema identity, table entry/head, PK metadata, stream
intent/configuration, and lifecycle-row absence;
3. acquire any global claims and then the `(table, branch)` write queue in
RFC-022 order, then freshly revalidate the complete `ReadSet`; a mismatch restarts
before any inline effect;
4. verify RFC-023's already-installed PK; enrollment never performs a first-use
PK migration; validate the sharding configuration;
5. write a generic sidecar with writer kind `stream_enrollment` and pre-commit
table pin;
6. create the MemWAL index, advancing Lance HEAD;
7. publish the new table version plus `stream_state = OPEN` in one manifest CAS,
including the table-head row when RFC-024 is active;
8. delete the sidecar best-effort after publication.
Recovery rolls the enrollment forward or back under the same classification and
audit machinery as other inline residuals. Repeating enrollment with identical
metadata is a no-op. A different PK, sharding spec, maintained-index set, or
writer-default configuration is a typed conflict.
No row is acknowledged until enrollment is manifest-committed.
Initial delivery supports one unsharded shard per `(table, main)`. Non-main
branches remain refused until the Lance branch-scoping question is proven by a
surface guard and end-to-end test. Later `bucket(id, N)` sharding must preserve
the one-key-to-one-shard rule.
## 4. New public API
The shipped `POST /graphs/{id}/ingest` path remains the deprecated, compatible
alias of `/load`. Streaming receives a new, non-conflicting surface:
```text
POST /graphs/{graph_id}/streams/{type_name}/ingest?branch=main
GET /graphs/{graph_id}/streams
GET /graphs/{graph_id}/streams/{type_name}
POST /graphs/{graph_id}/streams/{type_name}/fold
POST /graphs/{graph_id}/streams/{type_name}/quiesce
POST /graphs/{graph_id}/streams/{type_name}/resume
```
The ingest request and response use `Content-Type: application/x-ndjson` and
`Accept: application/x-ndjson`.
Each input line is one row payload. Each output line corresponds to the same
input ordinal:
```json
{"ordinal":17,"status":"durable","shard_id":"...","writer_epoch":8,"wal_position":42}
```
Synchronous validation failures return a per-row error before a WAL append.
Previously acknowledged rows in the same request remain durable; the response
is a stream, not an all-request transaction. Ordering, cancellation, and retry
rules are explicit:
- acknowledgements are emitted in input order for one HTTP stream;
- disconnecting does not cancel entries whose durability waiter resolved;
- a missing response is ambiguous; retrying the same `id` and payload may add
another WAL entry but produces the same last-write-wins graph state;
- server shutdown stops admission, drains durability waiters up to a bound, and
reports any unacknowledged tail as unknown to the client.
CLI commands mirror the new namespace rather than overloading deprecated
`omnigraph ingest`:
```text
omnigraph stream ingest <type> --data <ndjson> ...
omnigraph stream status [<type>] ...
omnigraph stream fold [<type>] ...
omnigraph stream quiesce [<type>] ...
omnigraph stream resume [<type>] ...
```
Every endpoint has a dedicated OpenAPI operation and handler tests. Ingest
passes the engine `stream_ingest` Cedar action and per-actor admission
accounting before acquiring a shard writer; fold/quiesce/resume use a separate
`stream_manage` action. Status is authorized like other graph operational
metadata. The same engine gates apply to embedded and remote CLI use.
## 5. Ack-path validation and writer lifecycle
Before append, OmniGraph applies checks that need no base-table read: Arrow
shape/type, required/default fields, enum/range/check constraints, reserved
columns, and stream mode. RI, cardinality, cross-version uniqueness, and
external embedding computation remain fold-time work.
One warm `ShardWriter` is held per active shard behind a bounded registry. The
registry has idle eviction and hard limits for resident writers, MemTable bytes,
unflushed WAL bytes, pending generations, and per-actor inflight bytes. Exceeding
a bound backpressures with a typed retryable response; it never drops a row.
Initial topology has one active ingest owner for each `(graph, table, main)`
shard. MemWAL's epoch fence makes restart/failover safe; it is not a load
balancer. A deployment with multiple server replicas must route a shard to its
current owner (or return a typed retry/redirect) instead of letting replicas
reclaim the epoch per request. General multi-owner routing waits for the
multi-shard phase and its ownership protocol.
## 6. Fold protocol
The fold consumes flushed generations in ascending order. Embeddings and
base-dependent validation run outside the table queue and register every probed
table plus stream configuration/generation in RFC-022's `ReadSet`. The commit
phase then:
1. stages accepted rows with Lance merge-insert and includes
`merged_generations` in that transaction;
2. stages any rejection/audit rows required by §7;
3. acquires every affected queue in canonical sorted order and revalidates the
complete `ReadSet`; any mismatch discards and replans the whole fold;
4. writes one generic RFC-022 recovery sidecar before the first
`commit_staged` call;
5. commits every staged Lance transaction;
6. publishes all data/internal table versions and lineage in one `__manifest`
CAS, including table heads when RFC-024 is active;
7. deletes the sidecar after successful publication.
The sidecar is mandatory even though merge-insert is staged. After
`commit_staged`, Lance HEAD and `merged_generations` have moved while the graph
manifest has not. A failure in that window is the ordinary multi-table recovery
gap, not invisible staged state.
MemWAL generation GC starts only after the exact fold is graph-visible, its
sidecar is resolved, index catchup permits reclamation, and no `FreshReadCut`
retention guard references the generation. Data-HEAD merge progress alone is
never permission to delete the only fresh-tier copy.
Concurrent folders reload `merged_generations`: a generation already committed
is skipped; otherwise the fold is replanned from current state. A forced fold
stops new flush creation for its cut, waits for in-flight durability waiters,
and never aborts an acknowledged row.
Fold lineage uses `omnigraph:ingest` as the mechanism actor and persists the
authenticated contributor actor for every folded WAL range in the same internal
audit participant. Commit and status output can therefore answer both who ran
the fold and who supplied the data after WAL GC.
## 7. Fold-time rejection is atomic
`strict` is the default. A permanent RI, cardinality, uniqueness, or embedding
failure stops the fold at the offending generation, marks the shard blocked,
and backpressures new ingestion once configured lag bounds are reached.
`dead_letter` is explicit. `_ingest_rejects` is then a versioned internal Lance
table and a participant in the same fold pipeline, not best-effort state outside
the commit protocol. Every reject has deterministic identity:
```text
(table_key, shard_id, generation, wal_position)
```
The fold stages reject rows, accepted rows, and merge progress before committing
any of them. The generic sidecar covers every participant; the single manifest
publish records both the base-table and reject-table versions. Replay is
idempotent by reject identity. There is no ordering in which progress can become
visible while the corresponding rejection is lost.
`stream status` reports blocked generations and typed reject details. Reject
retention is explicit and cannot be shorter than the WAL/fold audit retention
needed to explain a durable acknowledgement.
## 8. Epoch-fenced quiescence barrier
Branch operations, schema changes, stream teardown, and Lance upgrades require a
real barrier, not an empty check.
Each enrolled table has a durable
`stream_state:<stable-table-id>:<incarnation>` row in its manifest branch with
`OPEN | DRAINING | SEALED`, configuration hash, and epoch floor. The row is the
logical lifecycle authority and is updated by an RFC-022 CAS; MemWAL shard
epochs are the physical writer fence. Neither an in-memory registry nor an
empty-generation observation can substitute for both.
Lifecycle-only transitions are audited manifest metadata transactions; they do
not create graph-content commits or move `graph_head`.
The shared drain sequence is:
1. publish stream intent `OPEN -> DRAINING` for the target table/branch;
2. increment and persist each shard writer epoch and seal claims, fencing stale
writers across processes;
3. reject or backpressure new appends;
4. wait for in-flight durability waiters, flush active MemTables, and fold every
generation to empty;
5. verify shard manifests and base `merged_generations` agree on emptiness;
6. publish `DRAINING -> SEALED` with the verified generation/epoch cut;
7. for an operation-scoped drain, perform the guarded operation; persistent
public quiesce stops after step 6.
Each lifecycle CAS is an RFC-022 authority-first metadata write; each fold is a
separate normal RFC-022 graph write. `DRAINING` fully encodes the target epoch
floor, so an interrupted epoch mutation is idempotently resumed from that row.
The sequence is not one giant sidecar spanning multiple commits.
There are two dispositions after the drain reaches `SEALED`:
- **operation-scoped drain** — branch/schema maintenance automatically publishes
`SEALED -> OPEN` with a newer epoch only after the guarded operation succeeds
and the stream contract remains compatible;
- **persistent quiesce** — the public `quiesce` command leaves the stream
`SEALED`. It never auto-reopens. `stream resume` explicitly revalidates schema,
PK, configuration, MemWAL format, and epoch, then publishes a newer `OPEN`
state. Stream teardown deletes intent only from `SEALED`.
The barrier never holds the table write queue while waiting for a fold that
needs that queue. State transition and epoch fencing happen first; fold commit
then acquires the normal queue. Crash recovery resumes from the durable state
and epoch.
Schema apply must drain every affected enrolled type before changing fields,
constraints, PK, embeddings, or `@stream` and resumes only when compatible.
A Lance version upgrade requires persistent `stream quiesce --all`, but empty
generations alone are insufficient: the MemWAL system index, shard manifests,
epoch records, and generation directories may still use the old format. Before
the bump, the implementation must prove one of: (a) upstream guarantees and
cross-version tests cover every retained MemWAL artifact, (b) a public Lance
metadata migration converts them, or (c) OmniGraph tears down the enrolled
MemWAL metadata under recovery and re-enrolls after the bump. Without one of
those gates the upgrade refuses; `resume` never opens unverified old metadata.
## 9. Fresh-read cuts
Freshness is a first-class engine/IR enum:
```text
Committed
Fresh
```
At query planning, `Fresh` captures one `FreshReadCut` containing:
- the ordinary manifest snapshot;
- each selected shard-manifest version and writer epoch;
- included flushed-generation paths and maximum generation;
- the active same-process MemTable row-position watermark, when available;
- the base table's `merged_generations` and index-catchup state read from the
exact table version selected by the manifest snapshot, never from live HEAD.
Capture uses a retrying handshake:
1. read the selected shard manifests/epochs, acquire Lance generation retention
guards for the flushed files in the tentative cut, and under one
same-process writer snapshot capture/pin any active-MemTable watermark;
2. pin the graph manifest snapshot and read `merged_generations` from each exact
base-table version it selects;
3. re-read the shard manifest versions/epochs; any epoch/configuration change
restarts the whole capture;
4. if a generation from step 1 disappeared, accept that only when the pinned
base's `merged_generations` proves it is included; otherwise release guards,
discard the whole graph snapshot, and retry from step 1;
5. exclude generations that appeared after step 1 and hold the generation and
MemTable read guards captured in step 1 until query completion.
If Lance exposes no guard that prevents generation GC for the query lifetime,
cross-process `Fresh` does not ship. A missing generation is never interpreted
as “probably folded” against an older pinned base.
Execution never refreshes that cut mid-query. It excludes every flushed
generation `<= merged_generations[shard]`; otherwise old WAL data could outrank
or duplicate its newer base-table image.
Fresh reads have no cross-table atomicity. Same-process active MemTables provide
read-your-writes; other processes can promise only the latest flushed state
captured by their shard-manifest reads. The HTTP request and query docs state
those limits wherever the tier is exposed.
## 10. Observability and resource contracts
Per shard expose durable WAL position, replay position, active epoch, current
generation, flushed and merged generation, index catchup, pending rows/bytes,
oldest acknowledged age, last fold error, blocked reject, and quiescence state.
Metrics cover ack latency, durability-wait batching, fenced writers, replayed
entries, fold rows/bytes/generations, fold retries, lag, reject counts, and
sidecar recovery. Defaults for every byte/count/time bound are documented and
configuration changes are observable behavior.
`stream status` resolves the exact lifecycle rows and MemWAL metadata through a
structured, bounded access path; it may reuse RFC-024's scalar-index machinery
but cannot claim history-flat cost while scanning manifest history.
## 11. Acceptance gates
- Surface guards pin claim, append, durability waiter, flush, epoch fencing,
staged merge with `merged_generations`, index catchup, and seal/reopen APIs.
- A WAL append failure emits no durable acknowledgement. Every acknowledged row
survives crash, replay, fold, and recovery.
- Failpoints cover enrollment's inline-commit gap and every fold participant
around sidecar, `commit_staged`, reject persistence, and manifest publish.
- Enrollment restarts without an inline effect when schema, PK, table head, or
stream configuration changes between prepare and gated revalidation.
- Two folders converge exactly once; a fenced stale writer can never produce a
false durable acknowledgement after WAL GC.
- Two server replicas do not epoch-ping-pong one shard; owner failover fences
the old process before the new owner acknowledges.
- Quiescence tests race appends with branch and schema operations across two
coordinators and prove no post-drain tail appears; failpoints cover every
lifecycle-row, epoch-fence, fold, and `SEALED` boundary.
- Persistent quiesce never auto-reopens; explicit resume validates a newer
epoch. Upgrade tests cover every retained MemWAL artifact through declared
compatibility, migration, or teardown/re-enrollment.
- Fresh reads race capture with fold/GC, retry on an unexplained disappearing
generation, hold generation/MemTable guards through execution, exclude merged
generations, and document cross-table inconsistency explicitly.
- Server/OpenAPI tests preserve the old `/ingest` alias and cover the new route;
CLI parity covers embedded and remote stream commands.
- Ack-path object-store operations are O(1) and flat in graph history and WAL
depth. Fold cost is bounded by generations/rows folded, not graph history.
- S3 correctness runs against RustFS; API/format guards are rerun before every
Lance bump.
## 12. Phasing
| Phase | Content | Gate |
|---|---|---|
| A | MemWAL adapter, surface guards, enrollment sidecar | inline-commit crash matrix |
| B | new ingest route/CLI, durable ack, strict fold | ack durability; API compatibility; cost budget |
| C | atomic dead letter, audit provenance, status/bounds | reject crash matrix; backpressure tests |
| D | epoch-fenced drain, persistent quiesce/resume, schema/branch/upgrade integration | two-coordinator race and format-transition suite |
| E | fresh cuts and maintained-index reads; cross-process `Fresh` ships only if the substrate generation-retention guard exists (§9), otherwise same-process only | cut consistency; merged-generation exclusion |
| F | multi-shard upsert and stream deletes | one-key-one-shard proof; Lance re-audit |

254
docs/rfcs/rfc-027-lineage-merge-deltas.md Normal file
View file

@ -0,0 +1,254 @@
---
type: spec
title: "RFC-027 — Lineage-based merge deltas"
description: Research specification for replacing full-width branch-merge classification with Lance row-version lineage, explicitly blocked on a sublinear deletion-delta source and enforceable I/O cost gates.
status: research-blocked
tags: [eng, rfc, merge, lineage, change-feed, performance, lance, omnigraph]
timestamp: 2026-07-10
owner:
---
# RFC-027 — Lineage-based merge deltas
**Status:** Research / blocked on deletion-delta discovery
**Date:** 2026-07-10
**Depends on:** [RFC-022](rfc-022-unified-write-path.md)'s unified branch-merge
pipeline and capture-once write view
**Surveyed:** omnigraph 0.8.1; Lance 9.0.0-beta.15 (`f24e42c1`)
**Audience:** merge, storage, and performance maintainers
**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md).
Findings marked **BLOCKER** must be dispositioned before acceptance.
---
## 0. Status and decision boundary
The direction is recommended: branch merge should discover changed row IDs from
storage lineage, then read wide values only for those candidates. The proposed
replacement is **not implementation-ready** and this RFC does not authorize
removing `OrderedTableCursor`.
Two facts block the O(delta) claim:
1. filtering `_row_last_updated_at_version` is still a physical O(rows) column
scan unless a substrate index or change-log makes it selective;
2. a deleted row is absent from the target snapshot, so its version columns
cannot identify it. The current implementation finds deletions by scanning
and differencing both complete ID sets.
The RFC advances only when both candidate discovery and deletion discovery have
measured costs bounded by the changed working set. Until then the existing merge
classifier remains the correctness fallback.
## 1. Problem
Current three-way merge classification streams base, source, and target tables
and compares row signatures. A one-row change can therefore read every row and
every wide property, including embeddings and blobs, while holding merge-wide
coordination longer than necessary.
`changes/mod.rs` is narrower but not yet asymptotically different:
- changed live rows are filtered by `_row_last_updated_at_version`;
- insert/update classification builds the full base ID set;
- deletion classification scans both ID sets;
- cross-branch fallback scans and compares complete ordered rows.
Reusing that code unchanged is O(rows), not O(delta). This RFC replaces those
specific scans rather than relabeling them.
## 2. Target contract
For each table touched since the merge base:
1. discover the source and target candidate row IDs from Lance metadata;
2. classify insert, update, and delete without loading unrelated user columns;
3. join the two candidate sets into the existing merge truth table;
4. fetch complete rows only for candidates whose disposition needs values;
5. stage the resulting delta through RFC-022 and publish once.
The semantic oracle remains `merge_truth_table`. This RFC changes candidate
discovery and I/O, not conflict kinds, delete/update precedence, constraint
validation, or manifest atomicity.
## 3. Live-row candidates
On a stable-row-ID table in one physical lineage, Lance exposes:
- `_row_created_at_version` — first creation of the logical row;
- `_row_last_updated_at_version` — latest modification of the logical row.
For merge-base table version `Vb` and side version `Vs`, a live row is:
- an insert when `Vb < created_at <= Vs`;
- an update when `created_at <= Vb` and
`Vb < last_updated_at <= Vs`.
This classification removes the current full base-ID membership set. Candidate
scans project only `id`, edge endpoints when applicable, stable row ID, and the
two version columns.
It does **not** by itself make the scan O(delta). Phase R1 must prove one of:
1. Lance can build and maintain a scalar index over the version columns with
correct partial-coverage fallback;
2. transaction/fragment metadata exposes an equivalent bounded changed-row
iterator;
3. an upstream Lance change-feed primitive supplies these IDs directly.
An index is derived state. Missing or partial coverage falls back to a correct
narrow scan and is reported; it never makes merge fail or omit candidates.
## 4. Deletion delta is the blocker
Deleted logical rows have no live record whose version columns can be filtered.
The current `deleted_ids_by_set_diff` scans all IDs at base and side. That term
alone prevents an O(delta) merge, even if inserts and updates become selective.
Research must disposition these substrate-shaped options:
### 4.1 Deletion-vector and stable-row-ID lineage
Walk only transactions/fragments changed between `Vb` and `Vs`, compare their
deletion vectors, and translate newly deleted offsets to stable row IDs. This is
acceptable only if it handles merge-insert rewrites, updates that move rows,
compaction, fragment reuse, and deletion-vector materialization without scanning
unaffected fragments.
### 4.2 Upstream Lance change log
Consume a public Lance API that yields durable inserted/updated/deleted stable
row IDs by version range. A source-level prototype or private API is evidence,
not a dependency; the production surface must be public and pinned by
`lance_surface_guards.rs`.
### 4.3 Atomic OmniGraph deletion deltas
Write immutable per-commit deleted-ID rows in the same manifest CAS as the graph
commit. This is first-class commit metadata, not a side channel. It adds storage
and format liability and therefore needs a separate format decision before use;
it is not silently folded into v5.
Until one option passes the cost and correctness gates, delete-bearing histories
use the existing classifier. There is no "usually O(delta)" claim that excludes
deletes without saying so in plan and metrics.
## 5. Branch lineage and unsupported operations
Lance version numbers are branch-local and can overlap. Candidate discovery must
use each manifest entry's physical `(table_path, table_branch, table_version)`;
it must not subtract graph commit numbers or compare equal numeric versions from
different branch lineages.
Research must specify behavior for:
- a lazy fork that still physically reads its parent table branch;
- the first branch-owned write after a fork;
- compaction and index-only versions between base and side;
- overwrite, restore, schema rewrite, and hard-drop operations;
- tables without stable row IDs or version metadata;
- a table dropped or introduced on only one side.
Any shape not proven lineage-compatible takes the correctness fallback and
records a typed reason. Physical metadata gaps never weaken the logical merge.
## 6. Proposed planner shape
```text
ResolveMergeBase
-> DiscoverSideDelta(source)
-> DiscoverSideDelta(target)
-> JoinCandidateIds
-> FetchCandidateRows
-> ExistingTruthTableAndValidation
-> RFC022StageAndPublish
```
`DiscoverSideDelta` returns ordered, typed operations keyed by table and logical
row ID. Candidate order is deterministic. Payload fetches use structured key
lookups or SIP; they do not synthesize string `IN` filters or read embedding/blob
columns for candidates whose disposition needs only identity.
The merge base, physical entries, and candidate cuts are captured once before
heavy work. After acquiring publish queues, RFC-022's OCC/read-set rule either
confirms the cut or restarts discovery; it never publishes a delta classified
against a moved side.
## 7. Fallback and rollout safety
`OrderedTableCursor` remains the universal fallback through the research and
shadow phases. Fallback reasons are closed enum values, including:
- `DeletionDeltaUnavailable`;
- `VersionIndexUncovered`;
- `CrossLineageUnsupported`;
- `StableRowIdsUnavailable`;
- `OverwriteOrRestoreInRange`;
- `LineageMetadataInconsistent`.
Before enabling the new path, shadow mode runs both classifiers on the same
captured snapshots and compares ordered operation sets and merge outcomes. A
mismatch fails the test or records a production diagnostic; it never silently
chooses the new answer.
## 8. Cost contract
Use `helpers::cost` with fixed `delta = 1` and table sizes of at least 10k,
100k, and 1M rows, both scalar-only and embedding-bearing. Measure candidate
discovery separately from candidate payload fetch.
The acceptance target for a true lineage path is:
- insert and update discovery I/O is flat within named slack as rows grow;
- delete discovery I/O is flat within the same discipline;
- bytes read scale with candidate identity/version columns plus fetched delta
payload, not total row width;
- peak RSS is bounded by candidate batch size, not table size;
- graph-history depth and unrelated catalog width do not change the curve.
Local tests gate scan/fragment terms. S3 tests gate object-store RPC and opener
terms that local FS cannot expose. A benchmark without an asserted I/O slope is
supporting evidence, not the acceptance gate.
If only live-row indexing passes, this RFC may prototype or shadow an
insert/update-only path, but it does not authorize production shipping. A
partial fast path requires a separately accepted RFC that names its O(rows)
delete fallback and operational value. This RFC and the default-classifier
O(delta) claim remain blocked until R2 passes.
## 9. Correctness gates
- `merge_truth_table` remains byte-for-byte the semantic disposition oracle.
- Property tests compare lineage and cursor classifiers over inserts, updates,
deletes, endpoint moves, cycles, and conflicting constraints.
- Dedicated cases cover compaction, index maintenance, restore, overwrite,
lazy forks, and same-number/different-branch versions.
- A one-row delete in a 1M-row table is the blocker test: the RFC cannot leave
research status until it is both correct and flat in table size.
- Surface guards pin every Lance version-column, stable-row-ID, deletion-vector,
transaction, and index behavior used by the chosen implementation.
- Fault tests move a side after discovery and prove OCC restarts instead of
publishing stale classification.
## 10. Observability
For every merge table report classifier path, fallback reason, source/target
candidate counts, deleted-ID discovery mechanism, fragments and bytes read,
payload columns fetched, discovery latency, and whether shadow results matched.
Metrics distinguish logical delta size from physical discovery work. This is
required to catch an apparently correct path silently regressing to O(rows).
## 11. Research plan
| Phase | Content | Exit criterion |
|---|---|---|
| R0 | Instrument current cursor and `changes` paths; build shadow comparison harness | semantic and I/O baselines |
| R1 | Prove selective live-row discovery and branch-version mapping | insert/update flat-cost gate |
| R2 | Prototype all deletion-delta options against pinned Lance | one option passes delete correctness + flat-cost gate |
| R3 | Shadow new classifier across the full merge truth table and histories | zero mismatches; explicit fallback ledger |
| R4 | Enable lineage path behind a scoped feature/config gate | production diagnostics within budgets |
| R5 | Make lineage default and consider cursor retirement | all fallbacks dispositioned; no physical gap can break merge |
The RFC remains **research / blocked** through R2. Choosing an OmniGraph commit
delta format in §4.3 requires its own format amendment before R3.

12
docs/user/branching/index.md
View file

@ -33,8 +33,10 @@ conflict kinds are on the [merge](merge.md) page.
## L2 — Recovery audit trail
Interrupted multi-table writes are recovered automatically the next time the graph is opened read-write. Recovery commits are recorded in the audit trail under the actor `omnigraph:recovery`, so you can find them with:
```bash
omnigraph commit list --filter actor=omnigraph:recovery
```
Interrupted multi-table writes are recovered automatically the next time the
graph is opened read-write. Each completed recovery is recorded internally in
`_graph_commit_recoveries.lance`. A roll-forward keeps the interrupted
writer's original commit id and actor; rollback and legacy recovery commits use
the reserved actor `omnigraph:recovery`. Consequently, `commit list` is not a
complete recovery log, and the CLI does not currently expose a query for the
internal recovery-audit table.

6
docs/user/branching/transactions.md
View file

@ -19,7 +19,7 @@ Two primitives, two scopes:
| **One `.gq` query** (any number of statements inside) | The query itself — handled by the publisher's atomic manifest commit | Yes — all statements land together or none of them do | The publisher never publishes; target unchanged |
| **Many queries that must succeed together** | Branches: `branch_create` → run N queries on the branch → `branch_merge` | Yes — the merge is a single atomic publish | Drop the branch (`branch_delete`); main is unaffected |
Snapshot isolation is per-query — every read inside one query sees one consistent manifest version. Two concurrent queries on the same branch see independent snapshots; the publisher's CAS catches racing writes.
Snapshot isolation is per-query — every read inside one query sees one consistent manifest version. Two concurrent queries on the same branch see independent snapshots. Mutation/load capture the branch head as coarse OCC authority, so a prepared plan is never silently reparented after another graph commit.
## Comparison with `BEGIN` / `COMMIT`
@ -136,7 +136,9 @@ This is the workflow agentic loops are designed around: **branches are the unit
| Scenario | What happens | Caller action |
|---|---|---|
| Single query fails mid-flight | Publisher never publishes; target unchanged | Read the error, decide whether to retry |
| Concurrent writers race the same `(table, branch)` | Publisher CAS rejects the loser with a version-mismatch conflict | Refresh handle, retry the query |
| Branch authority changes before physical effects | Retryable inserts/loads fully reprepare; strict writes return `read_set_conflict` | For a surfaced strict conflict, refresh and retry deliberately |
| Authority changes after a physical effect | The write returns `recovery_required` and leaves its durable sidecar | Resolve recovery by read-write reopen/server restart before retrying |
| An overlapping recovery intent remains unresolved before effects | The write returns `recovery_required` with that intent's operation id and does not advance a table | Resolve recovery by read-write reopen/server restart before retrying |
| Branch with N successful mutations, then merge fails (three-way conflict) | Each individual mutation already committed on the branch; merge surfaces `MergeConflicts` | Inspect, decide whether to keep working on the branch, abandon it (`branch_delete`), or resolve and re-merge |
| Process crashes mid-branch-workflow | Each completed mutation on the branch is durable | Re-open the graph, continue where you left off |

2
docs/user/cli/reference.md
View file

@ -21,7 +21,7 @@ Top-level command families and subcommands. Graph-targeting commands accept a po
| `schema plan \| apply \| show (alias: get)` | migrations. `apply` refuses a cluster-managed graph (one whose storage is inside a cluster) and points at `cluster apply` — those graphs evolve through the cluster ledger, not a direct apply |
| `lint` (alias: `check`) | offline / graph-backed query validation. Replaces `query lint` / `query check`, which are kept as deprecated argv-level shims that print a one-line warning and rewrite to `omnigraph lint` |
| `cluster validate \| plan \| apply \| approve \| status \| refresh \| import \| force-unlock` | declarative cluster control plane. `validate` checks a local `cluster.yaml` folder and referenced schema/query/policy files; `plan` diffs it against local JSON state at `__cluster/state.json`, annotates dispositions, and embeds real schema-migration previews; `apply` converges the cluster — stored-query/policy catalog writes (content-addressed under `__cluster/resources/`), graph creates, schema updates (soft drops only; `--as` records the actor), and graph deletes behind a digest-bound approval from `cluster approve <resource> --as <actor>` (`apply`/`approve` default the actor from `~/.omnigraph/config.yaml`'s `operator.actor` when `--as` is omitted); what apply converges is what an `omnigraph-server --cluster <dir>` deployment serves on its next restart (`--cluster` is the server's only boot source — cluster-only); `status` reads the state ledger; `refresh`/`import` explicitly update local JSON state from read-only graph observations; `force-unlock <LOCK_ID>` manually removes a held local state lock by exact id |
| `optimize` | non-destructive Lance compaction (skips tables with `Blob` columns or uncovered drift; `--json` reports `skipped`) |
| `optimize` | non-destructive Lance compaction + index reconciliation (blob-bearing tables use the normal path; tables with uncovered drift are skipped and `--json` reports `skipped`) |
| `repair [--confirm] [--force]` | preview or explicitly publish uncovered manifest/head drift. `--confirm` heals verified maintenance drift and exits non-zero if suspicious/unverifiable drift is refused; `--force --confirm` publishes suspicious/unverifiable drift after operator review |
| `cleanup --keep N --older-than 7d --confirm` | destructive version GC (`--confirm` to execute; also needs `--yes` against a non-local `s3://` target — see *Write diagnostics & destructive confirmation*) |
| `embed` | offline JSONL embedding pipeline |

6
docs/user/concepts/storage.md
View file

@ -7,7 +7,7 @@ Every node type and every edge type is its own Lance dataset:
- **Columnar Arrow storage**: each property is a column; nullable per Arrow schema.
- **Fragments**: data is partitioned into fragments; new writes create new fragments.
- **Manifest versioning**: every commit produces a new dataset version; old versions remain readable.
- **Stable row IDs**: stable row IDs are enabled on every Lance dataset OmniGraph creates — node and edge data tables, `__manifest`, the commit-graph datasets, and any future system tables. This is an architectural invariant: the flag is one-way at dataset create, so a future change that introduces a Lance dataset must preserve it. Consequences: `_row_created_at_version` and `_row_last_updated_at_version` are available on every dataset (load-bearing for change-feed validators); indices survive `omnigraph optimize`. Pre-0.4.x graphs created before this code path settled may have datasets without the flag and cannot be retrofitted in place — the supported path is dump-and-reload. The rewrite path used by `schema_apply` preserves the flag.
- **Stable row IDs**: stable row IDs are enabled on every Lance dataset OmniGraph creates — node and edge data tables, `__manifest`, `_graph_commit_recoveries.lance`, and any future system tables. This is an architectural invariant: the flag is one-way at dataset create, so a future change that introduces a Lance dataset must preserve it. Consequences: `_row_created_at_version` and `_row_last_updated_at_version` are available on every dataset (load-bearing for change-feed validators); indices survive `omnigraph optimize`. Pre-0.4.x graphs created before this code path settled may have datasets without the flag and cannot be retrofitted in place — the supported path is dump-and-reload. The rewrite path used by `schema_apply` preserves the flag.
- **Append / delete / `merge_insert`**: native Lance write modes.
- **Per-dataset branches** (Lance native): copy-on-write at the dataset level.
- **Object-store agnostic**: file://, s3://, gs://, az://, http (read-only via Lance) — OmniGraph wires file:// and s3://.
@ -30,7 +30,7 @@ OmniGraph is **not** a single Lance dataset; it is a *graph* of datasets coordin
- **Snapshot reconstruction**: latest visible `table_version` per `(table_key, table_branch)` minus tombstones — rows where `object_type = table_tombstone`, whose own `table_version` (acting as the tombstone version) is `>= the entry's table_version`.
- **Atomic publish**: multi-dataset commits publish so that a single write to `__manifest` flips all the new sub-table versions visible at once.
- **Row-level CAS on the merge-insert join key**: `object_id` carries an unenforced-primary-key annotation so Lance's bloom-filter conflict resolver rejects two concurrent commits that land the same `object_id` row. Without this annotation, Lance's transparent rebase would admit silent duplicates from racing publishers.
- **Optimistic concurrency control on publish**: a publish asserts the manifest's current latest non-tombstoned version for each touched table is exactly what the caller observed; mismatches surface as an `ExpectedVersionMismatch` manifest conflict naming the table and the expected/actual versions. Concurrent advances surface as a conflict rather than being silently rebased through.
- **Optimistic concurrency control on publish**: legacy writers assert the manifest's current latest non-tombstoned version for each touched table; a mismatch surfaces as `ExpectedVersionMismatch`. RFC-022-enrolled mutation/load attempts use a stronger, branch-wide contract: preparation captures the Lance-native branch identity, the exact `graph_head` (including absence), the accepted schema identity/catalog, and one base table snapshot. Under root-shared schema → branch → sorted-table gates, the engine revalidates that complete authority before any physical effect, then the publisher rechecks the exact native branch identity/head plus the touched-table versions. An insert-only mutation or Append/Merge load whose authority changed before effects discards and fully reprepares the bounded attempt; Update/Delete/Overwrite returns `ReadSetChanged`. Once any Lance effect is durable, any later failure leaves the recovery sidecar authoritative and returns `RecoveryRequired` instead of silently rebasing or replaying the prepared plan.
### Internal schema versioning
@ -95,7 +95,7 @@ flowchart TB
- **`__manifest/`** is a Lance dataset whose rows describe which sub-table version is published at which graph-branch. Reading a snapshot starts here.
- **`nodes/`** and **`edges/`** are sibling directories holding one Lance dataset per declared type. Names are `fnv1a64-hex` of the type name to keep paths fixed-length and case-safe.
- The graph commit DAG lives in **`__manifest`** as `graph_commit` / `graph_head` rows written in the publish CAS (RFC-013 Phase 7). The former `_graph_commits.lance` / `_graph_commit_actors.lance` lineage tables are retired — a graph this binary creates has neither.
- **`_graph_commit_recoveries.lance`** — one row per crash-recovery action. Joined by `graph_commit_id` to the graph commit lineage (the `graph_commit` rows in `__manifest` since RFC-013 Phase 7); the linked commit carries `actor_id=omnigraph:recovery`. Operators correlate recoveries with the original mutations they rolled forward / back via this join.
- **`_graph_commit_recoveries.lance`** — one internal row per completed crash-recovery action, including its exact per-table outcomes and the original actor. It joins by `graph_commit_id` to the graph commit lineage in `__manifest`. A v3 roll-forward keeps the interrupted writer's original actor; rollback and legacy recovery commits use `omnigraph:recovery`. The CLI does not currently expose this internal table.
- **`__recovery/{ulid}.json`** — transient sidecar files written by a writer before it advances the underlying dataset, deleted once the matching manifest publish succeeds. A sidecar persisting after process exit means the writer crashed mid-commit; the next read-write open processes it. Steady-state directory is empty.
- **`_refs/branches/{name}.json`** is graph-level branch metadata — pointers from a branch name to the manifest version it heads.
- **Inside each Lance dataset** (orange): the standard Lance directory layout. `_versions/{n}.manifest` records every commit; `data/` holds the actual Arrow fragments; `_indices/{uuid}/` holds index segments with their own `fragment_bitmap` for partial coverage; `_refs/` holds Lance-native per-dataset branches and tags.

19
docs/user/mutations/index.md
View file

@ -21,6 +21,12 @@ properties in the assignment block (`insert WorksAt { person: $p, org: $o }`).
`<value>` is a literal, `$param`, or `now()`.
On a blob-bearing type, an update materializes and rewrites blob payloads only
for the rows matched by its predicate, including blobs the update does not
change. This keeps correctness independent of physical index state, but adds
read/write I/O proportional to the matched blob bytes; use selective update
predicates for large blobs.
## Atomicity
A change query publishes **one commit** at the end of the query. Multiple
@ -29,6 +35,19 @@ failure leaves the graph untouched. See [transactions](../branching/transactions
for the per-query atomicity contract and [branches](../branching/index.md) for
multi-query workflows.
Concurrent changes use optimistic concurrency over the whole target branch.
Insert/Merge/Append operations whose branch changed before physical effects are
discarded and fully revalidated with a bounded internal retry. Strict
Update/Delete/Overwrite operations instead return a structured conflict. This
branch-wide token is deliberately conservative: a change to a different table
can invalidate a prepared strict write because constraints may have read it.
If the synchronous barrier finds an unresolved overlapping recovery intent, or
if a conflict is discovered after a Lance table effect is durable, the request
returns `recovery_required` with an operation id. Do not immediately retry that
request; reopen the graph read-write (or restart the server) so the durable
recovery intent is resolved first.
## Inserts/updates and deletes cannot mix in one query
A single change query must be **either insert/update-only or delete-only**.

13
docs/user/operations/audit.md
View file

@ -31,13 +31,12 @@ List commits to see who made each change:
omnigraph commit list graph.omni
```
System-initiated writes use reserved actor ids — for example, automatic recovery
of an interrupted write records `omnigraph:recovery`, so operator changes and
machine repairs are distinguishable in the history:
```bash
omnigraph commit list --filter actor=omnigraph:recovery graph.omni
```
System-initiated writes use reserved actor ids. Rollback and legacy recovery
commits use `omnigraph:recovery`, while a v3 roll-forward preserves the
interrupted writer's original commit id and actor. Exact recovery actions and
per-table outcomes are stored in the internal
`_graph_commit_recoveries.lance` audit table; the CLI does not currently expose
that table, so `commit list` alone is not a complete recovery log.
## What is tracked

2
docs/user/operations/errors.md
View file

@ -8,9 +8,11 @@
- `Io(io::Error)`
- `Manifest(ManifestError { kind: BadRequest|NotFound|Conflict|Internal, details: Option<ManifestConflictDetails>, … })`
- `ManifestConflictDetails::ExpectedVersionMismatch { table_key, expected, actual }` — caller's `expected_table_versions` did not match the manifest's current latest non-tombstoned version (set by `OmniError::manifest_expected_version_mismatch`).
- `ManifestConflictDetails::ReadSetChanged { member, expected, actual }` — an RFC-022 prepared write's branch/head/table authority changed before physical effects. HTTP returns **409** with `read_set_conflict`. A retry must start from preparation; strict writes leave that choice to the caller.
- `ManifestConflictDetails::RowLevelCasContention` — Lance row-level CAS rejected the publish because a concurrent writer landed the same `object_id`. Retried internally by the publisher; only surfaces if the retry budget exhausts.
- **D₂ parse-time rejection**: a single mutation query that mixes inserts/updates with deletes errors out *before any I/O* with kind `BadRequest`. Message: `mutation '<name>' on the same query mixes inserts/updates and deletes; split into separate mutations: (1) inserts and updates, then (2) deletes`. See [query-language.md](../queries/index.md) for the rule.
- `MergeConflicts(Vec<MergeConflict>)`
- `RecoveryRequired { operation_id, reason }` — an overlapping durable recovery intent remains unresolved. Its physical effects may already have landed, or it may still be armed before the first effect. HTTP returns **503** with `recovery_required.operation_id`. Resolve the sidecar through a read-write reopen/server restart before retrying; this is intentionally not an ordinary OCC retry.
Compiler-side `CompilerError` covers parse / catalog / type / storage / plan / execution / arrow / lance / IO / manifest / unique-constraint, each with structured spans (`SourceSpan { start, end }`) for ariadne-style diagnostics. The legacy `NanoError` name remains as a deprecated compatibility alias.

11
docs/user/operations/maintenance.md
View file

@ -9,13 +9,13 @@
- **Also compacts the internal `__manifest` table** (RFC-013 step 2), which accumulates one fragment per commit — it now carries the graph lineage and actor rows inline (RFC-013 Phase 7: `graph_commit` / `graph_head` rows), so on the authenticated write path every commit's actor lands here too — and otherwise makes every write's metadata scan grow with history. (The `_graph_commits.lance` / `_graph_commit_actors.lance` tables are retired, so there is no separate lineage table to compact.) It takes a simpler path than data tables: `__manifest` is read at its latest version, so compaction just advances its version in place — **no manifest publish and no recovery sidecar**. (The sidecar-free property is not because it is one commit — `compact_files` can emit a `ReserveFragments` commit before the `Rewrite`, and the auto-cleanup strip below is a further commit — but because every one of those commits is content-preserving and the table is read at its latest version, so a crash at any point leaves it readable and content-identical and the next `optimize` re-plans.) It appears in the returned stats under `table_key` `"__manifest"`. It is **not yet covered by `cleanup`**, so its version chain still grows until the cleanup half lands (it requires a cleanup-resurrection safeguard first); run `optimize` on a cadence to keep per-write metadata scans flat.
- **`optimize` is non-destructive by construction — it never garbage-collects versions, on any table (data or internal).** Compaction rewrites fragments and advances the version; old versions stay reachable until you run `cleanup`. This holds even for a graph created by an older binary that stored an on-by-default Lance `auto_cleanup` hook: `compact_files` / `optimize_indices` commit with the hook enabled and expose no skip override, so before compacting **any** table `optimize` strips its stale `lance.auto_cleanup.*` config first, so Lance's commit-time GC hook cannot fire and silently prune `__manifest`-pinned versions. (Graphs created by current binaries store no such config; the strip is the upgrade-path safety net.) The internal-table path additionally tolerates a concurrent live writer: it runs a **bounded** rebase-and-retry, so transient contention does not fail the operator's `optimize` or the live write — but sustained contention past the retry budget surfaces a loud conflict error rather than looping forever (bounded and observable, not a silent give-up). The data-table path holds the per-table write queue while it compacts, so it does not contend with mutations on that table in the first place.
- **Reindex (index coverage maintenance).** A scalar/FTS/vector index only covers the fragments it was built over. Rows appended after the index was built (e.g. by `load --mode merge`, whose commit does not rebuild an already-existing index) are scanned unindexed, and compaction itself rewrites fragments out of an index's coverage. `optimize` runs Lance's incremental `optimize_indices` after compaction to fold those fragments back in (a delta merge, not a full retrain), restoring full coverage so equality/range/traversal predicates stay index-accelerated. This is why a table with **no compaction work but stale index coverage still commits** a new version under `optimize`. Run `optimize` on a cadence at least as frequent as your freshness window so recently-loaded rows do not linger in the unindexed flat-scan tail.
- **Create declared-but-missing indexes (the index reconciler).** `@index`/`@key` declares intent; `schema apply` records it but builds nothing, and `load`/`mutate` defer a column that cannot be built yet (a `Vector` column with no trainable vectors). `optimize` materializes any such declared-but-unbuilt index over the compacted layout — so it is the convergence path for an `@index` added after data exists, or a vector index whose embeddings arrived via a later `embed`. A column still not buildable (no vectors yet) is reported on the table's stat as `pending_indexes` (visible in `--json`), not treated as a failure; the next `optimize` retries. So `optimize` is the single operator-facing index reconciler: it compacts, restores coverage, **and** builds declared-but-missing indexes.
- **Create declared-but-missing indexes (the index reconciler).** `@index`/`@key` declares intent; `schema apply`, `load`, and `mutate` build no physical indexes inline. They record or publish only their exact logical/data effects and leave all index materialization to `ensure_indices`/`optimize`. `optimize` materializes every buildable declared-but-missing index over the compacted layout — so it is the convergence path for an `@index` added after data exists, or a vector index whose embeddings arrived via a later `embed`. A column still not buildable (no vectors yet) is reported on the table's stat as `pending_indexes` (visible in `--json`), not treated as a failure; the next `optimize` retries. So `optimize` is the single operator-facing index reconciler: it compacts, restores coverage, **and** builds declared-but-missing indexes.
- Each table's compact→reindex→publish serializes with concurrent mutations on the same table. A crash mid-operation is recovered automatically on the next open (both compaction and reindex are content-preserving, so roll-forward is always safe).
- **Requires a recovered graph.** `optimize` refuses (errors) when a pending crash-recovery operation is present — operating on an unrecovered graph could publish a partial write that recovery would roll back. Reopen the graph to run recovery, then re-run `optimize`.
- **Uncovered drift is skipped, not interpreted.** If a table's underlying version is ahead of the version recorded in `__manifest` and no crash-recovery record covers that movement, `optimize` reports `skipped: DriftNeedsRepair` with the manifest/head versions and leaves the table untouched. Run `omnigraph repair` to classify and explicitly publish that drift.
- Bounded by `OMNIGRAPH_MAINTENANCE_CONCURRENCY` (default 8).
- Returns per-table stats: `table_key, fragments_removed, fragments_added, committed, skipped, manifest_version, lance_head_version, pending_indexes` (the last lists any declared `@index` column the reconciler could not build this run, with the reason — e.g. a vector column with no trainable vectors yet).
- **Blob tables are skipped.** A table that declares any `Blob` property is not compacted: it is reported with `skipped: BlobColumnsUnsupportedByLance` (and logged) instead of compacted, and the rest of the sweep proceeds normally. **Reads and writes are unaffected** — only compaction is. Consequence: fragment count and deleted-row space on blob tables are not reclaimed; query results are never affected. A skipped blob table is also **not reindexed** in the same sweep (the skip happens before the reindex step), so its index coverage on appended rows is not refreshed by `optimize` today.
- **Blob tables use the normal compaction and reindex path.** Lance 8.0.0+ supports blob-v2 compaction, so OmniGraph no longer has a blob-specific skip or capability gate. Fragment reclamation and index-coverage repair therefore apply to blob-bearing tables like every other table.
## `repair` — explicit
@ -39,7 +39,12 @@
- CLI guards with `--confirm`; without it, prints a preview line.
- **Non-local consent.** Against a non-local target (an `s3://` store/cluster), `cleanup` additionally requires `--yes` on top of `--confirm`: a TTY is prompted, and a non-interactive run (no TTY, or `--json`) refuses rather than destroying. A local (`file://`) target needs only `--confirm`. The same `--yes` gate applies to overwrite `load` and `branch delete`; every maintenance run echoes its resolved target to stderr (suppress with `--quiet`).
- **Recovery floor:** `--keep < 3` may garbage-collect versions that crash recovery needs as a rollback target. Default `--keep 10` is safe.
- **Orphaned-branch reconciliation:** before the version GC, cleanup reclaims any per-table or commit-graph branch absent from the manifest branch list. These orphans arise when a `branch_delete` flips the manifest authority but a downstream best-effort reclaim does not complete (see [branches-commits.md](../branching/index.md)). The reconciler is idempotent (it no-ops once nothing is orphaned), runs regardless of the `keep_versions` / `older_than` values (those gate version GC only), and never reclaims `main` or system-branch forks. Reclaimed forks are logged.
- **Requires clean recovery state.** If any durable recovery intent is pending,
cleanup refuses before orphan reconciliation or version GC. Reopen the graph
read-write (or restart the server) to resolve recovery, then rerun cleanup;
deleting transaction/version history while an intent is pending would make
exact effect ownership unverifiable.
- **Orphaned-branch reconciliation:** before the version GC, cleanup reclaims any per-table Lance branch absent from the manifest branch list. These orphans arise when a `branch_delete` flips the manifest authority but a downstream best-effort reclaim does not complete (see [branches-commits.md](../branching/index.md)). The reconciler is idempotent (it no-ops once nothing is orphaned), runs regardless of the `keep_versions` / `older_than` values (those gate version GC only), and never reclaims `main` or system-branch forks. Reclaimed forks are logged. Graph lineage has no separate branch dataset: it lives in `__manifest`.
## Tombstones

54
docs/user/operations/server.md
View file

@ -162,25 +162,51 @@ Only `/export` streams (`application/x-ndjson`, MPSC channel + `Body::from_strea
## Error model
Uniform `ErrorOutput { error, code?, merge_conflicts[], manifest_conflict? }` with `code ∈ unauthorized | forbidden | bad_request | not_found | conflict | too_many_requests | internal`. Merge conflicts attach structured `MergeConflictOutput { table_key, row_id?, kind, message }`.
Uniform
`ErrorOutput { error, code?, merge_conflicts[], manifest_conflict?, read_set_conflict?, recovery_required? }`
with
`code ∈ unauthorized | forbidden | bad_request | not_found | method_not_allowed | conflict | too_many_requests | internal`.
Merge conflicts attach structured
`MergeConflictOutput { table_key, row_id?, kind, message }`.
`manifest_conflict` is set on **concurrent-write rejections** (HTTP 409): the
caller's pre-write view of one table's manifest version was stale.
`ManifestConflictOutput { table_key, expected, actual }` tells the client
which table to refresh and retry. This is the conflict shape produced by
concurrent `/mutate` (or its `/change` alias), `/load` (or its deprecated
`/ingest` alias) calls landing the same `(table, branch)` race.
`manifest_conflict` is set on legacy per-table manifest-version rejections
(HTTP 409). `ManifestConflictOutput { table_key, expected, actual }` tells the
client which table was stale. Mutation and load use the unified coarse-OCC
adapter described next; other writers retain this older conflict shape until
they are enrolled.
HTTP status codes used: 200, 400, 401, 403, 404, 409, 429, 500.
`read_set_conflict` is set when a prepared write is rejected before any table
effect because its branch authority changed. The HTTP status is 409 and
`ReadSetConflictOutput { member, expected, actual }` identifies the stale
authority member. The engine already performs a bounded full-attempt retry for
mutation inserts and load `append`/`merge`. Strict mutation updates/deletes and
load `overwrite` return the 409 to the caller instead of being replayed.
`recovery_required` is set when an overlapping durable recovery intent remains
unresolved; its table effects may or may not have started. The HTTP status is 503 and
`RecoveryRequiredOutput { operation_id }` names the durable recovery intent.
The optional `code` field is omitted for this response: adding a new value to
the closed error-code enum would break older clients, while the optional
structured field is additive and rolling-safe.
Do not blindly resubmit the write: let a read-write open or the recovery sweep
resolve that operation first, then retry from a fresh snapshot.
HTTP status codes used: 200, 400, 401, 403, 404, 405, 409, 429, 500, 503.
## Per-actor admission control
Disjoint
`(table, branch)` writes from different actors now run concurrently,
guarded only by the engine's per-(table, branch) write queue. To keep
one heavy actor from exhausting shared capacity (Lance I/O, manifest
churn, network), the server gates mutating handlers through per-process
admission limits configured from environment variables:
RFC-022-enrolled mutation/load preparation runs outside the effect gates, so
parsing, validation, and reclaimable fragment staging can overlap across branches.
Readers acquire none of these gates. Before the first durable effect, however, an
attempt acquires the exclusive root schema gate, then its branch-effect gate and
sorted table queues, and holds all of them through manifest publication. The root
schema gate means enrolled effect windows on one graph currently serialize
in-process even across different branches; the branch gate preserves one atomic
graph-head validation authority, while table queues protect each concrete Lance
effect and legacy writer. These are process-local ordering gates, not a
cross-process lock. To keep one heavy actor from exhausting shared capacity
(Lance I/O, manifest churn, network), the server gates mutating handlers through
per-process admission limits configured from environment variables:
| Env var | Default | Purpose |
|---|---|---|

3
docs/user/reference/constants.md
View file

@ -4,14 +4,13 @@
|---|---|---|
| `MANIFEST_DIR` | `__manifest` | manifest layout |
| Commit graph dirs (retired) | `_graph_commits.lance` / `_graph_commit_actors.lance` | retired in Phase B; lineage lives in `__manifest` (`graph_commit` / `graph_head` rows) since RFC-013 Phase 7. A graph this binary creates has neither. |
| Recovery audit dir | `_graph_commit_recoveries.lance` | one row per crash-recovery action (`omnigraph commit list --filter actor=omnigraph:recovery`) |
| Recovery audit dir | `_graph_commit_recoveries.lance` | internal exact record of completed crash-recovery actions; no public CLI query yet |
| Run branch prefix (legacy, removed) | `__run__` | pre-v0.4.0 Run state machine; no longer a reserved name. A graph still carrying `__run__*` branches is sub-v4 and refused on open (rebuild via export/import). |
| Schema apply lock | `__schema_apply_lock__` | schema apply |
| Manifest publisher retry budget | `PUBLISHER_RETRY_BUDGET = 5` | manifest publish |
| Internal manifest schema version | `INTERNAL_MANIFEST_SCHEMA_VERSION = 4` | manifest migrations (v4 = graph lineage in `__manifest`, RFC-013 Phase 7) |
| Merge stage batch | `MERGE_STAGE_BATCH_ROWS = 8192` | merge execution |
| Maintenance concurrency | `OMNIGRAPH_MAINTENANCE_CONCURRENCY=8` | optimize/cleanup |
| Lance blob compaction support | `LANCE_SUPPORTS_BLOB_COMPACTION = false` | optimize |
| Graph index cache size | `8` (LRU) | runtime cache |
| Expand indexed-path frontier ceiling | `OMNIGRAPH_EXPAND_INDEXED_MAX_FRONTIER=1024` | traversal |
| Expand indexed-path hop ceiling | `OMNIGRAPH_EXPAND_INDEXED_MAX_HOPS=6` | traversal |

2
docs/user/search/indexes.md
View file

@ -27,7 +27,7 @@ list/`Blob` columns → none.
## L2 — OmniGraph orchestration
- **`@index`/`@key` declares intent; the physical index is derived state.** A migration records the declaration in the catalog/IR and never fails on it — `schema apply` builds **no** indexes (adding an `@index` to an existing column is a pure metadata change that touches no table data). `load`/`mutate` build declared indexes inline as part of the write, but a column that can't be built yet (a `Vector` column with no trainable vectors — IVF k-means needs ≥1 vector, e.g. rows loaded before `embed` runs) is left **pending**, not fatal. Reads stay correct meanwhile: a missing/partial index degrades to a scan (vector search to brute-force). A later `ensure_indices`/`optimize` materializes the pending index once it is buildable. This mirrors how LanceDB builds indexes asynchronously and serves unindexed rows by brute-force.
- **`@index`/`@key` declares intent; the physical index is derived state.** A migration records the declaration in the catalog/IR and never fails on it — `schema apply` builds **no** indexes. Mutation/load likewise publish only their exact data effects; they do not widen the recovery plan with index commits. Reads stay correct while an index is missing or partially covered by falling back to scans (vector search to brute-force). A later `ensure_indices`/`optimize` materializes every buildable declaration; an untrainable Vector column remains pending rather than failing the logical write.
- `ensure_indices()` / `ensure_indices_on(branch)` — idempotent build of BTREE + inverted + vector indexes for the current head; safe to re-run; returns the columns it had to defer as pending. `optimize` runs it after compaction, so the maintenance cron is the convergence path for deferred indexes.
- Indexes are built on the *branch head* (not on a snapshot), so reads always see the current index state.
- **Lazy branch forking for indexes**: a branch that hasn't mutated a sub-table doesn't need its own index — the main lineage's index is reused until the first write triggers a copy-on-write fork.

165
openapi.json
View file

@ -212,6 +212,16 @@
}
}
}
},
"503": {
"description": "An overlapping durable recovery intent must be resolved before retry",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorOutput"
}
}
}
}
},
"security": [
@ -310,6 +320,16 @@
}
}
}
},
"503": {
"description": "An overlapping durable recovery intent must be resolved before retry",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorOutput"
}
}
}
}
},
"security": [
@ -397,6 +417,16 @@
}
}
}
},
"503": {
"description": "An overlapping durable recovery intent must be resolved before retry",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorOutput"
}
}
}
}
},
"security": [
@ -477,7 +507,7 @@
}
},
"409": {
"description": "Merge conflict",
"description": "Write-authority conflict",
"content": {
"application/json": {
"schema": {
@ -495,6 +525,16 @@
}
}
}
},
"503": {
"description": "An overlapping durable recovery intent must be resolved before retry",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorOutput"
}
}
}
}
},
"deprecated": true,
@ -795,6 +835,16 @@
}
}
},
"409": {
"description": "Prepared load authority changed before effects",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorOutput"
}
}
}
},
"429": {
"description": "Per-actor admission cap exceeded; honor `Retry-After` header",
"content": {
@ -804,6 +854,16 @@
}
}
}
},
"503": {
"description": "An overlapping durable recovery intent must be resolved before retry",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorOutput"
}
}
}
}
},
"deprecated": true,
@ -884,6 +944,16 @@
}
}
},
"409": {
"description": "Prepared load authority changed before effects",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorOutput"
}
}
}
},
"429": {
"description": "Per-actor admission cap exceeded; honor `Retry-After` header",
"content": {
@ -893,6 +963,16 @@
}
}
}
},
"503": {
"description": "An overlapping durable recovery intent must be resolved before retry",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorOutput"
}
}
}
}
},
"security": [
@ -908,7 +988,7 @@
"mutations"
],
"summary": "Apply a GQ mutation to a branch (canonical mutation endpoint).",
"description": "Writes to the named `branch` (defaults to `main`). Mutations are atomic\nper call and produce a new commit. Returns counts of nodes and edges\naffected. **Destructive**: on success the branch is updated; rejected\nmutations may still acquire locks briefly. Returns 409 on merge conflict.\n\nPairs with `POST /query` (read-only). The legacy `POST /change` route\nhas identical semantics and is kept as a deprecated alias.",
"description": "Writes to the named `branch` (defaults to `main`). Mutations are atomic\nper call and produce a new commit. Returns counts of nodes and edges\naffected. **Destructive**: on success the branch is updated; rejected\nmutations may still acquire locks briefly. Returns 409 when the prepared\nwrite authority changes before effects.\n\nPairs with `POST /query` (read-only). The legacy `POST /change` route\nhas identical semantics and is kept as a deprecated alias.",
"operationId": "cluster_mutate",
"parameters": [
{
@ -973,7 +1053,7 @@
}
},
"409": {
"description": "Merge conflict",
"description": "Write-authority conflict",
"content": {
"application/json": {
"schema": {
@ -991,6 +1071,16 @@
}
}
}
},
"503": {
"description": "An overlapping durable recovery intent must be resolved before retry",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorOutput"
}
}
}
}
},
"security": [
@ -1154,7 +1244,7 @@
}
},
"409": {
"description": "Merge conflict",
"description": "Stored mutation write-authority conflict",
"content": {
"application/json": {
"schema": {
@ -1182,6 +1272,16 @@
}
}
}
},
"503": {
"description": "A stored mutation is blocked by a durable recovery intent",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorOutput"
}
}
}
}
},
"security": [
@ -1900,6 +2000,28 @@
"items": {
"$ref": "#/components/schemas/MergeConflictOutput"
}
},
"read_set_conflict": {
"oneOf": [
{
"type": "null"
},
{
"$ref": "#/components/schemas/ReadSetConflictOutput",
"description": "Set when a prepared write's logical authority changed before effects."
}
]
},
"recovery_required": {
"oneOf": [
{
"type": "null"
},
{
"$ref": "#/components/schemas/RecoveryRequiredOutput",
"description": "Set when an overlapping durable recovery intent must be resolved before\nretry. Its table effects may or may not have started."
}
]
}
}
},
@ -2413,6 +2535,30 @@
}
}
},
"ReadSetConflictOutput": {
"type": "object",
"description": "Structured authority mismatch for an RFC-022 prepared write. Values are\nstrings because members include optional graph commit ids and future\nauthority tokens, not only numeric table versions.",
"required": [
"member"
],
"properties": {
"actual": {
"type": [
"string",
"null"
]
},
"expected": {
"type": [
"string",
"null"
]
},
"member": {
"type": "string"
}
}
},
"ReadTargetOutput": {
"type": "object",
"properties": {
@ -2430,6 +2576,17 @@
}
}
},
"RecoveryRequiredOutput": {
"type": "object",
"required": [
"operation_id"
],
"properties": {
"operation_id": {
"type": "string"
}
}
},
"SchemaApplyOutput": {
"type": "object",
"required": [