mirror of
https://github.com/ModernRelay/omnigraph.git
synced 2026-06-09 01:35:18 +02:00
docs: pre-stage write precondition tolerates benign drift, defers sidecar-covered
- writes.md: new subsection specifying the tolerant precondition (OCC fence = fresh manifest pin; benign drift proceeds, sidecar-covered defers, stale handle still 409s), the load-bearing content-preserving invariant, and the Hyrum's-law observable change (409 -> success on benign drift). - invariants.md: Truth Matrix row for the precondition + deny-list entry forbidding non-content-preserving uncovered HEAD advances without a sidecar. - testing.md: list the five new tolerance tests under the writes.rs / schema_apply.rs rows. - maintenance.md + AGENTS.md: correct the now-stale claim that optimize's publish is required for strict writes / schema apply to pass their precondition — they tolerate benign drift; the publish is for reader visibility and bounded drift.
This commit is contained in:
parent
954b5453d1
commit
595c6516f2
5 changed files with 67 additions and 4 deletions
|
|
@ -99,6 +99,7 @@ Use it this way:
|
|||
| Multi-table commit | Manifest CAS plus recovery sidecars; not a single Lance primitive | [writes.md](writes.md), [architecture.md](architecture.md) |
|
||||
| Constructive mutations | In-memory `MutationStaging`, one end-of-query table commit per touched table, then one manifest publish | [writes.md](writes.md), [execution.md](execution.md) |
|
||||
| Deletes | Inline-commit residual; delete-only queries allowed, mixed insert/update/delete rejected by D2 | [query-language.md](../user/query-language.md), [writes.md](writes.md) |
|
||||
| Pre-stage write precondition | Strict writers tolerate benign `HEAD > manifest` drift with no sidecar (proceed), defer when a recovery sidecar pins the table, and reject a stale caller pin (`caller pin != fresh manifest pin`) with `ExpectedVersionMismatch`. The OCC fence is the fresh manifest pin, never the caller's snapshot pin. Correct *only* because uncovered drift is always content-preserving | [writes.md](writes.md) |
|
||||
| Branch delete | Manifest is the single authority, flipped atomically first; per-table forks + commit-graph branch are derived state, reclaimed best-effort (`force_delete_branch`) with the `cleanup` reconciler as the guaranteed backstop. Reusing a name whose reclaim failed before `cleanup` surfaces an actionable error | [branches-commits.md](../user/branches-commits.md), [maintenance.md](../user/maintenance.md) |
|
||||
| Schema validation | Type checks, required fields, defaults, edge endpoint checks, and edge cardinality are enforced on write paths | [schema-language.md](../user/schema-language.md), [execution.md](execution.md) |
|
||||
| Unique constraints | Intra-batch and write-path checks exist; full cross-version uniqueness is still a gap | [schema-language.md](../user/schema-language.md) |
|
||||
|
|
@ -165,6 +166,11 @@ case is exceptional.
|
|||
snapshot.
|
||||
- New write paths that can advance Lance HEAD before manifest publish without a
|
||||
recovery sidecar.
|
||||
- Advancing Lance HEAD with *non-content-preserving* data and no recovery
|
||||
sidecar. The pre-stage write precondition tolerates uncovered `HEAD > manifest`
|
||||
drift (proceeds without deferring) on the assumption it is content-preserving;
|
||||
a path that breaks that assumption without registering a sidecar turns the
|
||||
tolerance into a silent-data-loss vector.
|
||||
- Cross-query `BEGIN`/`COMMIT` transactions in the OSS engine. Use branches and
|
||||
merges for multi-query workflows.
|
||||
- Acknowledging writes before durable Lance and manifest persistence.
|
||||
|
|
|
|||
|
|
@ -20,13 +20,13 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
|
|||
| `end_to_end.rs` | Full init → load → query/mutate flow |
|
||||
| `branching.rs` | Branch create / list / delete, lazy fork |
|
||||
| `merge_truth_table.rs` | Merge-pair truth table (MR-786): all 9×9 `(left_op, right_op)` cells from `{noop, addNode, removeNode, addEdge, removeEdge, setProperty, dropProperty, addLabel, removeLabel}`. Adding a new op to `OpVariant` forces a compile error in `build_case` until the new row + column are dispositioned. 36 executable cells run through real `branch_merge` with a structured oracle (`MergeOutcome` / `MergeConflictKind` + graph-state assert); 45 cells involving `dropProperty`/`addLabel`/`removeLabel` are recorded as `Unsupported` until the mutation grammar grows. |
|
||||
| `writes.rs` | Direct-publish writes: cancellation, concurrent-writer CAS, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery) |
|
||||
| `writes.rs` | Direct-publish writes: cancellation, concurrent-writer CAS, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery); pre-stage drift tolerance (`strict_update_proceeds_on_benign_drift_without_sidecar`, `delete_proceeds_on_benign_drift_without_sidecar`, `strict_update_defers_when_sidecar_pins_table`) |
|
||||
| `staged_writes.rs` | TableStore staged-write primitives (`stage_append`, `stage_merge_insert`, `commit_staged`, `scan_with_staged`, `count_rows_with_staged`) — primitive-level only; engine code uses the in-memory `MutationStaging` accumulator instead |
|
||||
| `lifecycle.rs` | Graph lifecycle, schema state |
|
||||
| `point_in_time.rs` | Snapshots, time travel (`snapshot_at_version`, `entity_at`) |
|
||||
| `changes.rs` | `diff_between` / `diff_commits` |
|
||||
| `consistency.rs` | Cross-table snapshot isolation, atomic publish |
|
||||
| `schema_apply.rs` | Migration plan + apply, schema-apply lock |
|
||||
| `schema_apply.rs` | Migration plan + apply, schema-apply lock; pre-stage drift tolerance (`additive_apply_proceeds_on_benign_drift_without_sidecar`, `additive_apply_defers_when_sidecar_pins_table`) |
|
||||
| `search.rs` | FTS / vector / hybrid (`bm25`, `nearest`, `rrf`) |
|
||||
| `traversal.rs` | `Expand`, variable-length hops, anti-join |
|
||||
| `aggregation.rs` | `count`, `sum`, `avg`, `min`, `max` |
|
||||
|
|
|
|||
|
|
@ -239,6 +239,63 @@ publisher commit produces exactly one winner. The residual above is
|
|||
about *our* abandoned commits in the failure path, not about
|
||||
concurrency races.
|
||||
|
||||
### Pre-stage write precondition: tolerate benign drift, defer sidecar-covered
|
||||
|
||||
Strict writers (Update / Delete / SchemaRewrite, and the schema-apply
|
||||
index rebuild) run a pre-stage precondition — `Omnigraph::ensure_writable_or_defer`
|
||||
— before staging. Insert/Merge skip it (Lance's auto-rebase + the queue +
|
||||
the publisher CAS handle their drift).
|
||||
|
||||
A table's **Lance HEAD can legitimately sit ahead of its manifest pin**
|
||||
between an in-place HEAD advance and the next manifest publish. Sources:
|
||||
`optimize` compaction *before its publish*, a recovery `Dataset::restore`,
|
||||
an old-binary optimize that never published, an *external* `compact_files`,
|
||||
or a finalize→publisher residual. All of these are **content-preserving**
|
||||
and carry **no recovery sidecar**. The only `HEAD > pin` state that is *not*
|
||||
safe to write over is a real in-flight partial write, which the writer
|
||||
protocol always covers with a `__recovery/{ulid}.json` sidecar (Phase A).
|
||||
|
||||
So the precondition disambiguates `HEAD > pin` by sidecar presence rather
|
||||
than rejecting it wholesale. The OCC fence is the **current** manifest pin,
|
||||
re-read fresh on the conflict path — *not* the caller's snapshot pin, which
|
||||
may be stale:
|
||||
|
||||
- `HEAD == caller pin` → fresh, no drift → proceed (fast path, no extra read).
|
||||
- `caller pin != current pin` → the caller's pre-write view is stale relative
|
||||
to the live manifest: a normal OCC conflict. Fail with
|
||||
`ExpectedVersionMismatch` **here**, before any staged commit or sidecar, so
|
||||
the client refreshes and retries with no residue left behind.
|
||||
- `caller pin == current pin`, `HEAD > pin`, **no sidecar** pins the table →
|
||||
benign content-preserving drift → **proceed**; the writer's own
|
||||
`commit_staged` + the publisher CAS reconcile the manifest at the commit
|
||||
boundary.
|
||||
- `caller pin == current pin`, `HEAD > pin`, **a sidecar** pins the table →
|
||||
defer with an actionable "reopen the graph to run the recovery sweep"
|
||||
error; never write onto state the open-time sweep may roll back.
|
||||
- `HEAD < current pin` → the manifest cannot lead durable Lance state under
|
||||
the commit protocol → loud invariant violation.
|
||||
|
||||
This is the **consumer-side** complement to the producer-side convergence
|
||||
above (recovery roll-back and `optimize` both publish so `manifest == HEAD`).
|
||||
Convergence keeps *system-produced* drift bounded; the precondition is the
|
||||
net for drift no sidecar covers — legacy old-binary optimize, external Lance
|
||||
compaction — which heals at the point of use on the next strict write.
|
||||
|
||||
> **Load-bearing invariant.** Tolerating uncovered drift is correct *only*
|
||||
> because such drift is always content-preserving: a strict write (and schema
|
||||
> apply, which reads source at the pinned version and rewrites onto HEAD)
|
||||
> overwrites the drifted HEAD assuming its rows equal the pinned version's
|
||||
> rows. A future code path that advances Lance HEAD with *different content*
|
||||
> and no sidecar would turn this tolerance into a silent-data-loss vector —
|
||||
> such a path must register a recovery sidecar. See
|
||||
> [docs/dev/invariants.md](invariants.md).
|
||||
|
||||
> **Observable change (Hyrum's Law).** A strict write or schema apply on a
|
||||
> benign-drifted table now **succeeds** where it previously returned 409
|
||||
> "stale view … refresh and retry". Clients that depended on the 409 to detect
|
||||
> compaction/recovery drift must not — that 409 is reserved for genuine OCC
|
||||
> conflicts (stale handle / concurrent publisher).
|
||||
|
||||
## Conflict shape
|
||||
|
||||
Concurrent writers to the same `(table, branch)` produce exactly one
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue