omnigraph/docs/rfcs/rfc-026-memwal-streaming-ingest.md

type	title	description	status	tags	timestamp	owner
spec	RFC-026 — MemWAL streaming ingest	Adopts Lance MemWAL as OmniGraph's strategic streaming-write architecture, with durable per-row acknowledgement, graph-atomic folds, epoch-fenced quiescence, and explicit fresh-read cuts on the RFC-022 unified write path.	draft	eng rfc streaming ingest wal memwal lance omnigraph	2026-07-10

RFC-026 — MemWAL streaming ingest

Status: Draft / for team review Date: 2026-07-10 Depends on: RFC-022's unified write and generic recovery-sidecar protocol, plus RFC-023 for the initial keyed graph-stream mode. Durable heads from RFC-024 are compatible but not required. Surveyed: omnigraph 0.8.1; Lance 9.0.0-beta.15 (f24e42c1); complete MemWAL format specification Audience: engine, server, CLI, policy, and operations maintainers Open architecture review: RFC-022–027 review ledger. Findings marked BLOCKER must be dispositioned before acceptance.

---

0. Decision and risk posture

OmniGraph adopts Lance MemWAL as its strategic streaming-write architecture. MemWAL is a major Lance architectural bet: a sharded LSM write path with durable WAL entries, flushed Lance generations, merge progress committed with base-table data, maintained indexes, and epoch-fenced writers. OmniGraph consumes that architecture rather than building a WAL, shard protocol, or LSM reader.

This RFC does not characterize the architecture as experimental. The risk is narrower: Rust API names, some format details, and operational helpers are still maturing across Lance releases. We manage that API/format-maturity risk with a small adapter, compile/runtime surface guards, a quiescence requirement before Lance upgrades, and a fresh full-spec alignment audit on every bump. It is not a reason to fork or reimplement MemWAL.

The contract is:

• stream acknowledgement means the row's WAL entry is durable;
• acknowledgement does not mean graph visibility;
• default queries see only the manifest-committed graph;
• a fold is an ordinary RFC-022 graph writer and is the sole visibility point;
• fresh reads are explicit and never claim cross-table atomicity.

1. Scope and non-goals

This RFC specifies enrollment, the public stream API, acknowledgement semantics, folding, fold-time integrity, dead-letter atomicity, branch/schema quiescence, fresh-read cuts, resource bounds, observability, testing, and upgrade posture.

It does not replace load or mutate, provide cross-query transactions, store manifest mutations in MemWAL, create a second metadata authority, or weaken default snapshot isolation. Stream-mode deletes remain out of the first delivery and require the Lance tombstone surface plus a separate acceptance pass.

2. Stream mode and key semantics

Initial delivery exposes @stream(mode="upsert", on_reject="strict") on a node or edge type; on_reject accepts strict or dead_letter. It requires the table's immutable unenforced primary key to equal OmniGraph's merge key: id for nodes and edges. All occurrences of one key map to one shard and MemWAL applies last-write-wins ordering.

Public append mode is deliberately out of scope. Nodes and edges always have logical identity; allowing a retry to append the same id twice would violate that contract. A future explicitly keyless, non-graph append-only table class may consume MemWAL append semantics under its own schema/API decision.

Stream ordering intentionally differs from the interactive fence:

• concurrent interactive same-key writes serialize or fail/retry loudly;
• same-key stream entries resolve by MemWAL generation/position order;
• duplicate keys inside one bulk-load input retain the existing load error.

The schema and user docs state all three together.

3. Enrollment is a recoverable inline commit

Schema apply records @stream intent only. First stream use enrolls the physical table by creating the singleton __lance_mem_wal system index and its sharding configuration.

Enrollment advances Lance HEAD inline. It therefore uses the RFC-022 generic recovery protocol, not an ad-hoc state machine:

1. run and await RFC-022's synchronous recovery barrier;
2. authorize, pin the manifest/schema/table state, and prepare a complete ReadSet containing schema identity, table entry/head, PK metadata, stream intent/configuration, and lifecycle-row absence;
3. acquire any global claims and then the (table, branch) write queue in RFC-022 order, then freshly revalidate the complete ReadSet; a mismatch restarts before any inline effect;
4. verify RFC-023's already-installed PK; enrollment never performs a first-use PK migration; validate the sharding configuration;
5. write a generic sidecar with writer kind stream_enrollment and pre-commit table pin;
6. create the MemWAL index, advancing Lance HEAD;
7. publish the new table version plus stream_state = OPEN in one manifest CAS, including the table-head row when RFC-024 is active;
8. delete the sidecar best-effort after publication.

Recovery rolls the enrollment forward or back under the same classification and audit machinery as other inline residuals. Repeating enrollment with identical metadata is a no-op. A different PK, sharding spec, maintained-index set, or writer-default configuration is a typed conflict.

No row is acknowledged until enrollment is manifest-committed.

Initial delivery supports one unsharded shard per (table, main). Non-main branches remain refused until the Lance branch-scoping question is proven by a surface guard and end-to-end test. Later bucket(id, N) sharding must preserve the one-key-to-one-shard rule.

4. New public API

The shipped POST /graphs/{id}/ingest path remains the deprecated, compatible alias of /load. Streaming receives a new, non-conflicting surface:

POST /graphs/{graph_id}/streams/{type_name}/ingest?branch=main
GET  /graphs/{graph_id}/streams
GET  /graphs/{graph_id}/streams/{type_name}
POST /graphs/{graph_id}/streams/{type_name}/fold
POST /graphs/{graph_id}/streams/{type_name}/quiesce
POST /graphs/{graph_id}/streams/{type_name}/resume

The ingest request and response use Content-Type: application/x-ndjson and Accept: application/x-ndjson.

Each input line is one row payload. Each output line corresponds to the same input ordinal:

{"ordinal":17,"status":"durable","shard_id":"...","writer_epoch":8,"wal_position":42}

Synchronous validation failures return a per-row error before a WAL append. Previously acknowledged rows in the same request remain durable; the response is a stream, not an all-request transaction. Ordering, cancellation, and retry rules are explicit:

• acknowledgements are emitted in input order for one HTTP stream;
• disconnecting does not cancel entries whose durability waiter resolved;
• a missing response is ambiguous; retrying the same id and payload may add another WAL entry but produces the same last-write-wins graph state;
• server shutdown stops admission, drains durability waiters up to a bound, and reports any unacknowledged tail as unknown to the client.

CLI commands mirror the new namespace rather than overloading deprecated omnigraph ingest:

omnigraph stream ingest <type> --data <ndjson> ...
omnigraph stream status [<type>] ...
omnigraph stream fold [<type>] ...
omnigraph stream quiesce [<type>] ...
omnigraph stream resume [<type>] ...

Every endpoint has a dedicated OpenAPI operation and handler tests. Ingest passes the engine stream_ingest Cedar action and per-actor admission accounting before acquiring a shard writer; fold/quiesce/resume use a separate stream_manage action. Status is authorized like other graph operational metadata. The same engine gates apply to embedded and remote CLI use.

5. Ack-path validation and writer lifecycle

Before append, OmniGraph applies checks that need no base-table read: Arrow shape/type, required/default fields, enum/range/check constraints, reserved columns, and stream mode. RI, cardinality, cross-version uniqueness, and external embedding computation remain fold-time work.

One warm ShardWriter is held per active shard behind a bounded registry. The registry has idle eviction and hard limits for resident writers, MemTable bytes, unflushed WAL bytes, pending generations, and per-actor inflight bytes. Exceeding a bound backpressures with a typed retryable response; it never drops a row.

Initial topology has one active ingest owner for each (graph, table, main) shard. MemWAL's epoch fence makes restart/failover safe; it is not a load balancer. A deployment with multiple server replicas must route a shard to its current owner (or return a typed retry/redirect) instead of letting replicas reclaim the epoch per request. General multi-owner routing waits for the multi-shard phase and its ownership protocol.

6. Fold protocol

The fold consumes flushed generations in ascending order. Embeddings and base-dependent validation run outside the table queue and register every probed table plus stream configuration/generation in RFC-022's ReadSet. The commit phase then:

1. stages accepted rows with Lance merge-insert and includes merged_generations in that transaction;
2. stages any rejection/audit rows required by §7;
3. acquires every affected queue in canonical sorted order and revalidates the complete ReadSet; any mismatch discards and replans the whole fold;
4. writes one generic RFC-022 recovery sidecar before the first commit_staged call;
5. commits every staged Lance transaction;
6. publishes all data/internal table versions and lineage in one __manifest CAS, including table heads when RFC-024 is active;
7. deletes the sidecar after successful publication.

The sidecar is mandatory even though merge-insert is staged. After commit_staged, Lance HEAD and merged_generations have moved while the graph manifest has not. A failure in that window is the ordinary multi-table recovery gap, not invisible staged state.

MemWAL generation GC starts only after the exact fold is graph-visible, its sidecar is resolved, index catchup permits reclamation, and no FreshReadCut retention guard references the generation. Data-HEAD merge progress alone is never permission to delete the only fresh-tier copy.

Concurrent folders reload merged_generations: a generation already committed is skipped; otherwise the fold is replanned from current state. A forced fold stops new flush creation for its cut, waits for in-flight durability waiters, and never aborts an acknowledged row.

Fold lineage uses omnigraph:ingest as the mechanism actor and persists the authenticated contributor actor for every folded WAL range in the same internal audit participant. Commit and status output can therefore answer both who ran the fold and who supplied the data after WAL GC.

7. Fold-time rejection is atomic

strict is the default. A permanent RI, cardinality, uniqueness, or embedding failure stops the fold at the offending generation, marks the shard blocked, and backpressures new ingestion once configured lag bounds are reached.

dead_letter is explicit. _ingest_rejects is then a versioned internal Lance table and a participant in the same fold pipeline, not best-effort state outside the commit protocol. Every reject has deterministic identity:

(table_key, shard_id, generation, wal_position)

The fold stages reject rows, accepted rows, and merge progress before committing any of them. The generic sidecar covers every participant; the single manifest publish records both the base-table and reject-table versions. Replay is idempotent by reject identity. There is no ordering in which progress can become visible while the corresponding rejection is lost.

stream status reports blocked generations and typed reject details. Reject retention is explicit and cannot be shorter than the WAL/fold audit retention needed to explain a durable acknowledgement.

8. Epoch-fenced quiescence barrier

Branch operations, schema changes, stream teardown, and Lance upgrades require a real barrier, not an empty check.

Each enrolled table has a durable stream_state:<stable-table-id>:<incarnation> row in its manifest branch with OPEN | DRAINING | SEALED, configuration hash, and epoch floor. The row is the logical lifecycle authority and is updated by an RFC-022 CAS; MemWAL shard epochs are the physical writer fence. Neither an in-memory registry nor an empty-generation observation can substitute for both. Lifecycle-only transitions are audited manifest metadata transactions; they do not create graph-content commits or move graph_head.

The shared drain sequence is:

1. publish stream intent OPEN -> DRAINING for the target table/branch;
2. increment and persist each shard writer epoch and seal claims, fencing stale writers across processes;
3. reject or backpressure new appends;
4. wait for in-flight durability waiters, flush active MemTables, and fold every generation to empty;
5. verify shard manifests and base merged_generations agree on emptiness;
6. publish DRAINING -> SEALED with the verified generation/epoch cut;
7. for an operation-scoped drain, perform the guarded operation; persistent public quiesce stops after step 6.

Each lifecycle CAS is an RFC-022 authority-first metadata write; each fold is a separate normal RFC-022 graph write. DRAINING fully encodes the target epoch floor, so an interrupted epoch mutation is idempotently resumed from that row. The sequence is not one giant sidecar spanning multiple commits.

There are two dispositions after the drain reaches SEALED:

• operation-scoped drain — branch/schema maintenance automatically publishes SEALED -> OPEN with a newer epoch only after the guarded operation succeeds and the stream contract remains compatible;
• persistent quiesce — the public quiesce command leaves the stream SEALED. It never auto-reopens. stream resume explicitly revalidates schema, PK, configuration, MemWAL format, and epoch, then publishes a newer OPEN state. Stream teardown deletes intent only from SEALED.

The barrier never holds the table write queue while waiting for a fold that needs that queue. State transition and epoch fencing happen first; fold commit then acquires the normal queue. Crash recovery resumes from the durable state and epoch.

Schema apply must drain every affected enrolled type before changing fields, constraints, PK, embeddings, or @stream and resumes only when compatible.

A Lance version upgrade requires persistent stream quiesce --all, but empty generations alone are insufficient: the MemWAL system index, shard manifests, epoch records, and generation directories may still use the old format. Before the bump, the implementation must prove one of: (a) upstream guarantees and cross-version tests cover every retained MemWAL artifact, (b) a public Lance metadata migration converts them, or (c) OmniGraph tears down the enrolled MemWAL metadata under recovery and re-enrolls after the bump. Without one of those gates the upgrade refuses; resume never opens unverified old metadata.

9. Fresh-read cuts

Freshness is a first-class engine/IR enum:

Committed
Fresh

At query planning, Fresh captures one FreshReadCut containing:

• the ordinary manifest snapshot;
• each selected shard-manifest version and writer epoch;
• included flushed-generation paths and maximum generation;
• the active same-process MemTable row-position watermark, when available;
• the base table's merged_generations and index-catchup state read from the exact table version selected by the manifest snapshot, never from live HEAD.

Capture uses a retrying handshake:

1. read the selected shard manifests/epochs, acquire Lance generation retention guards for the flushed files in the tentative cut, and under one same-process writer snapshot capture/pin any active-MemTable watermark;
2. pin the graph manifest snapshot and read merged_generations from each exact base-table version it selects;
3. re-read the shard manifest versions/epochs; any epoch/configuration change restarts the whole capture;
4. if a generation from step 1 disappeared, accept that only when the pinned base's merged_generations proves it is included; otherwise release guards, discard the whole graph snapshot, and retry from step 1;
5. exclude generations that appeared after step 1 and hold the generation and MemTable read guards captured in step 1 until query completion.

If Lance exposes no guard that prevents generation GC for the query lifetime, cross-process Fresh does not ship. A missing generation is never interpreted as “probably folded” against an older pinned base.

Execution never refreshes that cut mid-query. It excludes every flushed generation <= merged_generations[shard]; otherwise old WAL data could outrank or duplicate its newer base-table image.

Fresh reads have no cross-table atomicity. Same-process active MemTables provide read-your-writes; other processes can promise only the latest flushed state captured by their shard-manifest reads. The HTTP request and query docs state those limits wherever the tier is exposed.

10. Observability and resource contracts

Per shard expose durable WAL position, replay position, active epoch, current generation, flushed and merged generation, index catchup, pending rows/bytes, oldest acknowledged age, last fold error, blocked reject, and quiescence state.

Metrics cover ack latency, durability-wait batching, fenced writers, replayed entries, fold rows/bytes/generations, fold retries, lag, reject counts, and sidecar recovery. Defaults for every byte/count/time bound are documented and configuration changes are observable behavior.

stream status resolves the exact lifecycle rows and MemWAL metadata through a structured, bounded access path; it may reuse RFC-024's scalar-index machinery but cannot claim history-flat cost while scanning manifest history.

11. Acceptance gates

• Surface guards pin claim, append, durability waiter, flush, epoch fencing, staged merge with merged_generations, index catchup, and seal/reopen APIs.
• A WAL append failure emits no durable acknowledgement. Every acknowledged row survives crash, replay, fold, and recovery.
• Failpoints cover enrollment's inline-commit gap and every fold participant around sidecar, commit_staged, reject persistence, and manifest publish.
• Enrollment restarts without an inline effect when schema, PK, table head, or stream configuration changes between prepare and gated revalidation.
• Two folders converge exactly once; a fenced stale writer can never produce a false durable acknowledgement after WAL GC.
• Two server replicas do not epoch-ping-pong one shard; owner failover fences the old process before the new owner acknowledges.
• Quiescence tests race appends with branch and schema operations across two coordinators and prove no post-drain tail appears; failpoints cover every lifecycle-row, epoch-fence, fold, and SEALED boundary.
• Persistent quiesce never auto-reopens; explicit resume validates a newer epoch. Upgrade tests cover every retained MemWAL artifact through declared compatibility, migration, or teardown/re-enrollment.
• Fresh reads race capture with fold/GC, retry on an unexplained disappearing generation, hold generation/MemTable guards through execution, exclude merged generations, and document cross-table inconsistency explicitly.
• Server/OpenAPI tests preserve the old /ingest alias and cover the new route; CLI parity covers embedded and remote stream commands.
• Ack-path object-store operations are O(1) and flat in graph history and WAL depth. Fold cost is bounded by generations/rows folded, not graph history.
• S3 correctness runs against RustFS; API/format guards are rerun before every Lance bump.

12. Phasing

Phase	Content	Gate
A	MemWAL adapter, surface guards, enrollment sidecar	inline-commit crash matrix
B	new ingest route/CLI, durable ack, strict fold	ack durability; API compatibility; cost budget
C	atomic dead letter, audit provenance, status/bounds	reject crash matrix; backpressure tests
D	epoch-fenced drain, persistent quiesce/resume, schema/branch/upgrade integration	two-coordinator race and format-transition suite
E	fresh cuts and maintained-index reads; cross-process Fresh ships only if the substrate generation-retention guard exists (§9), otherwise same-process only	cut consistency; merged-generation exclusion
F	multi-shard upsert and stream deletes	one-key-one-shard proof; Lance re-audit