omnigraph/docs/dev/rfc-006-object-storage-native.md

RFC-006: Object-Storage-Native OmniGraph

Status: Proposed Depends on: RFC-005 (cluster serving, landed), Phase 4 (landed) Decides: how the cluster control plane migrates from raw-filesystem I/O to object storage, and what "local filesystem support" means afterwards.

Motivation

The engine is already object-storage-native: Lance datasets live behind the object_store abstraction, S3/RustFS graphs are CI-tested, and the classic single-graph deployment runs stateless against a bucket. The cluster control plane is not: the state ledger, lock, catalog blobs, recovery sidecars, approval artifacts, and — most consequentially — the derived graph roots are all raw fs::* against the config directory. Consequences:

• A cluster cannot put its data on S3 at all; cloud deployments need a persistent volume, abandoning the stateless-bucket shape the classic mode already has (cookbooks PR #12 validated it on Railway).
• The control plane carries a second storage layer with different semantics (rename-CAS, read_dir, remove_dir_all) instead of the one the substrate already provides.

The directive this RFC implements: OmniGraph is object-storage only. One storage interface; every stored byte addressed by URI; S3-compatible object storage is the deployment model. The local filesystem does not disappear — it is demoted from "a separate code path" to "one backend of the same interface" (file://), retained for development and tests, and removed from the production story.

Verified foundations

• S3 conditional writes work on the target backend. Tested against RustFS 1.0.0-beta.8 (2026-06-11): PUT with If-None-Match: * → first write 200, second 412; PUT with If-Match: <etag> → fresh etag 200, stale 412. AWS S3 has shipped both since 2024/2025; the object_store crate exposes them as PutMode::Create / PutMode::Update(version).
• Lance 6.x commits natively via S3 conditional writes — the engine's own multi-version safety on S3 needs no external lock table.
• Omnigraph::init/open accept S3 URIs today — derived graph roots on S3 require no engine work.

Design

D1. One storage interface (the rule)

A new sealed ClusterStore abstraction (thin over the object_store crate, reusing the engine's storage_for_uri URI plumbing) carries every cluster read/write: state ledger, lock, catalog payloads, recovery sidecars, approval artifacts. Raw fs::* in omnigraph-cluster for stored state becomes a deny-list entry in invariants.md. Backends: s3:// (and any S3-compatible endpoint — RustFS, MinIO, R2) and file:// (dev/test).

The one deliberate exception: declared configuration — cluster.yaml and the .pg/.gq/policy files it references — stays in the operator's working tree, read-only, exactly like Terraform reads .tf files locally while the state backend is remote. Config is versioned in git; state and data live in the store.

D2. The storage: root

version: 1
storage: s3://omnigraph-local/clusters/intel   # the cluster's home
graphs:
  spike:
    schema: schema.pg          # config: read from the working tree
    queries: queries/

Everything currently under <config-dir>/__cluster/ and <config-dir>/graphs/ moves under the storage root:

s3://omnigraph-local/clusters/intel/
├── state.json                  # the ledger (CAS via conditional put)
├── lock.json                   # create-only put; force-unlock = delete
├── resources/…                 # content-addressed catalog (immutable puts)
├── recoveries/…                # sidecars
├── approvals/…                 # approval artifacts
└── graphs/<id>                 # derived Lance roots (engine-native S3)

During the migration window storage: defaults to file://<config-dir> (today's layout, byte-compatible). After the deprecation boundary (D8) the key is required — naming your storage is the point; file:// remains legal but explicit.

Credentials are never in cluster.yaml: the standard AWS_* env contract (already documented for the engine) applies to the control plane identically.

D3. Ledger CAS and locking on object storage

• write_state (today: temp file + rename, guarded by recorded state_cas) becomes PutMode::Update(etag) — the etag read with the state replaces the sha256 sidecar field as the CAS token (the sha256 stays as content identity in audit/output). A 412 maps to the existing state_cas_conflict path.
• acquire_lock becomes PutMode::Create on lock.json; 412 → held (read the holder for the message); force-unlock <id> = read, verify id, delete. Same semantics as today, now correct across machines — which the file backend never was.
• Latency: one GET + one conditional PUT per command on the happy path — noise next to graph opens. The recovery sweep adds a LIST of recoveries/.

D4. Catalog, sidecars, approvals

Mechanical ports: content-addressed payloads are immutable puts (idempotent by construction — a re-put of the same digest is a no-op); sidecars and approvals are small JSON objects with LIST + GET + DELETE lifecycles. Approval files gain nothing; their digest-binding semantics are storage-agnostic.

D5. Derived graph roots become URIs

<storage>/graphs/<id> replaces <config-dir>/graphs/<id>.omni. Executors: create = Omnigraph::init(uri) (works today); schema apply = open(uri) (works today); approved delete = object-store prefix delete replacing remove_dir_all. The recovery sweep's root.exists() becomes a prefix LIST (non-empty = exists). Tombstones and the digest classification logic are unchanged — they never depended on the filesystem.

D6. Serving from a bucket

omnigraph-server --cluster <uri> accepts the storage root URI directly (--cluster s3://omnigraph-local/clusters/intel). read_serving_snapshot reads ledger + catalog through ClusterStore; graphs open by their S3 URIs. A cluster deployment becomes stateless again: no volume, restart-to-adopt unchanged, replicas trivially safe (boot is read-only). Railway = service + Bucket; ECS = task + S3; the PR-#12 topology and the cluster topology converge.

D7. Migration tooling

omnigraph cluster migrate-storage <dest-uri> --config <dir>: object-copy graphs + catalog + approvals to the destination (Lance layouts are path-relative; immutable files copy safely), write the ledger last via create-only put, then print the storage: line to commit into cluster.yaml. Idempotent and resumable (copy is keyed by listing diff; the ledger write is the atomic cutover). The reverse direction works identically (S3 → file:// for local debugging). Fallback path: cluster import against live S3 graphs already reconstructs a lost ledger.

D8. Deprecation of local-FS as an operating mode

Per axiom 15's bridge rule (every bridge names its replacement and sunset):

Phase	local FS status
Now → Stage C lands	implicit default (storage: absent ⇒ file://<config-dir>)
Stage D (docs flip)	S3-first everywhere: docs, cookbooks, skills, deployment recipes; file:// documented only under development/testing; absent storage: emits a deprecation warning naming this RFC
v0.9 boundary	storage: required; file:// stays a legal explicit backend for dev/test — it is the same code path, costs no second implementation, and keeps cargo test hermetic (no daemon dependency in unit tests)

Recommendation embedded here (the one place this RFC pushes back): "object storage only" is enforced at the interface level — one code path, every location a URI — not by deleting the file:// backend. Hard removal would force a RustFS daemon into every unit test and air-gapped dev loop while deleting zero code (the backend ships inside object_store either way). Terraform's local state backend survives for the same reason its S3 backend is still the only one anyone deploys. If a harder line is wanted later, it is a docs-and-validation flip, not an architecture change.

Staging

Stage	Delivers	Size
A	ClusterStore + ledger/lock/sidecars/approvals/catalog ported; file:// behavior byte-compatible; S3 backend live behind storage:; conditional-put CAS + cross-machine lock; RustFS-gated integration tests	the big one — touches every backend call in omnigraph-cluster
B	URI graph roots: executor init/apply/delete + sweep on URIs; prefix-delete; e2e: full lifecycle against RustFS	medium
C	--cluster <uri> serving + bucket-backed snapshot reads; system e2e: apply to RustFS, serve from it; Railway Bucket deploy validated (closes the loop with cookbooks PR #12's topology)	medium
D	migrate-storage, docs flip (S3-first), cookbooks/skills update, deprecation warning, deny-list entry	small code, wide docs

Each stage is a PR with the usual gates; A and B are separable but land best back-to-back (B is where the user-visible payoff starts).

Open questions

1. RustFS GA & conditional-write contract stability — beta.8 passes both probes; pin the probes as a lance_surface_guards-style integration test so a regression in a RustFS bump turns red here, not in production.
2. Multi-writer ergonomics — conditional puts make concurrent applies safe (one wins, one gets state_cas_conflict); whether we want lease semantics (lock TTL + auto-break) is a later UX question, not a correctness one.
3. Catalog GC on object storage — deletes leave blobs today on FS too; the existing gap carries over unchanged, tracked separately.
4. --config ergonomics — once state is remote, two operators sharing a bucket need only the config repo; document the "config in git, state in S3" workflow as the primary pattern (it is the Terraform workflow).

What this explicitly does not change

Engine storage (already object-store native), the .pg/.gq languages, the plan/apply/approve model, recovery semantics (sidecar classification is storage-agnostic), the serving API surface, and omnigraph.yaml's per-operator role.