Merge branch 'main' into devin/1778632299-mr-854-phase-1b-phase-9

docs/dev/codeowners.md

@@ -14,8 +14,8 @@ The tables below are **generated** from `.github/codeowners-roles.yml` by `.gith
 
 | Path | Owners | Role(s) |
 |---|---|---|
-| `*` | @ragnorc | engineering |
-| `crates/**` | @ragnorc | engineering |
+| `*` | @ragnorc @aaltshuler | engineering |
+| `crates/**` | @ragnorc @aaltshuler | engineering |
 | `docs/**` | @ragnorc | docs |
 | `README.md` | @ragnorc | docs |
 | `AGENTS.md` | @ragnorc | docs |
@@ -26,7 +26,7 @@ The tables below are **generated** from `.github/codeowners-roles.yml` by `.gith
 
 | Role | Members | Description |
 |---|---|---|
-| `engineering` | @ragnorc | All production code under crates/**. Engine, CLI, server, compiler. |
+| `engineering` | @ragnorc @aaltshuler | All production code under crates/**. Engine, CLI, server, compiler. |
 | `docs` | @ragnorc | Documentation under docs/**, plus repo-level docs (README.md, AGENTS.md, CLAUDE.md symlink, SECURITY.md). |
 
 <!-- END GENERATED OWNERSHIP -->

docs/dev/index.md

@@ -51,6 +51,18 @@ constraints. User-facing behavior should still be documented through
 | Install and deployment packaging | [install.md](../user/install.md), [deployment.md](../user/deployment.md) |
 | Release history | [releases/](../releases/) |
 
+## Contribution & Governance
+
+| Area | Read |
+|---|---|
+| How to contribute (external) | [CONTRIBUTING.md](../../CONTRIBUTING.md) |
+| Governance model, roles, decision authority | [GOVERNANCE.md](../../GOVERNANCE.md) |
+| Public contribution RFC track | [rfcs/](../rfcs/) |
+
+The `docs/rfcs/` track is the **public, externally-authorable** RFC process. The
+maintainer/internal RFCs below (`rfc-00N-*.md`) are a separate, team-owned
+track; don't conflate the two.
+
 ## Active Implementation Plans
 
 Working documents for in-flight feature work. Removed when the work lands.

docs/dev/writes.md

@@ -14,8 +14,11 @@ publisher's row-level CAS on `__manifest` is the single fence.
 
 - No `RunRecord`, no `_graph_runs.lance`, no `_graph_run_actors.lance`.
 - No `omnigraph run *` CLI subcommands and no `/runs/*` HTTP endpoints.
-- No `__run__<id>` staging branches. (Legacy on-disk artifacts from
-  pre-MR-771 repos are inert; MR-770 sweeps them in production.)
+- No `__run__<id>` staging branches; `__run__*` is no longer a reserved
+  name. The branch-name guard was removed in MR-770, and any stale
+  `__run__*` branch on an upgraded graph is swept off `__manifest` by the
+  v2→v3 internal-schema migration on first read-write open. (The inert
+  `_graph_runs.lance` bytes remain until a `delete_prefix` primitive lands.)
 - Cancelled mutation futures leave **no graph-level state** — only orphaned
   Lance fragments, which the existing `omnigraph cleanup` pipe reclaims.
 
@@ -241,9 +244,14 @@ list`.
 
 ## Migration code
 
-`db/manifest/migrations.rs` does not change. Active deletion of
-`_graph_runs.lance` belongs in MR-770 (the production sweep) — this PR
-stops *creating* run state but does not destroy legacy bytes on disk.
+`db/manifest/migrations.rs` carries the v2→v3 internal-schema step (MR-770):
+a one-time sweep that deletes legacy `__run__*` staging branches off
+`__manifest`. It runs in `Omnigraph::open(ReadWrite)` (via
+`manifest::migrate_on_open`, before the coordinator reads branch state) and
+again on the publisher's write path; both are idempotent once the stamp is at
+v3. Deleting the inert `_graph_runs.lance` / `_graph_run_actors.lance` dataset
+*bytes* is still deferred — it needs a `StorageAdapter::delete_prefix`
+primitive — but those bytes are invisible to graph-level state.
 
 ## Mid-query partial failure: closed by MR-794
 

docs/releases/v0.6.1.md

@@ -7,6 +7,7 @@ v0.6.1 focuses on operational polish after v0.6.0: stored-query registries, safe
 - **Stored-query registries.** `omnigraph.yaml` can declare curated `queries:` blocks per graph. Servers load and type-check them at startup, `omnigraph queries validate` checks them offline, `omnigraph queries list` shows exposed queries and typed params, `GET /queries` exposes a typed catalog, and `POST /queries/{name}` invokes a stored query without accepting ad hoc `.gq` source from the client.
 - **Stored-query policy gate.** New Cedar action `invoke_query` gates the stored-query invocation surface. Stored mutations are double-gated: `invoke_query` to reach the stored query and `change` for the actual write.
 - **Safer branch deletion.** `branch_delete` now treats the manifest as the authority, flips branch visibility atomically, and reclaims per-table/commit-graph forks as derived state. If best-effort reclaim is interrupted, `cleanup` reconciles orphaned forks; reusing a branch name before cleanup reports an actionable error.
+- **Legacy `__run__` cleanup (MR-770).** Removed the last functional remnant of the Run state machine (retired in v0.4.0): the `__run__` branch-name guard. A new v2→v3 `__manifest` internal-schema migration sweeps any stale `__run__*` staging branches on the first read-write open, so `__run__*` is no longer a reserved branch name. This closes the "unpromoted `__run__` branches block reads" condition behind the zombie-run cascade incident; the inert `_graph_runs.lance` row cleanup is tracked separately (it needs a `delete_prefix` primitive).
 - **Blob-safe optimize.** `omnigraph optimize` skips tables with `Blob` properties instead of failing the whole sweep on Lance's blob-v2 compaction decode bug. Skips are visible in human output, `--json` as `skipped`, `TableOptimizeStats.skipped`, and logs; non-blob tables still compact normally.
 - **Deployment improvements.** The container entrypoint now composes `OMNIGRAPH_TARGET_URI` with `OMNIGRAPH_CONFIG`, so operators can keep the graph URI in env while loading policy/query config from a mounted file. The local RustFS bootstrap pins RustFS beta.3 and allows the current insecure local-dev default credentials.
 - **Windows release support.** Tagged and edge releases now publish Windows x86_64 archives containing `omnigraph.exe` and `omnigraph-server.exe`, with a PowerShell installer and Windows install docs.
@@ -17,6 +18,7 @@ v0.6.1 focuses on operational polish after v0.6.0: stored-query registries, safe
 - A graph selected by name (`--target` or `server.graph`) now uses `graphs.<name>.policy` and `graphs.<name>.queries`. Top-level `policy` / `queries` blocks are only for anonymous bare-URI single-graph mode; using them with a named graph now fails loudly with migration guidance.
 - `mcp.expose` defaults to `true` for stored-query registry entries. Set `mcp: { expose: false }` for service-only queries that should not appear in the catalog.
 - `invoke_query` is graph-scoped, not branch-scoped. Branch/snapshot access remains enforced by the inner `read` / `change` gate.
+- **Legacy `__run__` migration.** Graphs created before v0.4.0 are migrated automatically on the first **read-write** open by a v0.6.1 binary (one-time `__manifest` stamp v2→v3 sweep of stale `__run__*` branches). No action required. Two caveats: (1) a graph opened **read-only** still lists any stale `__run__*` branch until its first read-write open, since the migration is write-path-only like all manifest migrations — long-lived read-only deployments should be opened read-write once after upgrading; (2) the inert `_graph_runs.lance` / `_graph_run_actors.lance` dataset bytes are left in place until a future `delete_prefix` primitive (they are invisible to graph-level state).
 - Blob tables are not compacted until the upstream Lance fix lands, so fragment count and deleted-row space on blob tables are not reclaimed by `optimize`. Reads, writes, and query results are unaffected; no on-disk migration is required.
 - `TableOptimizeStats` is now `#[non_exhaustive]` and gains a `skipped: Option<SkipReason>` field (so does the new `SkipReason` enum). This is a source-level change only for downstream code that built this returned result struct by literal — rare, since it is produced by `optimize` and consumed by reading its fields; field access is unaffected, and `#[non_exhaustive]` keeps future additions non-breaking.
 

docs/rfcs/0000-template.md

@@ -0,0 +1,54 @@
+# RFC NNNN: <title>
+
+| | |
+|---|---|
+| **Status** | Proposed |
+| **Author(s)** | <your name / handle> |
+| **Discussion** | <link to the originating Discussion, if any> |
+| **Implementation** | <issue/PR links, filled in as work lands> |
+
+> Status is maintained by maintainers: `Proposed` while the PR is open,
+> `Accepted` on merge, `Declined` on close, `Superseded by NNNN` later.
+
+## Summary
+
+One paragraph: what this changes, in plain terms.
+
+## Motivation
+
+What problem does this solve, and why is it worth the ongoing cost? Tie it to a
+concrete need (a Discussion, a recurring issue, a user request). Per the
+project's first principle, argue the *long-run liability*, not just the
+short-term convenience.
+
+## Guide-level explanation
+
+Explain the change as you'd teach it to a user or contributor: new commands,
+syntax, API shapes, behavior. Examples first.
+
+## Reference-level design
+
+The precise design: data structures, IR/AST/planner changes, storage/format
+impact, migration path, error behavior. Enough that a reviewer can find the
+holes.
+
+## Invariants & deny-list check
+
+Which Hard Invariants in [../dev/invariants.md](../dev/invariants.md) does this
+touch? Does it brush against any deny-list item — and if so, why is this the
+justified exception? State explicitly that no invariant is weakened, or which
+Known Gap moves.
+
+## Drawbacks & alternatives
+
+What does this cost, what did you reject, and why. "Do nothing" is a valid
+alternative to weigh.
+
+## Reversibility
+
+Is this reversible? On-disk/wire/format and substrate choices are near-permanent
+and demand more evidence; a CLI flag or doc is cheap to undo. Say which this is.
+
+## Unresolved questions
+
+What's deliberately left open for review to settle.

docs/rfcs/README.md

@@ -0,0 +1,66 @@
+# RFCs
+
+Substantial changes to OmniGraph — new user-facing surface, format or protocol
+changes, anything irreversible or cross-cutting — go through a lightweight RFC
+so the design is agreed *as reviewable code* before implementation starts. This
+is the public RFC track, open to **anyone, including external contributors**.
+
+This complements the always-on review bar in
+[../dev/invariants.md](../dev/invariants.md): the invariants say *what every
+change must respect*; an RFC says *why this particular change is worth making and
+how*.
+
+> **Two tracks, don't conflate them.** This `docs/rfcs/` directory is the
+> **public contribution** track (anyone authors; maintainers accept). The
+> maintainer-internal RFCs under `docs/dev/rfc-00N-*.md` are a separate,
+> team-owned track for in-flight internal work. If you're an outside
+> contributor, you're in the right place here.
+
+## When you need one
+
+- **RFC required:** new query/schema/CLI/HTTP surface; on-disk or wire-format
+  changes; a new substrate dependency; anything the deny-list in
+  [../dev/invariants.md](../dev/invariants.md) flags; anything irreversible
+  ("reversibility shapes evidence demand").
+- **RFC not required:** bug fixes for an `accepted` issue, and the trivial
+  fast-lane (typos, docs, deps) — see [../../CONTRIBUTING.md](../../CONTRIBUTING.md).
+
+If you're unsure, start a [Discussion](../../../discussions); a maintainer will
+tell you whether it needs an RFC.
+
+## Lifecycle
+
+```
+Discussion (incubate, get rough consensus)
+      │ graduate
+      ▼
+RFC pull request  →  adds docs/rfcs/NNNN-title.md  (Status: Proposed)
+      │
+maintainer review  ──▶  changes requested / declined (PR closed, with rationale)
+      │
+      ▼
+merged  ==  Accepted   (the merged file is the durable decision record)
+      │
+      ▼
+Implementation PR(s)  reference the accepted RFC
+```
+
+- **Author:** anyone. **Acceptance:** a maintainer decision, performed by
+  merging the RFC PR. Declining is closing it with rationale.
+- The merged RFC *is* the accepted record — there is no separate sign-off step.
+- Later reversals don't edit history: supersede with a new RFC that links back
+  and flip the old one's `Status` to `Superseded`.
+
+## Numbering & naming
+
+- File: `docs/rfcs/NNNN-kebab-title.md`, where `NNNN` is the next free
+  zero-padded integer (`0001`, `0002`, …). `0000-template.md` is reserved.
+- Pick the number when you open the PR; if it collides with another in-flight
+  RFC, the second to merge bumps theirs.
+
+## Status values
+
+`Proposed` (open PR) · `Accepted` (merged) · `Declined` (closed) ·
+`Superseded by NNNN` · `Implemented` (set once the work lands, optional).
+
+Copy [0000-template.md](0000-template.md) to start.

docs/user/audit.md

@@ -4,4 +4,4 @@
 - `_as` variants of every write API let callers override the actor: `mutate_as`, `ingest_as`, `branch_merge_as`, `apply_schema_as`, etc.
 - Actor IDs are persisted on `GraphCommit.actor_id` with split storage in `_graph_commit_actors.lance` (the commit graph is split into `_graph_commits.lance` for the linkage and `_graph_commit_actors.lance` for the actor map).
 - HTTP server uses the bearer-token actor automatically; CLI uses the local user / explicit env (no implicit actor).
-- Pre-v0.4.0 graphs also stored actor IDs on `RunRecord.actor_id` in `_graph_runs.lance` / `_graph_run_actors.lance`. The Run state machine was removed in MR-771; those files are inert post-v0.4.0 and reclaimed by MR-770's production sweep.
+- Pre-v0.4.0 graphs also stored actor IDs on `RunRecord.actor_id` in `_graph_runs.lance` / `_graph_run_actors.lance`. The Run state machine was removed in MR-771; those files are inert post-v0.4.0. The v2→v3 manifest migration sweeps any stale `__run__*` branches on first write-open (MR-770); the inert dataset bytes remain until a `delete_prefix` primitive lands.

docs/user/branches-commits.md

@@ -9,8 +9,8 @@ Lance supports branching at the dataset level: a branch is a named lineage of ve
 OmniGraph builds *graph branches* on top by branching every sub-table coherently:
 
 - `branch_create(name)` / `branch_create_from(target, name)` — disallowed name `main`; fails if branch exists; ensures the schema-apply lock is idle. Atomic and authority-first like `branch_delete`: it flips the `__manifest` branch (authority), then creates the derived commit-graph branch, force-dropping any orphaned commit-graph ref left by an incomplete prior delete (the manifest branch is fresh, so a same-named commit-graph branch is provably a zombie). If commit-graph creation fails, the manifest branch is rolled back so the name never half-exists.
-- `branch_list()` — returns public branches, **filters internal** `__run__…` and `__schema_apply_lock__` prefixes.
-- `branch_delete(name)` — refuses if there are descendants or active runs on the branch. The manifest is the single authority for branch existence: deletion flips the `__manifest` branch ref first (one atomic op), after which the branch is gone from every snapshot. The owned per-table forks and the commit-graph branch are derived state, reclaimed best-effort with `force_delete_branch` after the flip. A failure during that reclaim (transient object-store error) does not fail the call or block the authority flip; the leftover forks are unreachable orphans that the [`cleanup`](maintenance.md) reconciler converges. One consequence: if a delete's best-effort reclaim fails, reusing that branch name before the next `cleanup` surfaces a clear error pointing at `cleanup` (the stale fork would otherwise collide on first write).
+- `branch_list()` — returns public branches, **filters the internal** `__schema_apply_lock__` branch.
+- `branch_delete(name)` — refuses if there are descendants on the branch, or if it is the current branch. The manifest is the single authority for branch existence: deletion flips the `__manifest` branch ref first (one atomic op), after which the branch is gone from every snapshot. The owned per-table forks and the commit-graph branch are derived state, reclaimed best-effort with `force_delete_branch` after the flip. A failure during that reclaim (transient object-store error) does not fail the call or block the authority flip; the leftover forks are unreachable orphans that the [`cleanup`](maintenance.md) reconciler converges. One consequence: if a delete's best-effort reclaim fails, reusing that branch name before the next `cleanup` surfaces a clear error pointing at `cleanup` (the stale fork would otherwise collide on first write).
 - **Lazy forking**: a branch only forks a sub-table when that sub-table is first mutated on it. Pure-read branches share fragments with their source. A fork collision is classified by the manifest authority, not by Lance branch versions: if the live manifest already records the fork on the active branch, a concurrent first-write won and the caller gets a retryable "refresh and retry"; if the manifest does not, a physical branch there is an orphan and the caller is pointed at `cleanup`.
 - `sync_branch(branch)` — re-binds the in-memory handle to the latest head of the branch.
 
@@ -51,10 +51,10 @@ Notes:
 
 ## L2 — Internal system branches
 
-Filtered from `branch_list()` but visible to internals:
+Internal or legacy branch refs:
 
-- `__schema_apply_lock__` — serializes schema migrations.
-- `__run__<run-id>` — legacy from the pre-v0.4.0 Run state machine (removed in MR-771). The branch-name guard predicate `is_internal_run_branch` is kept as defense-in-depth so users cannot create a branch matching the legacy prefix; the filter will be removed once production legacy branches are swept (MR-770).
+- `__schema_apply_lock__` — serializes schema migrations; filtered from `branch_list()` but visible to internals.
+- `__run__<run-id>` — legacy from the pre-v0.4.0 Run state machine (removed in MR-771). These are swept off `__manifest` on the first read-write open by the v2→v3 internal-schema migration (MR-770), and `__run__*` is no longer a reserved name. Known limitation: a pre-v0.4.0 graph opened **read-only** still surfaces any stale `__run__*` branch in `branch_list()` until its first read-write open (the migration is write-path-only, like all manifest migrations).
 
 ## L2 — Recovery audit trail
 

docs/user/constants.md

@@ -4,11 +4,11 @@
 |---|---|---|
 | `MANIFEST_DIR` | `__manifest` | `db/manifest/layout.rs` |
 | Commit graph dir | `_graph_commits.lance` | `db/commit_graph.rs` |
-| Run registry dir (legacy, removed MR-771) | `_graph_runs.lance` | inert post-v0.4.0; reclaimed by MR-770 |
-| Run branch prefix (legacy, removed MR-771) | `__run__` | filtered by `is_internal_run_branch` defense-in-depth |
+| Run registry dir (legacy, removed MR-771) | `_graph_runs.lance` | inert post-v0.4.0; bytes remain until a `delete_prefix` primitive lands |
+| Run branch prefix (legacy, removed MR-771/MR-770) | `__run__` | swept off `__manifest` by the v2→v3 migration; no longer a reserved name |
 | Schema apply lock | `__schema_apply_lock__` | `db/mod.rs` |
 | Manifest publisher retry budget | `PUBLISHER_RETRY_BUDGET = 5` | `db/manifest/publisher.rs` |
-| Internal manifest schema version | `INTERNAL_MANIFEST_SCHEMA_VERSION = 2` | `db/manifest/migrations.rs` |
+| Internal manifest schema version | `INTERNAL_MANIFEST_SCHEMA_VERSION = 3` | `db/manifest/migrations.rs` |
 | Merge stage batch | `MERGE_STAGE_BATCH_ROWS = 8192` | `exec/merge.rs` |
 | Maintenance concurrency | `OMNIGRAPH_MAINTENANCE_CONCURRENCY=8` | `db/omnigraph/optimize.rs` |
 | Lance blob compaction support | `LANCE_SUPPORTS_BLOB_COMPACTION = false` | `db/omnigraph/optimize.rs` |

docs/user/storage.md

@@ -22,7 +22,7 @@ OmniGraph is **not** a single Lance dataset; it is a *graph* of datasets coordin
   - `edges/{fnv1a64-hex(edge_type_name)}` — one Lance dataset per edge type
   - `__manifest/` — the catalog of all sub-tables and their published versions
   - `_graph_commits.lance` / `_graph_commit_actors.lance` — the commit graph and its actor map
-  - (legacy `_graph_runs.lance` / `_graph_run_actors.lance` from pre-v0.4.0 graphs are inert; the run state machine was removed in MR-771 and these files are cleaned up via MR-770's production sweep)
+  - (legacy `_graph_runs.lance` / `_graph_run_actors.lance` from pre-v0.4.0 graphs are inert; the run state machine was removed in MR-771. The v2→v3 manifest migration sweeps stale `__run__*` branches on first write-open; the inert dataset bytes themselves remain until a `delete_prefix` storage primitive lands)
 - **Manifest row schema** (`object_id, object_type, location, metadata, base_objects, table_key, table_version, table_branch, row_count`):
   - `object_type` ∈ `table | table_version | table_tombstone`
   - `table_key` ∈ `node:<TypeName> | edge:<EdgeName>`
@@ -47,6 +47,7 @@ Adding a new on-disk shape change is one constant bump (`INTERNAL_MANIFEST_SCHEM
 |---|---|
 | v1 (implicit, pre-stamp) | `__manifest.object_id` had no PK annotation; publisher had no row-level CAS protection. |
 | v2 | `__manifest.object_id` carries `lance-schema:unenforced-primary-key=true`; row-level CAS engaged. Stamped as `omnigraph:internal_schema_version=2`. |
+| v3 | One-time sweep of legacy `__run__*` staging branches (pre-v0.4.0 Run state machine, removed MR-771) off `__manifest`. Runs at `Omnigraph::open(ReadWrite)` and on publish. Stamped as `omnigraph:internal_schema_version=3`. |
 
 ## On-disk layout
 
@@ -91,7 +92,7 @@ flowchart TB
 - **Graph root** is one directory (or S3 prefix). Everything below is part of one OmniGraph graph.
 - **`__manifest/`** is a Lance dataset whose rows describe which sub-table version is published at which graph-branch. Reading a snapshot starts here.
 - **`nodes/`** and **`edges/`** are sibling directories holding one Lance dataset per declared type. Names are `fnv1a64-hex` of the type name to keep paths fixed-length and case-safe.
-- **`_graph_commits.lance`** is an L2 dataset that records the graph-level commit DAG, with a paired `_graph_commit_actors.lance` for the actor map. (Pre-v0.4.0 graphs also have inert `_graph_runs.lance` / `_graph_run_actors.lance` from the removed Run state machine; MR-770 sweeps these in production.)
+- **`_graph_commits.lance`** is an L2 dataset that records the graph-level commit DAG, with a paired `_graph_commit_actors.lance` for the actor map. (Pre-v0.4.0 graphs also have inert `_graph_runs.lance` / `_graph_run_actors.lance` from the removed Run state machine; the v2→v3 migration sweeps their stale `__run__*` branches, and the dataset bytes are reclaimed once `delete_prefix` lands.)
 - **`_graph_commit_recoveries.lance`** — one row per recovery sweep action. Joined to `_graph_commits.lance` by `graph_commit_id`; the linked commit row carries `actor_id=omnigraph:recovery`. Operators correlate recoveries with the original mutations they rolled forward / back via this join. See `crates/omnigraph/src/db/recovery_audit.rs`.
 - **`__recovery/{ulid}.json`** — transient sidecar files written by the four migrated writers (`MutationStaging::finalize`, `schema_apply`, `branch_merge`, `ensure_indices`) before Phase B begins, deleted after Phase C succeeds. A sidecar persisting after process exit means the writer crashed in the Phase B → Phase C window; the next `Omnigraph::open` recovery sweep processes it. Steady-state directory is empty. See `crates/omnigraph/src/db/manifest/recovery.rs`.
 - **`_refs/branches/{name}.json`** is graph-level branch metadata — pointers from a branch name to the manifest version it heads.