fix(engine): close the 2 Lance 7.0.0 alignment failures (immutable PK + native namespace) (#236)

* fix(engine): make the v1→v2 manifest migration idempotent under Lance 7's immutable unenforced primary key

Lance 7 (dataset/transaction.rs) makes the unenforced primary key immutable
once set: any write touching the reserved `lance-schema:unenforced-primary-key`
field metadata after the PK is set errors "cannot be changed once set" — even
re-applying the same value. `migrate_v1_to_v2` previously relied on the old Lance
6 idempotency (re-applying the annotation was a no-op-ish bump), which it needs
for crash-recovery: a v1 graph that crashes after the field-set but before the
stamp bump re-enters the migration with the PK already present.

Under Lance 7 that re-entry now errors, so a real pre-v0.4.0 graph crashing in
that window could never complete its migration. Guard the field-set with
`schema().unenforced_primary_key().is_empty()` so a genuine first-set still runs
but a re-set is skipped — restoring crash-idempotency by construction. (Fresh
graphs bake the PK into manifest_schema() at init and never run this migration.)

The existing test_publish_migrates_pre_stamp_manifest_to_current_version is the
regression guard: red under Lance 7 before this change, green after.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(engine): realign the native-namespace surface guard to Lance 7 (TableNotFound)

`test_directory_namespace_direct_publish_cannot_replace_native_omnigraph_write_path`
pokes Lance's NATIVE DirectoryNamespace (not omnigraph's production write path,
which is the manifest merge_insert publisher) to document that it cannot replace
omnigraph's authority.

Lance 7's DirectoryNamespace routes list/describe/create_table_version through
`check_table_status`, which now reports an omnigraph-manifest-tracked table as
absent — so all three return TableNotFound for `node:Person` (observed). The
native namespace is now fully decoupled from omnigraph's manifest: it cannot
enumerate, inspect, or publish over omnigraph's tables. This strengthens the
guard's thesis. Realigned the assertions to the v7 behavior and kept the
authority check (omnigraph's refresh ignores the direct append; row_count stays
0). Test-only; no production impact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(lance): document the 2 runtime behavior changes in the 7.0.0 alignment stanza

The #229 stanza verified a clean engine *build* but not the test suite, and
claimed "no Lance API surface omnigraph uses changed." Two runtime behaviors did,
caught only by the full test suite:

- the unenforced primary key is immutable once set in v7 (transaction.rs) — broke
  the v1→v2 manifest migration's crash-idempotency; fixed by an is-set guard;
- the native DirectoryNamespace returns TableNotFound for omnigraph
  manifest-tracked tables (dir.rs) — test-only; the surface guard was realigned.

Corrects the over-broad "no surface changed" claim, adds both findings, and notes
the lesson: a clean build is not a clean alignment — run cargo test --workspace
before declaring a Lance bump done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

crates/omnigraph/src/db/manifest/migrations.rs

@@ -113,20 +113,27 @@ pub(super) async fn migrate_internal_schema(dataset: &mut Dataset) -> Result<()>
 /// so the merge-insert conflict resolver enforces row-level CAS at commit
 /// time, then bump the stamp.
 ///
-/// Both steps are idempotent under retry: re-applying the field annotation
-/// at its current value is a no-op-ish bump in Lance, and the stamp is a
-/// simple key-value write. A crash between the two leaves the field set
-/// without a stamp; the next open re-runs this fn and only the stamp lands.
+/// Idempotent under crash-retry by construction. Lance 7 makes the unenforced
+/// primary key **immutable once set**: any write that touches the reserved
+/// `lance-schema:unenforced-primary-key` field metadata after the PK is set
+/// errors ("cannot be changed once set", `lance::dataset::transaction`), even
+/// re-applying the same value. A crash between the field-set and the stamp
+/// bump leaves the field set without a stamp, so the next open re-enters here
+/// with the PK already present — we must therefore set it only when absent.
+/// (Fresh graphs bake the PK into `manifest_schema()` at init and never run
+/// this migration; only genuine pre-v0.4.0 graphs do.)
 async fn migrate_v1_to_v2(dataset: &mut Dataset) -> Result<()> {
-    dataset
-        .update_field_metadata()
-        .update(
-            "object_id",
-            [(OBJECT_ID_PK_KEY.to_string(), "true".to_string())],
-        )
-        .map_err(|e| OmniError::Lance(e.to_string()))?
-        .await
-        .map_err(|e| OmniError::Lance(e.to_string()))?;
+    if dataset.schema().unenforced_primary_key().is_empty() {
+        dataset
+            .update_field_metadata()
+            .update(
+                "object_id",
+                [(OBJECT_ID_PK_KEY.to_string(), "true".to_string())],
+            )
+            .map_err(|e| OmniError::Lance(e.to_string()))?
+            .await
+            .map_err(|e| OmniError::Lance(e.to_string()))?;
+    }
     set_stamp(dataset, 2).await
 }
 

crates/omnigraph/src/db/manifest/tests.rs

@@ -336,40 +336,66 @@ async fn test_directory_namespace_direct_publish_cannot_replace_native_omnigraph
         .await
         .unwrap();
 
-    let versions = namespace
-        .list_table_versions(ListTableVersionsRequest {
-            id: Some(vec!["node:Person".to_string()]),
-            descending: Some(true),
-            ..Default::default()
-        })
-        .await
-        .unwrap();
-    assert_eq!(
-        versions.versions[0].version as u64,
-        person_entry.table_version
+    // Lance 7: the native `DirectoryNamespace` no longer recognizes omnigraph's
+    // manifest-tracked tables. `check_table_status` reports the table absent
+    // (`lance-namespace-impls` dir.rs), so list / describe / create_table_version
+    // all return `TableNotFound`. The native path is fully decoupled from
+    // omnigraph's manifest authority: it cannot enumerate, inspect, or publish
+    // over omnigraph's tables. (Pre-v7 the native namespace could still *see* the
+    // published version but never replace the write path; v7 only widens the gap.)
+    let assert_table_not_found = |what: &str, dbg: String| {
+        assert!(
+            dbg.contains("TableNotFound") && dbg.contains("node:Person"),
+            "{what}: expected TableNotFound for node:Person, got: {dbg}"
+        );
+    };
+    assert_table_not_found(
+        "list_table_versions",
+        format!(
+            "{:?}",
+            namespace
+                .list_table_versions(ListTableVersionsRequest {
+                    id: Some(vec!["node:Person".to_string()]),
+                    descending: Some(true),
+                    ..Default::default()
+                })
+                .await
+                .unwrap_err()
+        ),
+    );
+    assert_table_not_found(
+        "describe_table_version",
+        format!(
+            "{:?}",
+            namespace
+                .describe_table_version(DescribeTableVersionRequest {
+                    id: Some(vec!["node:Person".to_string()]),
+                    version: Some(person_version as i64),
+                    ..Default::default()
+                })
+                .await
+                .unwrap_err()
+        ),
+    );
+    assert_table_not_found(
+        "create_table_version",
+        format!(
+            "{:?}",
+            namespace
+                .create_table_version(version_metadata.to_create_table_version_request(
+                    "node:Person",
+                    person_version,
+                    1,
+                    None,
+                ))
+                .await
+                .unwrap_err()
+        ),
     );
 
-    let err = namespace
-        .describe_table_version(DescribeTableVersionRequest {
-            id: Some(vec!["node:Person".to_string()]),
-            version: Some(person_version as i64),
-            ..Default::default()
-        })
-        .await
-        .unwrap_err();
-    assert!(err.to_string().contains("not found"));
-
-    let err = namespace
-        .create_table_version(version_metadata.to_create_table_version_request(
-            "node:Person",
-            person_version,
-            1,
-            None,
-        ))
-        .await
-        .unwrap_err();
-    assert!(err.to_string().contains("already exists"));
-
+    // omnigraph's manifest stays authoritative: refresh ignores the direct
+    // `person_ds.append` above (it was never manifest-published), so the row
+    // count stays 0 and the version is unchanged.
     mc.refresh().await.unwrap();
     assert_eq!(
         mc.snapshot().entry("node:Person").unwrap().table_version,