schema-lint chassis v1.0: DropProperty Soft + code-tagged diagnostics (MR-694) (#90)

* schema-lint chassis v1 (WIP): tier surfacing + plan doc

First commit of the chassis v1 branch. Lands a small, foundational
slice without behavior change, plus a planning doc that lays out the
remaining 7 commits in sequence so the PR can be reviewed
incrementally.

This commit:

- Adds SchemaMigrationStep::diagnostic() returning the full
  &'static DiagnosticCode (family + tier + severity) for
  UnsupportedChange steps with codes. Renderers can now reach the
  tier without re-implementing the code → tier lookup.

- CLI `omnigraph schema plan` output now displays tier alongside
  code:

    unsupported change on node:Person.age [OG-DS-104, destructive]:
        removing property 'Person.age' is not supported in schema
        migration v1

  Operators see at-a-glance the kind of risk each rejection
  represents — not just the rule identifier.

- No behavior change. All 11 existing schema_apply tests still pass.

Planning doc at docs/schema-lint-v1-plan.md tracks the 7 remaining
commits to bring v1 to feature-complete:

  1. (this commit) Tier surfacing in plan output.
  2. Soft / Hard mode enum on drop steps.
  3. Tombstone fields on catalog IR.
  4. Planner emits DropProperty { Soft } by default.
  5. Apply path implements Soft mode.
  6. Convert PR #62 destructive-rejection tests.
  7. --allow-data-loss flag + Hard mode.
  8. (optional) Tombstone unhide / restore command.

Delete the planning doc when v1 lands. Intentionally checked in to
the WIP branch so the scope is reviewable; not intended as a
permanent doc.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* schema-lint v1 commit 2: DropMode + dormant Drop* variants

Second commit of the chassis v1 branch. Lands the type-level shape
of soft/hard drops without wiring them up. Variants are reachable
from emitters but the planner doesn't produce them yet; the apply
path returns an explicit not-yet-implemented error if one shows up
via deserialization.

Added:

- `DropMode { Soft, Hard }` — orthogonal to `SafetyTier`. Tier
  classifies the rule's risk class; mode is the operator's intent
  for data treatment.
    - `Soft` → catalog tombstone, data retained. Tier: safe.
    - `Hard` → Lance-level removal. Tier: destructive; will require
      --allow-data-loss to apply (commit 7).

- `SchemaMigrationStep::DropType { type_kind, name, mode }` and
  `SchemaMigrationStep::DropProperty { type_kind, type_name,
  property_name, mode }` variants.

- Re-export `DropMode` from `omnigraph_compiler::DropMode` so
  downstream crates don't reach into the catalog submodule.

- CLI `render_schema_plan_step` arms for both variants, surfacing
  the mode in plan output: `drop property 'Person.age' of node
  'Person' (soft mode)`.

- `apply_schema_with_lock` exhaustive match arm for the two new
  variants that returns `manifest_internal` with a clear
  not-yet-implemented message. If a SchemaIR JSON containing
  Drop{Type,Property} arrives (e.g. from a future tool or hand-
  written), the apply path fails explicitly rather than silently
  misclassifying.

- Two new in-source tests:
    - `drop_steps_round_trip_through_serde` — pins the wire shape
      for all four (variant × mode) combinations.
    - `drop_mode_serde_uses_snake_case` — pins external-tool-
      friendly serialization (`"soft"` / `"hard"`).

Build: clean, only pre-existing warnings.
Tests:
- omnigraph-compiler schema_plan: 6/6 (4 existing + 2 new).
- omnigraph-engine schema_apply: 11/11 (unchanged — planner still
  emits UnsupportedChange for removal paths).

Next commit (commit 3 per docs/schema-lint-v1-plan.md): add the
`tombstoned: bool` fields to NodeIR / EdgeIR / PropertyIR for the
catalog representation of soft-mode tombstones.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* plan doc: reframe v1 around Lance native drop_columns

After a substrate audit of the Lance data-evolution guide on
2026-05-13, the v1 plan was simplified. Two key findings:

1. Lance's `drop_columns()` is already metadata-only and reversible
   via time travel until cleanup. No need for a parallel
   `tombstoned: bool` field in our catalog IR — Lance's version
   graph IS the tombstone.

2. The full schema_apply substrate migration (add_columns,
   drop_columns, alter_columns vs. stage_overwrite across all step
   types) is consolidated in MR-948 as a sibling issue. v1 only
   uses the relevant slice (drop_columns for OG-DS-1XX).

Net plan changes:

- Commit 3 (original): tombstone fields on catalog IR → dropped.
  No catalog IR change needed. The Lance drop_columns commit IS the
  tombstone.

- Commit 5 (original): apply path writes tombstoned: true → replaced
  with: apply path calls Dataset::drop_columns([name]).

- Commit 7 Hard mode: stage_overwrite removing the column → replaced
  with: drop_columns + compact_files + cleanup_old_versions. Same
  APIs omnigraph cleanup already uses.

- Commit 8 (original): omnigraph schema unhide → dropped. Time
  travel is the undo (omnigraph snapshot --at <commit>).

Net result: 8 commits → 5 commits. ~250 LoC less surface. More
substrate-aligned.

The chassis types from commit 2 (DropMode enum, DropType /
DropProperty variants) are kept exactly as designed; only the
implementation strategy changed.

The Lance docs quote is included in the doc so future readers see
the substrate behavior cited verbatim.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* schema-lint v1 commit 3: emit + apply DropProperty { Soft }

Wire the dormant DropProperty variant end-to-end for the Soft case.
Per docs/schema-lint-v1-plan.md, commit #3 of the schema-lint chassis
v1 series (MR-694).

Planner (schema_plan.rs):
- plan_properties: emit DropProperty { type_kind, type_name,
  property_name, mode: Soft } instead of UnsupportedChange when a
  property exists in accepted but not in desired. Plan is now
  supported = true for drop-only changes.

Apply (schema_apply.rs):
- Route DropProperty { Soft } through rewritten_tables. The existing
  batch_for_schema_apply_rewrite path already iterates the *target*
  schema fields, so a property absent from desired_catalog is
  naturally projected away. The prior Lance version retains the
  dropped column for time-travel reversibility (until cleanup runs).
- DropType still errors (lands in commit #4 with different mechanics:
  __manifest entry removal instead of column projection).
- DropProperty { Hard } still errors (lands in commit #5 with
  --allow-data-loss CLI flag + immediate compact_files +
  cleanup_old_versions).

Tests:
- Planner unit test plan_emits_soft_drop_for_removed_nullable_property
  asserts the variant emission + supported = true + no UnsupportedChange.
- Integration test apply_schema_drops_a_nullable_property_softly_
  preserves_prior_version (replaces the former
  apply_schema_rejects_dropping_a_property_with_data) asserts:
  (a) plan contains DropProperty { Soft }
  (b) apply succeeds + manifest advances + row count unchanged
  (c) current dataset schema lacks the dropped column
  (d) snapshot_at_version(pre_drop) still has the dropped column
  (e) reopen consistency — drop preserved across engine restart

Recovery: rides on SidecarKind::SchemaApply per MR-847. No new
sidecar kind needed; the entire apply path is already sidecar-wrapped.

Substrate alignment: this commit uses the stage_overwrite full-rewrite
path (full_rewrite cost class) rather than Lance native drop_columns
(catalog_only cost class). MR-948 is the follow-up substrate-alignment
refactor that introduces a LanceColumnOp surface and switches the
metadata-only case onto drop_columns. Functional outcome is identical;
cost-class improvement deferred.

Test results:
- cargo test -p omnigraph-compiler --lib: 238 passed
- cargo test -p omnigraph-engine --test schema_apply: 11 passed

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs: move schema-lint-v1-plan into docs/dev/ + add to index

Post-rebase fixup for the docs split (#93). The plan doc was added
to docs/ at the top level before main reorganized to docs/{user,dev}/.
This moves it into docs/dev/ and adds an entry to docs/dev/index.md
under a new "Active Implementation Plans" section so the
check-agents-md.sh link check passes.

Per the original commit message (617a77d), the plan doc is intentionally
temporary — it will be deleted when v1 lands.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

crates/omnigraph/src/db/omnigraph.rs

@@ -18,8 +18,8 @@ use omnigraph_compiler::catalog::{Catalog, EdgeType, NodeType};
 use omnigraph_compiler::schema::parser::parse_schema;
 use omnigraph_compiler::types::ScalarType;
 use omnigraph_compiler::{
-    SchemaIR, SchemaMigrationPlan, SchemaMigrationStep, SchemaTypeKind, build_catalog_from_ir,
-    build_schema_ir, plan_schema_migration,
+    DropMode, SchemaIR, SchemaMigrationPlan, SchemaMigrationStep, SchemaTypeKind,
+    build_catalog_from_ir, build_schema_ir, plan_schema_migration,
 };
 
 use crate::db::graph_coordinator::{GraphCoordinator, PublishedSnapshot};

crates/omnigraph/src/db/omnigraph/schema_apply.rs

@@ -138,6 +138,41 @@ pub(super) async fn apply_schema_with_lock(
             }
             SchemaMigrationStep::UpdateTypeMetadata { .. }
             | SchemaMigrationStep::UpdatePropertyMetadata { .. } => {}
+            SchemaMigrationStep::DropProperty {
+                type_kind,
+                type_name,
+                mode,
+                ..
+            } => {
+                // Soft = reuse the existing stage_overwrite rewrite
+                // path. batch_for_schema_apply_rewrite iterates the
+                // *target* schema fields, so a property absent from
+                // desired_catalog is naturally projected away. The
+                // prior Lance version retains the dropped column,
+                // so reads at the previous snapshot still see it
+                // (time-travel reversibility). Hard mode (immediate
+                // compact_files + cleanup_old_versions for actual
+                // data deletion) lands in commit #5 gated by
+                // --allow-data-loss.
+                if !matches!(mode, DropMode::Soft) {
+                    return Err(OmniError::manifest_internal(
+                        "DropProperty { Hard } not yet implemented (commit #5)",
+                    ));
+                }
+                let table_key = schema_table_key(*type_kind, type_name);
+                if table_key.starts_with("edge:") {
+                    changed_edge_tables = true;
+                }
+                rewritten_tables.insert(table_key);
+            }
+            SchemaMigrationStep::DropType { .. } => {
+                // DropType (whole-table drop via __manifest entry
+                // removal) lands in commit #4 — different mechanics
+                // from DropProperty.
+                return Err(OmniError::manifest_internal(
+                    "DropType not yet implemented (commit #4)",
+                ));
+            }
             step @ SchemaMigrationStep::UnsupportedChange { .. } => {
                 return Err(OmniError::manifest(
                     step.unsupported_error_message()

crates/omnigraph/tests/schema_apply.rs

@@ -101,17 +101,23 @@ async fn apply_schema_unsupported_plan_does_not_advance_manifest() {
     );
 }
 
-// ─── Destructive / safety-tier rejections ────────────────────────────────────
+// ─── Destructive / safety-tier behavior ──────────────────────────────────────
 //
-// Schema migration v1 only accepts additive change: add type, add nullable
-// property, add index, rename. Every other shape returns an
-// `UnsupportedChange` step that surfaces as an error from `apply_schema`,
-// without advancing the manifest. These tests pin that contract for the
-// destructive shapes (drop type, drop property, narrow type, add required,
-// remove constraint) so a regression in the planner can't silently allow them.
+// Schema migration v1 accepts:
+// - Additive change: add type, add nullable property, add index, rename.
+// - DropProperty { Soft } via the schema-lint v1 chassis (commit #3 of MR-694)
+//   — the dropped column is removed from the current manifest version but
+//   remains reachable via Lance time travel at the prior version, until
+//   `omnigraph cleanup` runs. Hard mode (immediate data cleanup) lands in
+//   commit #5 gated by `--allow-data-loss`.
+//
+// Every other destructive shape (drop type, narrow type, add required without
+// backfill, remove constraint) still returns an `UnsupportedChange` step that
+// surfaces as an error from `apply_schema`. These tests pin the current
+// contract so a regression in the planner can't silently change behavior.
 
 #[tokio::test]
-async fn apply_schema_rejects_dropping_a_property_with_data() {
+async fn apply_schema_drops_a_nullable_property_softly_preserves_prior_version() {
     let dir = tempfile::tempdir().unwrap();
     let mut db = init_and_load(&dir).await;
 
@@ -122,25 +128,96 @@ async fn apply_schema_rejects_dropping_a_property_with_data() {
         .unwrap()
         .version();
 
-    // Drop `age` from Person. v1 doesn't support property removal even when
-    // the column is nullable — it would silently destroy data.
+    // Drop `age` from Person. v1 + chassis commit #3 emit
+    // `DropProperty { Soft }`; the rewrite path projects to the
+    // target schema (no `age`), commits via stage_overwrite. Row
+    // counts are unchanged — only the column is dropped from the
+    // current schema view.
     let desired = TEST_SCHEMA.replace("    age: I32?\n", "");
-    let err = db.apply_schema(&desired).await.unwrap_err();
-    let msg = err.to_string();
+
+    // Confirm the plan emits DropProperty { Soft } (not UnsupportedChange).
+    let plan = db.plan_schema(&desired).await.unwrap();
+    assert!(plan.supported, "drop-property plan must be supported");
     assert!(
-        msg.contains("OG-DS-104"),
-        "expected schema-lint code OG-DS-104 in error, got: {msg}"
+        plan.steps.iter().any(|step| matches!(
+            step,
+            SchemaMigrationStep::DropProperty {
+                type_kind: SchemaTypeKind::Node,
+                type_name,
+                property_name,
+                mode: omnigraph_compiler::DropMode::Soft,
+                ..
+            } if type_name == "Person" && property_name == "age"
+        )),
+        "expected DropProperty {{ type=Person, property=age, mode=Soft }} in plan; got {plan:?}",
     );
 
-    // Manifest didn't advance and existing rows are untouched.
-    assert_eq!(
-        db.snapshot_of(ReadTarget::branch("main"))
-            .await
-            .unwrap()
-            .version(),
-        before_version
+    let result = db.apply_schema(&desired).await.unwrap();
+    assert!(result.supported);
+    assert!(result.applied);
+
+    // Manifest advanced; row count unchanged.
+    let after_version = db
+        .snapshot_of(ReadTarget::branch("main"))
+        .await
+        .unwrap()
+        .version();
+    assert!(
+        after_version > before_version,
+        "manifest version should advance after soft drop; before={before_version}, after={after_version}",
     );
     assert_eq!(count_rows(&db, "node:Person").await, people_before);
+
+    // (a) Current snapshot: `age` is gone from the dataset schema.
+    let current_snapshot = db.snapshot_of(ReadTarget::branch("main")).await.unwrap();
+    let current_ds = current_snapshot.open("node:Person").await.unwrap();
+    let current_fields = current_ds
+        .schema()
+        .fields
+        .iter()
+        .map(|f| f.name.clone())
+        .collect::<Vec<_>>();
+    assert!(
+        !current_fields.iter().any(|f| f == "age"),
+        "current Person dataset schema must not include 'age' after soft drop; got fields {current_fields:?}",
+    );
+
+    // (b) Time travel: at the pre-drop manifest version, the prior
+    // Person dataset version still has `age`. Soft drop is reversible
+    // via Lance's version graph until `omnigraph cleanup` runs.
+    let pre_drop_snapshot = db.snapshot_at_version(before_version).await.unwrap();
+    let pre_drop_ds = pre_drop_snapshot.open("node:Person").await.unwrap();
+    let pre_drop_fields = pre_drop_ds
+        .schema()
+        .fields
+        .iter()
+        .map(|f| f.name.clone())
+        .collect::<Vec<_>>();
+    assert!(
+        pre_drop_fields.iter().any(|f| f == "age"),
+        "pre-drop Person dataset schema must still include 'age' (time-travel reversibility); got fields {pre_drop_fields:?}",
+    );
+
+    // (c) Reopen consistency: close the engine, reopen, verify the
+    // drop is preserved (column still absent from current schema).
+    let uri = dir.path().to_str().unwrap().to_string();
+    drop(db);
+    let reopened = Omnigraph::open(&uri).await.unwrap();
+    let reopened_snapshot = reopened
+        .snapshot_of(ReadTarget::branch("main"))
+        .await
+        .unwrap();
+    let reopened_ds = reopened_snapshot.open("node:Person").await.unwrap();
+    let reopened_fields = reopened_ds
+        .schema()
+        .fields
+        .iter()
+        .map(|f| f.name.clone())
+        .collect::<Vec<_>>();
+    assert!(
+        !reopened_fields.iter().any(|f| f == "age"),
+        "after reopen, Person dataset schema must still lack 'age'; got fields {reopened_fields:?}",
+    );
 }
 
 #[tokio::test]