test(cli): address review — assert schema-show success, document exit-code stance, add e2e opt-out

- The drift-heal verification now asserts `schema show` succeeded and
  produced a schema before checking the rogue field's absence (a failed
  command previously made the negative assertion vacuously pass).
- cluster_cli documents why it deliberately does not assert exit codes
  (blocked applies exit non-zero by contract while emitting the structured
  output callers assert on).
- The comprehensive lifecycle e2es honor OMNIGRAPH_SKIP_SYSTEM_E2E=1
  (graceful skip-with-message, the S3-gate pattern) for constrained
  sandboxes; requirements + suppression documented in testing.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
aaltshuler 2026-06-10 19:05:12 +03:00
parent 7d70811df1
commit d8354ac213
2 changed files with 41 additions and 2 deletions

View file

@ -1717,6 +1717,13 @@ graphs:
// ---- Comprehensive full-cycle cluster e2e (Phases 1-5 composed) ----
/// Run a `cluster` subcommand and return its JSON output. Deliberately does
/// NOT assert a zero exit code: blocked/unconverged runs (e.g. an `apply`
/// awaiting an approval) exit non-zero by contract while still emitting the
/// structured output the caller asserts on (`ok`/`converged`/dispositions).
/// Commands where failure is never expected must assert on those fields
/// (every call here checks `ok` or `converged`) or use `cli()` directly with
/// `status.success()`.
fn cluster_cli(dir: &std::path::Path, args: &[&str]) -> serde_json::Value {
let mut command = cli();
command.arg("cluster");
@ -1813,12 +1820,30 @@ fn invoke_query(
(status, body)
}
/// Opt-out for the comprehensive system e2es below. They need no external
/// services — only the workspace-built `omnigraph`/`omnigraph-server`
/// binaries (cargo provides them via `CARGO_BIN_EXE_*`), ephemeral localhost
/// ports, and local-FS temp dirs — but they spawn real server processes and
/// run multi-stage lifecycles, so constrained sandboxes can suppress them:
/// `OMNIGRAPH_SKIP_SYSTEM_E2E=1 cargo test ...` (same skip-with-message
/// pattern as the S3 tests' `OMNIGRAPH_S3_TEST_BUCKET` gate).
fn skip_system_e2e(test_name: &str) -> bool {
if std::env::var("OMNIGRAPH_SKIP_SYSTEM_E2E").is_ok_and(|v| !v.is_empty() && v != "0") {
eprintln!("skipping {test_name}: OMNIGRAPH_SKIP_SYSTEM_E2E is set");
return true;
}
false
}
/// The whole control-plane story in one test: declare two graphs → converge
/// (apply creates them) → serve → evolve schema+query in one apply → restart
/// serves the new shape → out-of-band drift converged back → approved graph
/// delete → restart serves the survivor only → plan empty.
#[test]
fn local_cluster_full_lifecycle_declare_serve_evolve_delete() {
if skip_system_e2e("local_cluster_full_lifecycle_declare_serve_evolve_delete") {
return;
}
let temp = tempfile::tempdir().unwrap();
let dir = temp.path();
write_two_graph_cluster(dir);
@ -1931,8 +1956,15 @@ fn local_cluster_full_lifecycle_declare_serve_evolve_delete() {
.output()
.unwrap();
assert!(
!String::from_utf8_lossy(&schema_show.stdout).contains("rogue"),
"drift must be soft-dropped back to the declared schema"
schema_show.status.success(),
"schema show failed: {}",
String::from_utf8_lossy(&schema_show.stderr)
);
let shown = String::from_utf8_lossy(&schema_show.stdout);
assert!(shown.contains("Person"), "schema show produced no schema: {shown}");
assert!(
!shown.contains("rogue"),
"drift must be soft-dropped back to the declared schema: {shown}"
);
// Retire engineering: gated delete, then the server serves the survivor.
@ -2005,6 +2037,9 @@ graphs:
/// owns query invocation — enforced over HTTP with bearer-resolved actors.
#[test]
fn local_cluster_serving_enforces_applied_policy_bindings() {
if skip_system_e2e("local_cluster_serving_enforces_applied_policy_bindings") {
return;
}
let temp = tempfile::tempdir().unwrap();
let dir = temp.path();
std::fs::write(

View file

@ -69,6 +69,10 @@ CI runs three S3-backed tests against a containerized RustFS server (`.github/wo
Locally, set `OMNIGRAPH_S3_TEST_BUCKET` (and the usual `AWS_*` vars including `AWS_ENDPOINT_URL_S3` for non-AWS) before running. Without those, S3 tests skip gracefully.
## System e2e requirements and suppression
The CLI system tests (`system_local.rs`) spawn the workspace-built `omnigraph` and `omnigraph-server` binaries (cargo provides paths via `CARGO_BIN_EXE_*`), bind ephemeral localhost ports, and use local-FS temp dirs — no external services, no env vars required; they run in the default `cargo test --workspace`. The comprehensive cluster lifecycle e2es (multi-server-restart flows) honor an opt-out for constrained sandboxes: set `OMNIGRAPH_SKIP_SYSTEM_E2E=1` to skip them with a logged message (the same graceful-skip pattern as the S3 gate). Cargo-native filtering also works: `cargo test --test system_local -- --skip local_cluster`.
## OpenAPI drift
`crates/omnigraph-server/tests/openapi.rs` regenerates `openapi.json` and diffs against the checked-in copy. CI auto-commits the regeneration on same-repository PRs and otherwise runs in strict-check mode (env: `OMNIGRAPH_UPDATE_OPENAPI`).