From dbde85b68d07bb527a3e75a15161790e95d78998 Mon Sep 17 00:00:00 2001
From: andrew <andrew@collectivelab.io>
Date: Sun, 19 Apr 2026 22:27:49 +0300
Subject: [PATCH] Raise LANCE_MEM_POOL_SIZE to 1 GB in .cargo/config.toml
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Fixes flaky omnigraph-server integration suite under parallel cargo
test. Lance defaults to a 100 MB FairSpillPool per Omnigraph instance
(lance-datafusion/src/exec.rs:316). That's fine in prod (one server
process, bounded concurrent sorts) but too small when cargo test
spawns many Omnigraph instances in parallel, each running concurrent
BTree index builds during load.

Failure signature:
  Lance("create BTree index on node:Person(id): ... LanceError(IO):
  Not enough memory to continue external sort. ... 0.0 B remain
  available for the total pool")

Before: 10/41 OOM-fail on parallel run; passed with --test-threads=1.
After:  41/41 pass in parallel in ~3s.

[env] in .cargo/config.toml applies to cargo-launched processes only.
Shipped binaries (release tarballs, Docker images) are unaffected —
they inherit whatever the runtime env provides, defaulting to Lance's
100 MB when unset.
---
 .cargo/config.toml | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
 create mode 100644 .cargo/config.toml

diff --git a/.cargo/config.toml b/.cargo/config.toml
new file mode 100644
index 0000000..d169a4a
--- /dev/null
+++ b/.cargo/config.toml
@@ -0,0 +1,22 @@
+# Cargo config for the omnigraph workspace.
+#
+# `[env]` entries here apply to anything cargo launches in this repo —
+# `cargo build`, `cargo test`, `cargo run`. Shipped binaries are NOT
+# affected: the env var lives only in the cargo-launched process, not in
+# release artifacts or Docker images.
+
+[env]
+# Lance's DataFusion memory pool per `Omnigraph` instance. Lance defaults
+# to 100 MB (`lance-datafusion/src/exec.rs`), which is fine for a single
+# server process in production.
+#
+# Under `cargo test` it's a problem: the omnigraph-server integration
+# suite spins up many `Omnigraph` instances in parallel (one per test,
+# default `num_cpus` threads), each with concurrent BTree index builds
+# sharing that test's 100 MB pool — saturates quickly and surfaces as
+# `Not enough memory to continue external sort` / `0.0 B remain
+# available for the total pool`.
+#
+# 1 GB is comfortable headroom per-test pool; prod and CI-for-prod
+# builds set the variable themselves (or keep the 100 MB default).
+LANCE_MEM_POOL_SIZE = "1073741824"