omnigraph/research/lance-autoresearch/docs/adding-a-target.md

Adding a new target

Walk through this when spinning up a new optimization target (A1 cosine, A4 bitpack, etc.). It's a cp -r plus surgical edits — no architectural decisions to make per target if the kernel fits the autoresearch shape.

If your target's per-trial eval is more than ~30 seconds, or the correctness oracle can't be a deterministic comparison against a scalar reference, this harness is the wrong fit — see design.md "When to revisit" for the boundary.

Steps

1. Pick a template target

The closest existing target. For now there's just pq-l2, but as more land:

• Distance / scoring kernels that take a query and return per-row scores → template off pq-l2.
• Decode kernels that take encoded bytes and return an Arrow array → template off bitpack once it lands.
• Scan / merge kernels → template off topk-merge once it lands.

cp -r crates/pq-l2 crates/<my-target>

2. Rewrite Cargo.toml

[package]
name = "<my-target>"
# version, edition, license, publish stay the same

Add the target to the workspace members in the root Cargo.toml:

[workspace]
members = [
    "crates/harness-common",
    "crates/pq-l2",
    "crates/<my-target>",   # add this
]

3. Rewrite src/lib.rs

Define the target's Shape type (analogue of PqShape) and any other types shared between kernels.rs and reference.rs and inputs.rs. Document which fields are pinned by the harness vs. agent-tunable.

This file is immutable to the agent. The shape parameters define the optimization target — changing them changes what's being optimized.

4. Rewrite src/reference.rs

Implement the scalar reference kernel — the math, in plain Rust, no SIMD, no cleverness. This is what the agent's kernel is compared against. Mirror the public API of kernels.rs exactly.

For float kernels, also export max_abs_err(a, b) and topk_consistent(...) (or analogues) — the comparison helpers the bench uses to assert near-bit-exact equivalence with harness_common::MAX_ABS_ERR / TOPK_DIST_TOL.

For integer / byte kernels, the comparison is simpler — assert_eq! on the returned Arrow array. No tolerance constants needed.

5. Rewrite src/inputs.rs

Two surfaces:

• correctness_battery(seed) -> Vec<CorrectnessCase> — diverse shape × distribution combinations, sized small enough that the correctness phase finishes in seconds. The point is breadth, not realism.
• speed_workloads(seed) -> Vec<SpeedWorkload> — larger shape × distribution combinations sized for stable timings. Aim for total trial wall-clock ≤ 60s; the agent's iteration latency dominates correctness elsewhere.

Use harness_common::SplitMix64 for determinism. Same seed → same battery across trials.

6. Rewrite src/kernels.rs (the agent's playground)

Implement a clean scalar baseline matching the algorithm shape of the Lance upstream code. The header comment must:

• Cite the upstream Lance source (lance-format/lance rev / file path) the algorithm is modeled on.
• Document the public API the bench calls — these are the surfaces the agent may NOT change.
• List "what you can do" / "what you cannot do" rules specific to this target.

The starting kernel must be correct (passes the correctness phase against reference.rs) and lint-clean. The agent's job is to make it faster.

7. Rewrite src/bin/run_experiment.rs

Two phases:

• Correctness phase: for each CorrectnessCase, run agent kernel + reference, compare. Any mismatch → print correctness: fail, diagnostic line, exit 2.
• Speed phase: for each SpeedWorkload, run agent kernel and time per query / per row / per byte. Aggregate geomean / worst / best across all combos. Print fixed-format result block.

Universal output fields (every target) are listed in HARNESS.md "The metric." Add per-target fields above them as needed (e.g., bit_widths_tested for bitpack).

Use:

• harness_common::geomean for the aggregator
• harness_common::peak_rss_mb for memory readback
• harness_common::TIME_BUDGET_SECS for the time-budget check

8. (Optional) Rewrite benches/<my-target>.rs

Criterion benchmark with the same kernel calls as run_experiment but under criterion's statistical-sampling harness. Optional — the per-trial binary is the agent's primary measurement; criterion is for the human's deeper investigation.

9. Write program.md

Per-target agent skill, layered on top of HARNESS.md. Sections:

• Setup — which files to read at session start (always include ../../HARNESS.md).
• Public API contract — the exact functions / structs the agent must keep stable.
• Target-specific priors — known SIMD techniques for this kernel shape, algorithmic transformations worth trying, common pitfalls. This is the highest-leverage content; spend time on it.
• results.tsv header — the per-target column set.

10. Write the per-target capsule in docs/targets/<my-target>.md

A short doc covering:

• What's optimized (one sentence)
• Upstream Lance source pointers (rev, file paths, function names)
• Oracle definition (bit-exact / max_abs_err)
• Speed workload shape (what shapes × distributions span)
• Status (candidate / landed / has-results)

11. Verify end-to-end

cargo build --release -p <my-target>
cargo clippy --release -p <my-target> --all-targets -- -D warnings
cargo run --release --bin run_experiment -p <my-target>

The baseline trial must:

• Print correctness: pass
• Exit 0
• Finish within ~60s
• Reference a sensible geomean_ns_per_* baseline number

Smoke-test the gate: deliberately break kernels.rs (e.g., return constant zero), confirm the trial exits 2 with correctness: fail. Restore.

12. Add the target row to the top-level README.md

In the targets table at the top of the README, change the new target's row from candidate to landed.

13. Commit

One commit for the target's scaffolding. Don't bundle multiple targets in one commit — each target's history should be independently revertible.

Common gotchas

• Forgetting the empty [workspace] block at the root means cargo walks up to the omnigraph parent workspace. Already handled; just don't remove it.
• Per-target Cargo.toml referencing the wrong harness-common path. Use harness-common = { path = "../harness-common" }.
• Picking a SHAPES set that's too small. Three shapes is the floor; with one shape an agent could specialize and pass, with two there's not enough variety. Ensure the shapes span at least one "outlier" (e.g., for PQ, one shape with sub_vector_dim != 8).
• Correctness battery too narrow. Five distributions is the floor: at minimum Gaussian / uniform / sparse / large-dynamic-range / mostly-zero (or the integer analogue: uniform / clustered / skewed / few-distinct / monotonic).
• Trial time too long. If the speed phase exceeds ~60s, agent iteration rate drops below useful. Reduce workload sizes; the speed metric is per-operation, not per-workload, so absolute size doesn't change the comparison.