SurfSense/plans/backend/revamp phases 4-7/05b-intelligence.md

Phase 5b — Intelligence (the decision-grounded engine)

Part of Phase 5 — Intelligence & Timeline. Sibling: 05a-timeline.md (the state it writes). Build after 05a (the tables) and 04a (the verbs it calls). Together, 05a + 05b ship the Product B engine (the Tracker, locked schema, hot loop, deltas). Scope guardrail: Phases 1–3 SHIPPED/FIXED. This is net-new and is not the KB and not the automations subsystem. Name: the standing-concern primitive is the Tracker.

Objective

Turn repeated capability calls into decision-relevant structured signal. The motto: the agent judges, code computes. This replaces the old "pipeline" as the standing concern: Intelligence is the process that mutates state; Timeline (05a) is the state. Everything below the Access→Intelligence boundary stays pure functions.

STATELESS (Product A):  04a Capabilities + 04b Access   → call → data → bill, nothing persists
STATEFUL  (Product B):  05b Intelligence + 05a Timeline  → the Timeline IS the state

Current state (cited)

• Capability executors (04a) — called directly by the loop (not through a door).
• Content-hash pre-check — WebCrawlerConnector.format_to_structured_document(exclude_metadata=True) produces the stable text the loop hashes against entity_current_state.content_hash (05a).
• Run/audit substrate to reuse — the existing AutomationRun (status/error/timing/step_results; note the executor writes results to step_results[n].result, not run.output/artifacts, which stay unwritten) + the automations executor's PENDING→running gate (execute_run() early-returns unless status == PENDING; acks_late is global). Caveat: the gate is not atomic (no UPDATE … WHERE status='pending'), so it stops terminal/post-mark_running redelivery but not concurrent double-claim — pair it with the per-Tracker lock (06). Plus the chat job record (04a) + deliverable_wait. No new run table.
• CI context uploads — the existing folder-upload machinery (POST /documents/folder-upload, param root_folder_id — there is no destination_folder) + KB retrieval scoped via mention-pins → referenced_document_ids(folder_ids=…) → SearchScope.document_ids (no direct folder_id on search).

Target design

The primitive — Tracker

A saved, decision-grounded subject that accumulates structured signal over time:

Field	Meaning
decision	the question being tracked toward ("is this competitor pulling ahead?")
capability_binding	which verb + input feeds it (maps.place(X), web.scrape([Y]))
definition (locked, versioned)	{ field_schema, identity_rule, materiality } — the agent-drafted, human-locked contract
status	draft → locked/active

One entity per Tracker for MVP (one place / one URL). Multi-entity (maps.search → many) is deferred (the 05a model stays multi-entity-ready so it's additive).

Setup (once) — the agent-designed schema flow (IN MVP)

The product must not be rigid: we cannot author one schema that serves everyone, so the schema is derived from the user's decision by an agent and locked by the human. Conversationally (chat-first):

1. Bind a capability + input.
2. Sample fetch — one real capability call so the agent drafts against actual returned data.
3. Agent drafts the definition from decision + the sample: field_schema (fields + types), materiality (numeric thresholds where possible; "ask agent" otherwise), identity_rule (stable entity key — Maps place_id, canonical URL), and a reserved notable_signals escape-hatch field.
4. Human reviews & locks (in chat: "looks good" / "add field X"). Locked ⇒ stable run-to-run.
5. Versioned — a locked definition is a snapshot; edits create a new version (mirrors how automations snapshots definition).

The hot loop (per refresh) — refresh(tracker)

1. Crawl — call the bound capability (04a) → raw data.
2. Cheap pre-check — content hash; identical to the stored content_hash → stamp last_checked_at, stop (no LLM cost).
3. Fill — agent conforms raw data to the locked field_schema via structured output; it does not invent fields. Unanticipated observations go into notable_signals.
4. Diff (code) — deterministic compare of the new record vs Current state (05a) → raw deltas.
5. Judge — the materiality split:
   • deterministic (code): numeric/clear-cut rules from materiality, applied for free, 100% reproducible (e.g. rating Δ≥0.2 → material, review_count Δ≥10 → material, 1¢ wobble → noise).
   • agent (only on ambiguous): anything a rule can't decide — reworded description, a new notable_signals entry → one LLM call rules material/noise.
6. Append — if material: write a Change + update Current state (05a). Else: only last_checked_at. No change → no row.

Worked example (maps.place refresh):

rating 4.4 → 4.3 (Δ0.1)    → code: < 0.2 → NOISE        (no LLM)
review_count 312 → 470     → code: ≥ 10 → MATERIAL      (no LLM)
hours unchanged            → no delta
description reworded        → code: no rule → ASK AGENT → NOISE
⇒ one Change row (review spike); one cheap LLM call; zero LLM on the rating tick.

Refresh execution & idempotency — ride the invoking surface (no new run table)

refresh(tracker) is a headless unit of work; the run/audit record + idempotency live on whatever surface invoked it:

• Recurring (in-app): invoked by the CI automation action (06) → the existing AutomationRun is the run record and the automations executor's PENDING-gate handles redelivery. Reused, not re-written — but the gate is not atomic against concurrent claims, so the per-Tracker lock (06) is the actual double-refresh guard.
• Chat (manual / agent): invoked via the chat job record (04a) + deliverable_wait.
• Billing idempotency is per capability call, not per run: each executor bills a success once via the billing service (04a); the content-hash pre-check (step 2) prevents needless re-crawls/charges. No run-level charged_micros ledger required for MVP.

Net: the only genuinely new state is the Timeline (05a); execution accounting is borrowed.

User-supplied context files (the F idea, generalized)

When a user uploads a file in a CI chat (e.g. "our own price list"), it goes into the KB as normal — uploads create Documents and are indexed/embedded, exactly as today. (The "don't index" rule applies only to crawled data.) The CI-specific part is organization + use:

• Routed to a dedicated folder for that CI chat/Tracker (reuse POST /documents/folder-upload with root_folder_id; uploads land as Documents with folder_id).
• The judge step (5) may consult them — retrieved from the KB, scoped to that folder via the mention-pin path (referenced_document_ids(folder_ids=…) → SearchScope.document_ids), since search has no direct folder_id filter:

competitor price 12.00 → 9.90   + user's context file says "our price is 10.00"
   → agent: competitor crossed *below our price* → MATERIAL (and explain why)

The user's private context shapes what counts as material. Reuses existing KB upload + folder + retrieval machinery (nothing new). MVP-optional (the loop works without it); design the seam now.

Where it lives / decoupling

• New Apache-2 package app/intelligence/ (schema-design agent, hot loop, materiality evaluator). Calls capability executors directly.
• Exposes refresh(tracker). Who calls it (manual / agent / external cron / optional automation) is the Triggers domain's concern (06) — Intelligence has no dependency on any scheduler.

Work items

1. Tracker model + persistence: decision · capability_binding · versioned locked definition · status.
2. Schema-design flow: bind → sample-fetch → agent-drafts definition → human review/lock → version.
3. Materiality evaluator: deterministic rule engine (numeric/clear) + the agent-on-ambiguous fallback.
4. The hot loop refresh(tracker): crawl → hash pre-check → fill → diff → judge → append (writes 05a).
5. Idempotency wiring: ride AutomationRun (recurring) / chat job record (manual) — no new run table.
6. Context-folder seam: optional KB-retrieval hook into the judge, scoped to the Tracker's folder.

Tests

• Pre-check short-circuit: identical content hash → no fill, no LLM, only last_checked_at bumped.
• Fill conforms to lock: extra observed fields land in notable_signals, never invent schema fields.
• Materiality split: numeric Δ over threshold → decided_by=code, material; ambiguous reword → decided_by=agent; sub-threshold → noise, no row.
• Append semantics: a material run writes one entity_changes row + overwrites current state.
• Idempotency: a redelivered recurring refresh (same AutomationRun) does not double-write or double-bill.
• Context folder (optional): judge ruling flips when a user context file changes the decision frame.

Risks / trade-offs

• Refresh failure path (capability returns FAILED/partial): skip vs retain vs retry vs alert — an implementation-time call (no architecture impact).
• Agent fill cost on changed pages: bounded by the hash pre-check; only changed content reaches the LLM.
• Schema lock rigidity: locking trades flexibility for run-to-run stability; re-lock creates a new version.

Resolved decisions

1. Tracker is the standing-concern primitive; replaces "pipeline".
2. Stateless (04/Product A) vs stateful (05/Product B) is the Access→Intelligence boundary.
3. Agent-designed schema flow is in MVP (not hand-authored) — sample-grounded, human-locked, versioned.
4. Single entity per Tracker for MVP.
5. Materiality = deterministic numeric/clear rules in code + agent only on ambiguous.
6. Content-hash pre-check short-circuits unchanged pages before any LLM spend.
7. app/intelligence/ Apache-2; refresh(tracker) is trigger-agnostic.
8. No new run table — refresh audit rides AutomationRun (recurring) or the chat job record; redelivery is stopped by the PENDING-gate plus a per-Tracker lock (the gate is non-atomic); billing idempotency is per-capability-call + the content-hash gate. Only Timeline (05a) is new state.
9. CI context folder (F): user files uploaded in a CI chat go into the KB as normal (indexed), routed to a dedicated Folder, and may feed the judge via KB retrieval. "Don't index" is for crawled data only. MVP-optional seam.

Out of scope (hand-offs)

• The state tables (the three stores, content-hash, read API) → 05a.
• When refresh fires + recurrence/delivery → 06 (Triggers).
• The human-facing crafting/answering experience (the intelligence_agent subagent + prompt) → 07.
• Schema auto-evolution, multi-entity Trackers, backward-replay, coverage-confidence, the resale/data-product stage → north star (deferred).

Open questions (carry forward)

• How the schema-design agent surfaces "review & lock" before the frontend exists (pure-chat confirmation?).
• Versioning policy on re-lock (new version vs in-place) — lean new version.
• Where the schema-design agent itself runs (a setup capability? a chat sub-flow?).
• Context-folder → judge wiring (how much to load; per-Tracker vs per-chat scope).