mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-06 22:12:12 +02:00
docs(revamp): reshape Intelligence into Phase 5b
This commit is contained in:
CREDO23
parent
3b92e40bf6
commit
5c2428541d
1 changed files with 115 additions and 95 deletions
|
|
@ -1,26 +1,37 @@
|
|||
# Domain ③ — Intelligence (the decision-grounded engine) (CI pivot revamp · WIP)
|
||||
# Phase 5b — Intelligence (the decision-grounded engine)
|
||||
|
||||
> **WIP design doc.** Part of the Phase 4 → end revamp. Pairs with `04-timeline.md` (the state it writes).
|
||||
> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This is net-new and is **not** the KB and **not**
|
||||
> the automations subsystem.
|
||||
> Part of **Phase 5 — Intelligence & Timeline**. Sibling: `05a-timeline.md` (the state it writes).
|
||||
> **Build after** `05a` (the tables) and `04a` (the verbs it calls). Together, `05a + 05b` ship the
|
||||
> **Product B engine** (the Tracker, locked schema, hot loop, deltas).
|
||||
> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This is net-new and is **not** the KB and **not** the
|
||||
> automations subsystem.
|
||||
> **Name:** the standing-concern primitive is the **Tracker**.
|
||||
|
||||
## The stateless / stateful line
|
||||
|
||||
```
|
||||
STATELESS (Product A): ① Capabilities + ② Access → call → data → bill, nothing persists
|
||||
STATEFUL (Product B): ③ Intelligence + ④ Timeline → the Timeline IS the state
|
||||
```
|
||||
|
||||
Intelligence is the **process** that mutates state; Timeline (`04`) is the **state**. Everything
|
||||
below the Access→Intelligence boundary stays pure functions.
|
||||
|
||||
## Purpose
|
||||
## Objective
|
||||
|
||||
Turn repeated capability calls into **decision-relevant structured signal**. The motto: **the agent
|
||||
judges, code computes.** This replaces the old "pipeline" as the *standing concern*.
|
||||
judges, code computes.** This replaces the old "pipeline" as the *standing concern*: Intelligence is the
|
||||
**process** that mutates state; Timeline (`05a`) is the **state**. Everything below the
|
||||
Access→Intelligence boundary stays pure functions.
|
||||
|
||||
## The primitive — `Tracker`
|
||||
```
|
||||
STATELESS (Product A): 04a Capabilities + 04b Access → call → data → bill, nothing persists
|
||||
STATEFUL (Product B): 05b Intelligence + 05a Timeline → the Timeline IS the state
|
||||
```
|
||||
|
||||
## Current state (cited)
|
||||
|
||||
- **Capability executors** (`04a`) — called directly by the loop (not through a door).
|
||||
- **Content-hash pre-check** — `WebCrawlerConnector.format_to_structured_document(exclude_metadata=True)`
|
||||
produces the stable text the loop hashes against `entity_current_state.content_hash` (`05a`).
|
||||
- **Run/audit substrate to reuse** — the existing **`AutomationRun`** (status/error/timing/`step_results`)
|
||||
+ the automations executor's PENDING→running idempotency gate (safe under Celery `acks_late`); and the
|
||||
chat **job record** (`04a`) + `deliverable_wait`. **No new run table.**
|
||||
- **CI context uploads** — the existing folder-upload / `destination_folder` machinery + KB retrieval.
|
||||
|
||||
## Target design
|
||||
|
||||
### The primitive — `Tracker`
|
||||
|
||||
A saved, decision-grounded subject that accumulates structured signal over time:
|
||||
|
||||
|
|
@ -32,129 +43,138 @@ A saved, decision-grounded subject that accumulates structured signal over time:
|
|||
| `status` | `draft` → `locked`/`active` |
|
||||
|
||||
One **entity per Tracker for MVP** (one place / one URL). Multi-entity (`maps.search → many`) is deferred
|
||||
(the Timeline model stays multi-entity-ready so it's additive).
|
||||
(the `05a` model stays multi-entity-ready so it's additive).
|
||||
|
||||
## Setup (once) — the agent-designed schema flow (IN MVP)
|
||||
### Setup (once) — the agent-designed schema flow (IN MVP)
|
||||
|
||||
The product must not be rigid: **we cannot author one schema that serves everyone**, so the schema is
|
||||
derived from the *user's* decision by an agent and locked by the human. Conversationally (chat-first,
|
||||
no UI needed):
|
||||
derived from the *user's* decision by an agent and locked by the human. Conversationally (chat-first):
|
||||
|
||||
1. **Bind** a capability + input (the thing to track).
|
||||
2. **Sample fetch** — one real capability call so the agent drafts against *actual* returned data,
|
||||
not a hallucinated shape.
|
||||
3. **Agent drafts the `definition`** from `decision` + the sample:
|
||||
- `field_schema` — the fields that matter + types (structured-output contract).
|
||||
- `materiality` — per-field rules (numeric thresholds where possible; "ask agent" otherwise).
|
||||
- `identity_rule` — the stable entity key (e.g. Maps `place_id`, canonical URL).
|
||||
- a reserved **`notable_signals`** escape-hatch field for the unanticipated.
|
||||
1. **Bind** a capability + input.
|
||||
2. **Sample fetch** — one real capability call so the agent drafts against *actual* returned data.
|
||||
3. **Agent drafts the `definition`** from `decision` + the sample: `field_schema` (fields + types),
|
||||
`materiality` (numeric thresholds where possible; "ask agent" otherwise), `identity_rule` (stable
|
||||
entity key — Maps `place_id`, canonical URL), and a reserved **`notable_signals`** escape-hatch field.
|
||||
4. **Human reviews & locks** (in chat: "looks good" / "add field X"). Locked ⇒ stable run-to-run.
|
||||
5. **Versioned** — a locked `definition` is a snapshot; edits create a new version (mirrors how
|
||||
`automations` snapshots `definition`).
|
||||
|
||||
## The hot loop (per refresh) — `refresh(tracker)`
|
||||
### The hot loop (per refresh) — `refresh(tracker)`
|
||||
|
||||
1. **Crawl** — call the bound capability (Domain ①) → raw data.
|
||||
2. **Cheap pre-check** — content hash via the existing
|
||||
`WebCrawlerConnector.format_to_structured_document(exclude_metadata=True)`; identical hash to the
|
||||
stored `content_hash` → stamp `last_checked_at`, **stop** (no LLM cost).
|
||||
1. **Crawl** — call the bound capability (`04a`) → raw data.
|
||||
2. **Cheap pre-check** — content hash; identical to the stored `content_hash` → stamp `last_checked_at`,
|
||||
**stop** (no LLM cost).
|
||||
3. **Fill** — agent conforms raw data to the **locked `field_schema`** via structured output; it does
|
||||
**not** invent fields. Unanticipated observations go into `notable_signals`.
|
||||
4. **Diff (code)** — deterministic compare of the new record vs Current state (`04`) → raw deltas.
|
||||
4. **Diff (code)** — deterministic compare of the new record vs Current state (`05a`) → raw deltas.
|
||||
5. **Judge — the materiality split:**
|
||||
- **deterministic (code):** numeric / clear-cut rules from `materiality`, applied for free, 100%
|
||||
reproducible. e.g. `rating Δ≥0.2 → material`, `review_count Δ≥10 → material`, `1¢ price wobble
|
||||
→ noise`, `any hours change → material`.
|
||||
- **deterministic (code):** numeric/clear-cut rules from `materiality`, applied for free, 100%
|
||||
reproducible (e.g. `rating Δ≥0.2 → material`, `review_count Δ≥10 → material`, `1¢ wobble → noise`).
|
||||
- **agent (only on ambiguous):** anything a rule can't decide — reworded `description`, a new
|
||||
`notable_signals` entry, "does this matter *for the decision*?" → one LLM call rules
|
||||
material/noise.
|
||||
6. **Append** — if material: write a Change + update Current state (`04`). Else: only `last_checked_at`.
|
||||
`notable_signals` entry → one LLM call rules material/noise.
|
||||
6. **Append** — if material: write a Change + update Current state (`05a`). Else: only `last_checked_at`.
|
||||
**No change → no row.**
|
||||
|
||||
**Worked example (`maps.place` refresh):**
|
||||
```
|
||||
rating 4.4 → 4.3 (Δ0.1) → code: < 0.2 → NOISE (no LLM)
|
||||
review_count 312 → 470 → code: ≥ 10 → MATERIAL (no LLM)
|
||||
hours unchanged → no delta
|
||||
description reworded → code: no rule → ASK AGENT → NOISE
|
||||
rating 4.4 → 4.3 (Δ0.1) → code: < 0.2 → NOISE (no LLM)
|
||||
review_count 312 → 470 → code: ≥ 10 → MATERIAL (no LLM)
|
||||
hours unchanged → no delta
|
||||
description reworded → code: no rule → ASK AGENT → NOISE
|
||||
⇒ one Change row (review spike); one cheap LLM call; zero LLM on the rating tick.
|
||||
```
|
||||
|
||||
## Where it lives / decoupling
|
||||
### Refresh execution & idempotency — ride the invoking surface (no new run table)
|
||||
|
||||
- New **Apache-2** package `app/intelligence/` (the schema-design agent, the hot loop, the materiality
|
||||
evaluator). Calls capability `executor`s directly (not through a door).
|
||||
- Exposes **`refresh(tracker)`**. *Who* calls it (manual / agent / external cron / optional automation) is
|
||||
the **Triggers** domain's concern — Intelligence has **no dependency** on any scheduler.
|
||||
`refresh(tracker)` is a **headless unit of work**; the run/audit record + idempotency live on **whatever
|
||||
surface invoked it**:
|
||||
|
||||
## Refresh execution & idempotency — ride the invoking surface (no new run table)
|
||||
- **Recurring (in-app):** invoked by the **CI automation action** (`06`) → the existing **`AutomationRun`**
|
||||
is the run record and the automations executor provides the **PENDING→running idempotency gate**. This is
|
||||
exactly the rigor old `06` hand-built — reused, not re-written.
|
||||
- **Chat (manual / agent):** invoked via the chat **job record** (`04a`) + `deliverable_wait`.
|
||||
- **Billing idempotency is per *capability call*, not per run:** each `executor` bills a success once via
|
||||
the billing service (`04a`); the content-hash pre-check (step 2) prevents needless re-crawls/charges. No
|
||||
run-level `charged_micros` ledger required for MVP.
|
||||
|
||||
`refresh(tracker)` is a **headless unit of work**; the **run/audit record + idempotency live on whatever
|
||||
surface invoked it**, so we do *not* rebuild old Phase-6 `pipeline_runs`:
|
||||
Net: the only genuinely new state is the **Timeline** (`05a`); execution accounting is borrowed.
|
||||
|
||||
- **Recurring (in-app):** invoked by the **CI automation action** (`05`) → the existing
|
||||
**`AutomationRun`** is the run record (status / error / timing / `step_results`) and the automations
|
||||
executor already provides the **PENDING→running idempotency gate** (safe under Celery `acks_late`
|
||||
redelivery). This is exactly the rigor old `06` hand-built — reused, not re-written.
|
||||
- **Chat (manual / agent):** invoked via the chat **job record** (`01`) + `deliverable_wait` — status
|
||||
lives there.
|
||||
- **Billing idempotency is per *capability call*, not per run:** each `executor` bills a success once
|
||||
via the billing service (`01`); the **content-hash pre-check** (step 2) is what prevents needless
|
||||
re-crawls/charges on an unchanged page. So no run-level `charged_micros` ledger is required for MVP.
|
||||
### User-supplied context files (the F idea, generalized)
|
||||
|
||||
Net: the only genuinely new state is the **Timeline** (`04`); execution accounting is borrowed from
|
||||
`AutomationRun` / the job record.
|
||||
When a user uploads a file *in a CI chat* (e.g. "our own price list"), it goes into the **KB as normal** —
|
||||
uploads create `Document`s and are indexed/embedded, exactly as today. **(The "don't index" rule applies
|
||||
only to *crawled* data.)** The CI-specific part is **organization + use**:
|
||||
|
||||
## User-supplied context files (the F idea, generalized)
|
||||
|
||||
When a user uploads a file *in a CI chat* (e.g. "our own price list", a competitor brochure), it goes
|
||||
into the **KB as normal** — uploads create `Document`s and are indexed/embedded, exactly as today.
|
||||
**(The "don't index" rule applies only to *crawled* data, not to user uploads.)** The CI-specific part
|
||||
is purely **organization + use**:
|
||||
|
||||
- **Routed to a dedicated folder** for that CI chat/Tracker (reuse the existing folder-upload /
|
||||
`destination_folder` machinery), so the chat's reference files are scoped together.
|
||||
- **The judge step (5) may consult them** — retrieved from the KB, scoped to that folder — when ruling
|
||||
materiality:
|
||||
- **Routed to a dedicated folder** for that CI chat/Tracker (reuse the existing folder-upload machinery).
|
||||
- **The judge step (5) may consult them** — retrieved from the KB, scoped to that folder:
|
||||
|
||||
```
|
||||
competitor price 12.00 → 9.90 + user's context file says "our price is 10.00"
|
||||
→ agent: competitor crossed *below our price* → MATERIAL (and explain why)
|
||||
```
|
||||
|
||||
So the user's private context **shapes what counts as material** — a real differentiator — and it
|
||||
reuses the existing KB upload + folder + retrieval machinery (nothing new). **MVP-optional** (the loop
|
||||
works without it); design the seam now so the judge can read the folder later.
|
||||
The user's private context **shapes what counts as material**. Reuses existing KB upload + folder +
|
||||
retrieval machinery (nothing new). **MVP-optional** (the loop works without it); design the seam now.
|
||||
|
||||
## MVP cut vs north star
|
||||
### Where it lives / decoupling
|
||||
|
||||
- **MVP:** agent-designed-schema flow (conversational, sample-grounded, human-locked) · single entity
|
||||
per Tracker · the hot loop with content-hash pre-check + code-threshold + agent-on-ambiguous ·
|
||||
`refresh(tracker)` fired manually / by agent / by external cron.
|
||||
- **North star (deferred):** schema **auto-evolution** from recurring `notable_signals` · multi-entity
|
||||
Trackers (`maps.search`) · backward-replay reconstruction · coverage-confidence · full
|
||||
provenance/explainability · the resale/data-product stage.
|
||||
- New **Apache-2** package `app/intelligence/` (schema-design agent, hot loop, materiality evaluator). Calls
|
||||
capability `executor`s directly.
|
||||
- Exposes **`refresh(tracker)`**. *Who* calls it (manual / agent / external cron / optional automation) is
|
||||
the **Triggers** domain's concern (`06`) — Intelligence has **no dependency** on any scheduler.
|
||||
|
||||
## Locked decisions
|
||||
## Work items
|
||||
|
||||
1. **Tracker model + persistence**: `decision` · `capability_binding` · versioned locked `definition` · `status`.
|
||||
2. **Schema-design flow**: bind → sample-fetch → agent-drafts `definition` → human review/lock → version.
|
||||
3. **Materiality evaluator**: deterministic rule engine (numeric/clear) + the agent-on-ambiguous fallback.
|
||||
4. **The hot loop** `refresh(tracker)`: crawl → hash pre-check → fill → diff → judge → append (writes `05a`).
|
||||
5. **Idempotency wiring**: ride `AutomationRun` (recurring) / chat job record (manual) — no new run table.
|
||||
6. **Context-folder seam**: optional KB-retrieval hook into the judge, scoped to the Tracker's folder.
|
||||
|
||||
## Tests
|
||||
|
||||
- **Pre-check short-circuit**: identical content hash → no fill, no LLM, only `last_checked_at` bumped.
|
||||
- **Fill conforms to lock**: extra observed fields land in `notable_signals`, never invent schema fields.
|
||||
- **Materiality split**: numeric Δ over threshold → `decided_by=code, material`; ambiguous reword →
|
||||
`decided_by=agent`; sub-threshold → noise, no row.
|
||||
- **Append semantics**: a material run writes one `entity_changes` row + overwrites current state.
|
||||
- **Idempotency**: a redelivered recurring refresh (same `AutomationRun`) does not double-write or double-bill.
|
||||
- **Context folder (optional)**: judge ruling flips when a user context file changes the decision frame.
|
||||
|
||||
## Risks / trade-offs
|
||||
|
||||
- **Refresh failure path** (capability returns `FAILED`/partial): skip vs retain vs retry vs alert — an
|
||||
implementation-time call (no architecture impact).
|
||||
- **Agent fill cost** on changed pages: bounded by the hash pre-check; only changed content reaches the LLM.
|
||||
- **Schema lock rigidity**: locking trades flexibility for run-to-run stability; re-lock creates a new version.
|
||||
|
||||
## Resolved decisions
|
||||
|
||||
1. `Tracker` is the standing-concern primitive; replaces "pipeline".
|
||||
2. Stateless (①②/Product A) vs stateful (③④/Product B) is the Access→Intelligence boundary.
|
||||
2. Stateless (`04`/Product A) vs stateful (`05`/Product B) is the Access→Intelligence boundary.
|
||||
3. **Agent-designed schema flow is in MVP** (not hand-authored) — sample-grounded, human-locked, versioned.
|
||||
4. Single entity per Tracker for MVP.
|
||||
5. Materiality = deterministic numeric/clear rules in code + agent only on ambiguous.
|
||||
6. Content-hash pre-check short-circuits unchanged pages before any LLM spend.
|
||||
7. `app/intelligence/` Apache-2; `refresh(tracker)` is trigger-agnostic.
|
||||
8. **No new run table** — refresh audit/idempotency ride `AutomationRun` (recurring) or the chat job
|
||||
record; billing idempotency is per-capability-call + the content-hash gate. Only Timeline (`04`) is
|
||||
new state.
|
||||
8. **No new run table** — refresh audit/idempotency ride `AutomationRun` (recurring) or the chat job record;
|
||||
billing idempotency is per-capability-call + the content-hash gate. Only Timeline (`05a`) is new state.
|
||||
9. **CI context folder** (F): user files uploaded in a CI chat go into the **KB as normal** (indexed),
|
||||
routed to a dedicated `Folder`, and may feed the judge step via KB retrieval. The "don't index"
|
||||
rule is for *crawled* data only. MVP-optional seam.
|
||||
routed to a dedicated `Folder`, and may feed the judge via KB retrieval. "Don't index" is for *crawled*
|
||||
data only. MVP-optional seam.
|
||||
|
||||
## Out of scope (hand-offs)
|
||||
|
||||
- **The state tables** (the three stores, content-hash, read API) → `05a`.
|
||||
- **When refresh fires + recurrence/delivery** → `06` (Triggers).
|
||||
- **The human-facing crafting/answering experience** (the `intelligence_agent` subagent + prompt) → `07`.
|
||||
- **Schema auto-evolution**, multi-entity Trackers, backward-replay, coverage-confidence, the
|
||||
resale/data-product stage → north star (deferred).
|
||||
|
||||
## Open questions (carry forward)
|
||||
|
||||
- How the schema-design agent surfaces the "review & lock" step before the frontend exists (pure-chat
|
||||
confirmation for MVP?).
|
||||
- How the schema-design agent surfaces "review & lock" before the frontend exists (pure-chat confirmation?).
|
||||
- Versioning policy on re-lock (new version vs in-place) — lean new version.
|
||||
- Where the schema-design agent itself runs (a setup capability? a chat sub-flow?).
|
||||
- Context-folder → judge wiring (how much of the folder to load; per-Tracker vs per-chat scope).
|
||||
- Context-folder → judge wiring (how much to load; per-Tracker vs per-chat scope).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue