diff --git a/plans/backend/revamp/05a-timeline.md b/plans/backend/revamp/05a-timeline.md index d66bc23e6..436257cd4 100644 --- a/plans/backend/revamp/05a-timeline.md +++ b/plans/backend/revamp/05a-timeline.md @@ -1,23 +1,36 @@ -# Domain ④ — Timeline (the moat asset) (CI pivot revamp · WIP) +# Phase 5a — Timeline (the moat asset) -> **WIP design doc.** Part of the Phase 4 → end revamp. Pairs with `03-intelligence.md` (the process -> that writes it). -> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. The Timeline is **CI-owned, new tables** — it is -> **not** the Knowledge Base (documents/embeddings) and **not** `automation_runs`. +> Part of **Phase 5 — Intelligence & Timeline**. Sibling: `05b-intelligence.md` (the process that writes +> this state). **Build first** within Phase 5 — the tables must exist before the hot loop writes them. +> **Depends on** `04a` (the verbs the loop calls) and Phase 1 (the DB / migration baseline). +> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. The Timeline is **CI-owned, new tables** — it is **not** +> the Knowledge Base (documents/embeddings) and **not** `automation_runs`. -## Purpose +## Objective Durably store the **time-shaped truth** for each Tracker. This is the asset: **time is the moat** — -accumulated history a later entrant cannot re-create. Design rule: **store deltas, not snapshots — -no change, no row.** Storage grows with the *rate of change*, not the number of runs. +accumulated history a later entrant cannot re-create. Design rule: **store deltas, not snapshots — no +change, no row.** Storage grows with the *rate of change*, not the number of runs. -## The state / process split +## Current state (cited) -Intelligence (`03`) is the only **writer** (via the hot loop). **Readers** — the Conversation domain -today, future dashboards / alerts / the deferred resale product — read the Timeline **directly, -without running the loop**. That separation is the reason Timeline is its own domain. +- **No CI state exists today** — crawled data is explicitly *not* indexed; the only persisted CI state is + what this phase introduces. +- **Reference models** for shape/placement: connectors/folders ORM in `app/db.py` (full-row Zero + publication, like folders/connectors); `automations`/`automation_runs` as the *orchestration* analog + (this is the **fact** analog, deliberately separate). +- **Hot-loop pre-check** (`05b` step 2) reads `WebCrawlerConnector.format_to_structured_document( + exclude_metadata=True)` to compute the `content_hash` stored here. -## The three stores +## Target design + +### The state / process split + +Intelligence (`05b`) is the only **writer** (via the hot loop). **Readers** — the Conversation domain +today, future dashboards/alerts/the deferred resale product — read the Timeline **directly, without +running the loop**. That separation is why Timeline is its own domain. + +### The three stores | Store | Role | Write pattern | |-------|------|---------------| @@ -26,9 +39,9 @@ without running the loop**. That separation is the reason Timeline is its own do | `entity_changes` (the change log) | append-only material deltas | **appended**, never overwritten | **The timeline = the change log read in order.** To reconstruct a past point: take Current state and -replay deltas backward (north-star tooling; not built in MVP — we just *store* the deltas). +replay deltas backward (north-star tooling; MVP just *stores* the deltas). -## Data model sketch (new tables) +### Data model sketch (new tables) ``` tracked_entities @@ -38,7 +51,7 @@ tracked_entities entity_current_state entity_id (FK, unique) · tracker_id · fields JSONB (latest, conforms to locked field_schema) · content_hash · last_checked_at · updated_at - # overwritten each material run; content_hash powers the hot-loop cheap pre-check (03 step 2) + # overwritten each material run; content_hash powers the hot-loop cheap pre-check (05b step 2) entity_changes # the append-only change log id · entity_id (FK) · tracker_id · captured_at @@ -49,33 +62,47 @@ entity_changes # the append-only change log · note TEXT NULL # optional agent rationale (the "why material") ``` -- **No change → no row** in `entity_changes`; an unchanged refresh only bumps - `entity_current_state.last_checked_at`. -- `decided_by` records whether code or the agent ruled the change material (provenance seed for the - north-star explainability work). +- **No change → no row** in `entity_changes`; an unchanged refresh only bumps `last_checked_at`. +- `decided_by` records whether code or the agent ruled the change material (provenance seed). -## What it is NOT +### What it is NOT -- **Not the KB** — no `Document` rows, no embeddings, no indexing pipeline. (The meeting's "don't - index crawled data" holds.) +- **Not the KB** — no `Document` rows, no embeddings, no indexing. ("Don't index crawled data" holds.) - **Not `automation_runs`** — that's an orchestration artifact; this is the durable fact store. - **Not a resurrected `pipeline_runs`.** -## Where it lives +### Where it lives -- New CI-owned tables (in `app/db.py` alongside the other core entities, or a small `app/timeline/` - package — decide at write-up). **Apache-2** (it stores facts; the moat is in Acquisition + the Maps - extractor, not here). +- New CI-owned tables (in `app/db.py` alongside core entities, or a small `app/timeline/` package — decide + at write-up). **Apache-2** (it stores facts; the moat is in Acquisition + the Maps extractor). - Published to Zero full-row later if/when a UI needs live sync (deferred with the frontend). -## MVP cut vs north star +## Work items -- **MVP:** the three stores · single entity per Tracker · deltas with `decided_by`/`source_ref` · - `content_hash` pre-check support. -- **North star (deferred):** backward-replay reconstruction queries · trend/series read APIs · - coverage-confidence metadata · multi-entity scale · the resale/data-product surface. +1. **Models + migration**: `tracked_entities` / `entity_current_state` / `entity_changes` + Alembic migration. +2. **Write API**: `upsert_current_state(...)` (overwrite) and `append_change(...)` (insert) used by the `05b` loop. +3. **Read API**: `query_timeline(tracker_id, …)` + `get_current_state(entity_id)` for the conversation/read side. +4. **Content-hash field**: store + expose `content_hash` so `05b` step 2 can short-circuit. +5. **Zero publication entry** (full-row) — wired but inert until a UI consumes it (deferred). -## Locked decisions +## Tests + +- **No change → no row**: an unchanged refresh bumps `last_checked_at` only; `entity_changes` count is unchanged. +- **Append-only log**: a material refresh inserts exactly one `entity_changes` row and overwrites `entity_current_state`. +- **Provenance**: `decided_by` is `code` for threshold rules and `agent` for ambiguous calls. +- **Identity**: `entity_key` is unique per tracker; re-refresh of the same entity reuses the row. +- **Read API**: `query_timeline` returns deltas in `captured_at` order. + +## Risks / trade-offs + +- **First-run baseline**: with no prior `entity_current_state`, the diff has nothing to compare against — + baseline semantics (silent establish vs flood the log) are an implementation-time call. +- **High-velocity entities**: append-only growth is bounded by change rate, but retention/archival for very + chatty entities is deferred. +- **Schema-validate-on-write**: validating `fields` against the locked `field_schema` is a cheap integrity + guard (lean: yes) but adds a write-path dependency on `05b`'s lock. + +## Resolved decisions 1. Three stores: `tracked_entities` / `entity_current_state` / `entity_changes`. 2. Store deltas, not snapshots; **no change → no row**; storage ∝ rate of change. @@ -84,9 +111,15 @@ entity_changes # the append-only change log 5. `decided_by` on changes records the code-vs-agent materiality provenance. 6. Single entity per Tracker for MVP; schema stays multi-entity-ready (additive later). +## Out of scope (hand-offs) + +- **The writer** (hot loop, materiality, schema lock) → `05b`. +- **Backward-replay reconstruction**, trend/series read APIs, coverage-confidence, multi-entity scale, the + resale/data-product surface → north star (deferred). +- **Live UI sync** (Zero consumption, dashboards) → frontend umbrella. + ## Open questions (carry forward) - ORM home: `app/db.py` vs a dedicated `app/timeline/` package. -- Whether `entity_current_state.fields` should be validated against the Tracker's locked `field_schema` - at write time (lean: yes, cheap integrity guard). +- Whether `entity_current_state.fields` is validated against the locked `field_schema` at write time (lean: yes). - Retention / archival policy for very high-velocity entities (deferred).