docs(revamp): reshape Timeline into Phase 5a

plans/backend/revamp/05a-timeline.md

@@ -1,23 +1,36 @@
-# Domain ④ — Timeline (the moat asset) (CI pivot revamp · WIP)
+# Phase 5a — Timeline (the moat asset)
 
-> **WIP design doc.** Part of the Phase 4 → end revamp. Pairs with `03-intelligence.md` (the process
-> that writes it).
-> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. The Timeline is **CI-owned, new tables** — it is
-> **not** the Knowledge Base (documents/embeddings) and **not** `automation_runs`.
+> Part of **Phase 5 — Intelligence & Timeline**. Sibling: `05b-intelligence.md` (the process that writes
+> this state). **Build first** within Phase 5 — the tables must exist before the hot loop writes them.
+> **Depends on** `04a` (the verbs the loop calls) and Phase 1 (the DB / migration baseline).
+> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. The Timeline is **CI-owned, new tables** — it is **not**
+> the Knowledge Base (documents/embeddings) and **not** `automation_runs`.
 
-## Purpose
+## Objective
 
 Durably store the **time-shaped truth** for each Tracker. This is the asset: **time is the moat** —
-accumulated history a later entrant cannot re-create. Design rule: **store deltas, not snapshots —
-no change, no row.** Storage grows with the *rate of change*, not the number of runs.
+accumulated history a later entrant cannot re-create. Design rule: **store deltas, not snapshots — no
+change, no row.** Storage grows with the *rate of change*, not the number of runs.
 
-## The state / process split
+## Current state (cited)
 
-Intelligence (`03`) is the only **writer** (via the hot loop). **Readers** — the Conversation domain
-today, future dashboards / alerts / the deferred resale product — read the Timeline **directly,
-without running the loop**. That separation is the reason Timeline is its own domain.
+- **No CI state exists today** — crawled data is explicitly *not* indexed; the only persisted CI state is
+  what this phase introduces.
+- **Reference models** for shape/placement: connectors/folders ORM in `app/db.py` (full-row Zero
+  publication, like folders/connectors); `automations`/`automation_runs` as the *orchestration* analog
+  (this is the **fact** analog, deliberately separate).
+- **Hot-loop pre-check** (`05b` step 2) reads `WebCrawlerConnector.format_to_structured_document(
+  exclude_metadata=True)` to compute the `content_hash` stored here.
 
-## The three stores
+## Target design
+
+### The state / process split
+
+Intelligence (`05b`) is the only **writer** (via the hot loop). **Readers** — the Conversation domain
+today, future dashboards/alerts/the deferred resale product — read the Timeline **directly, without
+running the loop**. That separation is why Timeline is its own domain.
+
+### The three stores
 
 | Store | Role | Write pattern |
 |-------|------|---------------|
@@ -26,9 +39,9 @@ without running the loop**. That separation is the reason Timeline is its own do
 | `entity_changes` (the change log) | append-only material deltas | **appended**, never overwritten |
 
 **The timeline = the change log read in order.** To reconstruct a past point: take Current state and
-replay deltas backward (north-star tooling; not built in MVP — we just *store* the deltas).
+replay deltas backward (north-star tooling; MVP just *stores* the deltas).
 
-## Data model sketch (new tables)
+### Data model sketch (new tables)
 
 ```
 tracked_entities
@@ -38,7 +51,7 @@ tracked_entities
 entity_current_state
   entity_id (FK, unique) · tracker_id · fields JSONB (latest, conforms to locked field_schema)
   · content_hash · last_checked_at · updated_at
-  # overwritten each material run; content_hash powers the hot-loop cheap pre-check (03 step 2)
+  # overwritten each material run; content_hash powers the hot-loop cheap pre-check (05b step 2)
 
 entity_changes                         # the append-only change log
   id · entity_id (FK) · tracker_id · captured_at
@@ -49,33 +62,47 @@ entity_changes                         # the append-only change log
   · note TEXT NULL         # optional agent rationale (the "why material")
 ```
 
-- **No change → no row** in `entity_changes`; an unchanged refresh only bumps
-  `entity_current_state.last_checked_at`.
-- `decided_by` records whether code or the agent ruled the change material (provenance seed for the
-  north-star explainability work).
+- **No change → no row** in `entity_changes`; an unchanged refresh only bumps `last_checked_at`.
+- `decided_by` records whether code or the agent ruled the change material (provenance seed).
 
-## What it is NOT
+### What it is NOT
 
-- **Not the KB** — no `Document` rows, no embeddings, no indexing pipeline. (The meeting's "don't
-  index crawled data" holds.)
+- **Not the KB** — no `Document` rows, no embeddings, no indexing. ("Don't index crawled data" holds.)
 - **Not `automation_runs`** — that's an orchestration artifact; this is the durable fact store.
 - **Not a resurrected `pipeline_runs`.**
 
-## Where it lives
+### Where it lives
 
-- New CI-owned tables (in `app/db.py` alongside the other core entities, or a small `app/timeline/`
-  package — decide at write-up). **Apache-2** (it stores facts; the moat is in Acquisition + the Maps
-  extractor, not here).
+- New CI-owned tables (in `app/db.py` alongside core entities, or a small `app/timeline/` package — decide
+  at write-up). **Apache-2** (it stores facts; the moat is in Acquisition + the Maps extractor).
 - Published to Zero full-row later if/when a UI needs live sync (deferred with the frontend).
 
-## MVP cut vs north star
+## Work items
 
-- **MVP:** the three stores · single entity per Tracker · deltas with `decided_by`/`source_ref` ·
-  `content_hash` pre-check support.
-- **North star (deferred):** backward-replay reconstruction queries · trend/series read APIs ·
-  coverage-confidence metadata · multi-entity scale · the resale/data-product surface.
+1. **Models + migration**: `tracked_entities` / `entity_current_state` / `entity_changes` + Alembic migration.
+2. **Write API**: `upsert_current_state(...)` (overwrite) and `append_change(...)` (insert) used by the `05b` loop.
+3. **Read API**: `query_timeline(tracker_id, …)` + `get_current_state(entity_id)` for the conversation/read side.
+4. **Content-hash field**: store + expose `content_hash` so `05b` step 2 can short-circuit.
+5. **Zero publication entry** (full-row) — wired but inert until a UI consumes it (deferred).
 
-## Locked decisions
+## Tests
+
+- **No change → no row**: an unchanged refresh bumps `last_checked_at` only; `entity_changes` count is unchanged.
+- **Append-only log**: a material refresh inserts exactly one `entity_changes` row and overwrites `entity_current_state`.
+- **Provenance**: `decided_by` is `code` for threshold rules and `agent` for ambiguous calls.
+- **Identity**: `entity_key` is unique per tracker; re-refresh of the same entity reuses the row.
+- **Read API**: `query_timeline` returns deltas in `captured_at` order.
+
+## Risks / trade-offs
+
+- **First-run baseline**: with no prior `entity_current_state`, the diff has nothing to compare against —
+  baseline semantics (silent establish vs flood the log) are an implementation-time call.
+- **High-velocity entities**: append-only growth is bounded by change rate, but retention/archival for very
+  chatty entities is deferred.
+- **Schema-validate-on-write**: validating `fields` against the locked `field_schema` is a cheap integrity
+  guard (lean: yes) but adds a write-path dependency on `05b`'s lock.
+
+## Resolved decisions
 
 1. Three stores: `tracked_entities` / `entity_current_state` / `entity_changes`.
 2. Store deltas, not snapshots; **no change → no row**; storage ∝ rate of change.
@@ -84,9 +111,15 @@ entity_changes                         # the append-only change log
 5. `decided_by` on changes records the code-vs-agent materiality provenance.
 6. Single entity per Tracker for MVP; schema stays multi-entity-ready (additive later).
 
+## Out of scope (hand-offs)
+
+- **The writer** (hot loop, materiality, schema lock) → `05b`.
+- **Backward-replay reconstruction**, trend/series read APIs, coverage-confidence, multi-entity scale, the
+  resale/data-product surface → north star (deferred).
+- **Live UI sync** (Zero consumption, dashboards) → frontend umbrella.
+
 ## Open questions (carry forward)
 
 - ORM home: `app/db.py` vs a dedicated `app/timeline/` package.
-- Whether `entity_current_state.fields` should be validated against the Tracker's locked `field_schema`
-  at write time (lean: yes, cheap integrity guard).
+- Whether `entity_current_state.fields` is validated against the locked `field_schema` at write time (lean: yes).
 - Retention / archival policy for very high-velocity entities (deferred).