Phase 5a — Timeline (the moat asset)

Part of Phase 5 — Intelligence & Timeline. Sibling: 05b-intelligence.md (the process that writes this state). Build first within Phase 5 — the tables must exist before the hot loop writes them. Depends on 04a (the verbs the loop calls) and Phase 1 (the DB / migration baseline). Scope guardrail: Phases 1–3 SHIPPED/FIXED. The Timeline is CI-owned, new tables — it is not the Knowledge Base (documents/embeddings) and not automation_runs.

Objective

Durably store the time-shaped truth for each Tracker. This is the asset: time is the moat — accumulated history a later entrant cannot re-create. Design rule: store deltas, not snapshots — no change, no row. Storage grows with the rate of change, not the number of runs.

Current state (cited)

No CI state exists today — crawled data is explicitly not indexed; the only persisted CI state is what this phase introduces.
Reference models for shape/placement: connectors/folders ORM in app/db.py (full-row Zero publication, like folders/connectors); automations/automation_runs as the orchestration analog (this is the fact analog, deliberately separate).
Hot-loop pre-check (05b step 2) reads WebCrawlerConnector.format_to_structured_document( exclude_metadata=True) to compute the content_hash stored here.

Target design

The state / process split

Intelligence (05b) is the only writer (via the hot loop). Readers — the Conversation domain today, future dashboards/alerts/the deferred resale product — read the Timeline directly, without running the loop. That separation is why Timeline is its own domain.

The three stores

Store	Role	Write pattern
`tracked_entities`	stable identity per tracked thing (the Tracker's `identity_rule` → `entity_key`)	written once
`entity_current_state`	latest values + `content_hash` + `last_checked_at` per entity	overwritten each run
`entity_changes` (the change log)	append-only material deltas	appended, never overwritten

The timeline = the change log read in order. To reconstruct a past point: take Current state and replay deltas backward (north-star tooling; MVP just stores the deltas).

Data model sketch (new tables)

tracked_entities
  id · workspace_id · tracker_id (FK) · entity_key (unique per tracker) · first_seen_at
  # MVP: exactly one row per Tracker (single-entity). Table stays multi-entity-ready.

entity_current_state
  entity_id (FK, unique) · tracker_id · fields JSONB (latest, conforms to locked field_schema)
  · content_hash · last_checked_at · updated_at
  # overwritten each material run; content_hash powers the hot-loop cheap pre-check (05b step 2)

entity_changes                         # the append-only change log
  id · entity_id (FK) · tracker_id · captured_at
  · delta JSONB            # { field: { from, to } }
  · materiality            # material | notable(=notable_signals-sourced)
  · decided_by             # code | agent   (audit of the materiality split)
  · source_ref             # url / blob key the change was observed from
  · note TEXT NULL         # optional agent rationale (the "why material")

No change → no row in entity_changes; an unchanged refresh only bumps last_checked_at.
decided_by records whether code or the agent ruled the change material (provenance seed).

What it is NOT

Not the KB — no Document rows, no embeddings, no indexing. ("Don't index crawled data" holds.)
Not automation_runs — that's an orchestration artifact; this is the durable fact store.
Not a resurrected pipeline_runs.

Where it lives

New CI-owned tables (in app/db.py alongside core entities, or a small app/timeline/ package — decide at write-up). Apache-2 (it stores facts; the moat is in Acquisition + the Maps extractor).
Published to Zero full-row later if/when a UI needs live sync (deferred with the frontend).

Work items

Models + migration: tracked_entities / entity_current_state / entity_changes + Alembic migration.
Write API: upsert_current_state(...) (overwrite) and append_change(...) (insert) used by the 05b loop.
Read API: query_timeline(tracker_id, …) + get_current_state(entity_id) for the conversation/read side.
Content-hash field: store + expose content_hash so 05b step 2 can short-circuit.
Zero publication entry (full-row) — wired but inert until a UI consumes it (deferred).

Tests

No change → no row: an unchanged refresh bumps last_checked_at only; entity_changes count is unchanged.
Append-only log: a material refresh inserts exactly one entity_changes row and overwrites entity_current_state.
Provenance: decided_by is code for threshold rules and agent for ambiguous calls.
Identity: entity_key is unique per tracker; re-refresh of the same entity reuses the row.
Read API: query_timeline returns deltas in captured_at order.

Risks / trade-offs

First-run baseline: with no prior entity_current_state, the diff has nothing to compare against — baseline semantics (silent establish vs flood the log) are an implementation-time call.
High-velocity entities: append-only growth is bounded by change rate, but retention/archival for very chatty entities is deferred.
Schema-validate-on-write: validating fields against the locked field_schema is a cheap integrity guard (lean: yes) but adds a write-path dependency on 05b's lock.

Resolved decisions

Three stores: tracked_entities / entity_current_state / entity_changes.
Store deltas, not snapshots; no change → no row; storage ∝ rate of change.
CI-owned new tables; not KB, not automation_runs.
content_hash on current state powers the hot-loop cheap pre-check.
decided_by on changes records the code-vs-agent materiality provenance.
Single entity per Tracker for MVP; schema stays multi-entity-ready (additive later).

Out of scope (hand-offs)

The writer (hot loop, materiality, schema lock) → 05b.
Backward-replay reconstruction, trend/series read APIs, coverage-confidence, multi-entity scale, the resale/data-product surface → north star (deferred).
Live UI sync (Zero consumption, dashboards) → frontend umbrella.

Open questions (carry forward)

ORM home: app/db.py vs a dedicated app/timeline/ package.
Whether entity_current_state.fields is validated against the locked field_schema at write time (lean: yes).
Retention / archival policy for very high-velocity entities (deferred).

6.7 KiB Raw Blame History Unescape Escape