feat(chat): add multi-agent mode routing scaffold and telemetry.

docs/multi-agent-architecture-plan.md

@@ -0,0 +1,211 @@
+# Multi-Agent Architecture Plan (Phased)
+
+This document defines a phased migration from the current `single_agent` flow to a multi-agent architecture while keeping rollback simple and immediate.
+
+## Naming
+
+- `single_agent`: current architecture (default at start)
+- `shadow_multi_agent_v1`: run multi-agent path in background, return `single_agent` output
+- `multi_agent_v1`: multi-agent architecture is the user-facing path
+
+---
+
+## Phase 1 - Parallel Safety Layer
+
+**Goal:** Add safe routing controls with zero behavior change.
+
+### Todo
+
+- [ ] Add mode selector with values: `single_agent`, `shadow_multi_agent_v1`, `multi_agent_v1`
+- [ ] Add global kill switch: force all traffic to `single_agent`
+- [ ] Add mode resolution priority:
+  1. kill switch
+  2. request override
+  3. system default
+- [ ] Keep `single_agent` as default mode
+- [ ] Keep frontend stream/output contract unchanged
+- [ ] Add telemetry tags:
+  - `architecture_mode`
+  - `worker_count`
+  - `retry_count`
+  - `latency_ms`
+  - `token_total`
+- [ ] Write short rollback runbook
+
+### Exit Criteria
+
+- [ ] Can switch modes in staging
+- [ ] Kill switch verified
+- [ ] No frontend contract regressions
+
+---
+
+## Phase 2 - Orchestrator Core and Contracts
+
+**Goal:** Build multi-agent control-plane only (planner/router/merge), with strict schemas.
+
+### Todo
+
+- [ ] Implement orchestrator responsibilities:
+  - intent detection
+  - routing
+  - delegation
+  - fan-in merge
+- [ ] Add budget controls:
+  - max workers per turn
+  - max parallel workers
+  - max turn duration
+- [ ] Add loop/stall guard:
+  - repeated task signature detection
+  - no-progress threshold
+- [ ] Define `WorkerTask` schema:
+  - `domain`, `goal`, `constraints`, `budget`
+- [ ] Define `WorkerResult` schema:
+  - `status`, `summary`, `evidence[]`, `artifacts[]`, `needs_human`, `error_class`
+- [ ] Add schema validation on send/receive boundaries
+- [ ] Add controlled fallback on invalid worker results
+
+### Exit Criteria
+
+- [ ] Orchestrator works end-to-end with mock workers
+- [ ] Invalid worker payloads are blocked cleanly
+
+---
+
+## Phase 3 - Pilot Workers (Gmail and Calendar)
+
+**Goal:** Validate multi-agent architecture with two real domains only.
+
+### Todo
+
+- [ ] Create Gmail worker
+  - [ ] domain-scoped prompt
+  - [ ] domain-only tool loadout
+  - [ ] local query rewrite
+  - [ ] normalized `WorkerResult`
+- [ ] Create Calendar worker
+  - [ ] domain-scoped prompt
+  - [ ] domain-only tool loadout
+  - [ ] local query rewrite/time normalization
+  - [ ] normalized `WorkerResult`
+- [ ] Enforce no cross-domain tool access
+- [ ] Preserve HITL for write actions
+- [ ] Add retry policy by `error_class`
+- [ ] Add tests for routing, loadout isolation, HITL behavior
+
+### Exit Criteria
+
+- [ ] Gmail and Calendar tasks complete in `multi_agent_v1`
+- [ ] No cross-domain tool leakage
+- [ ] HITL still enforced for sensitive writes
+
+---
+
+## Phase 4 - Knowledge Base and Evidence Normalization
+
+**Goal:** Isolate KB retrieval and make evidence citation-ready.
+
+### Todo
+
+- [ ] Move KB retrieval behind dedicated worker/stage
+- [ ] Reuse current KB retrieval logic, but return compact structured evidence only
+- [ ] Define `EvidenceItem` fields:
+  - `claim`, `source_type`, `source_ref`, `confidence`, `snippet`
+- [ ] Add top-k and output-size controls
+- [ ] Add quote-first extraction mode for long contexts
+- [ ] Add tests for traceability and bounded payloads
+
+### Exit Criteria
+
+- [ ] Orchestrator consumes compact evidence (no raw KB dumps)
+- [ ] Citation refs remain valid and traceable
+
+---
+
+## Phase 5 - Verifier and Citation Gate
+
+**Goal:** Prevent unsupported factual claims in final responses.
+
+### Todo
+
+- [ ] Add verifier stage before final synthesis
+- [ ] Enforce claim-to-evidence checks
+- [ ] Add conflict handling policy:
+  - consistent evidence -> accept
+  - conflicting evidence -> label uncertainty or retry
+- [ ] Add unsupported-claim policy:
+  - remove claim or mark uncertain
+- [ ] Add verifier telemetry:
+  - supported claims
+  - unsupported claims
+  - conflicts
+- [ ] Support strict gate and warning modes
+
+### Exit Criteria
+
+- [ ] Unsupported factual claims are blocked or clearly annotated
+- [ ] Citation precision improves on evaluation set
+
+---
+
+## Phase 6 - Shadow Evaluation and Canary
+
+**Goal:** Ship based on data, not intuition.
+
+### Todo
+
+- [ ] Enable `shadow_multi_agent_v1` for selected traffic
+- [ ] Compare metrics vs `single_agent`:
+  - success rate
+  - citation precision
+  - tool-selection accuracy
+  - p95 latency
+  - tokens/request
+  - cost per successful task
+- [ ] Define rollout gates and auto-stop thresholds
+- [ ] Start canary rollout for `multi_agent_v1`
+- [ ] Ramp traffic only if quality and reliability gates pass
+- [ ] Keep kill switch live for entire rollout
+- [ ] Record go/no-go decision with evidence
+
+### Exit Criteria
+
+- [ ] Clear decision based on measured outcomes
+- [ ] Rollback tested successfully during canary
+
+---
+
+## Phase 7 - Domain Expansion and Heavy Tool Reassignment
+
+**Goal:** Scale multi-agent architecture safely across more domains.
+
+### Todo
+
+- [ ] Add domains incrementally (`notion`, `slack`, `jira`, ...)
+- [ ] For each new domain enforce:
+  - scoped tool loadout
+  - local query rewrite
+  - contract validation
+  - eval plus canary gate
+- [ ] Move heavy tools to specialist workers:
+  - podcast generation
+  - artifact/report generation
+  - video presentation
+- [ ] Keep orchestrator toolbelt minimal and control-plane focused
+- [ ] Regularly prune prompts and tool descriptions
+
+### Exit Criteria
+
+- [ ] New domains onboard without reliability regressions
+- [ ] Orchestrator remains lean and stable
+- [ ] Cost per successful task stays controlled
+
+---
+
+## Always-On Checklist
+
+- [ ] Keep `single_agent` path healthy until rollout completion
+- [ ] Keep one-click rollback available at all times
+- [ ] Update observability dashboards every phase
+- [ ] Track failure taxonomy and review weekly
+- [ ] Validate prompt/tool changes via eval before broad rollout

docs/multi-agent-phase1-runbook.md

@@ -0,0 +1,70 @@
+# Multi-Agent Architecture Phase 1 Runbook
+
+## Scope
+
+This runbook covers mode selection and emergency rollback for:
+
+- `single_agent`
+- `shadow_multi_agent_v1`
+- `multi_agent_v1`
+
+Phase 1 keeps execution behavior on the current single-agent path while mode wiring and telemetry are introduced.
+
+## Resolution Priority
+
+Mode resolution follows this fixed order:
+
+1. Global kill switch (`FORCE_SINGLE_AGENT`)
+2. Request override (`architecture_mode` in chat payload)
+3. System default (`AGENT_ARCHITECTURE_MODE`)
+4. Safe fallback (`single_agent`)
+
+## Configuration
+
+Set environment values in backend runtime:
+
+- `AGENT_ARCHITECTURE_MODE=single_agent` (default)
+- `FORCE_SINGLE_AGENT=FALSE` (default)
+
+Changes require backend restart because config is loaded at process startup.
+
+## Mode Switching
+
+### System default switch
+
+1. Set `AGENT_ARCHITECTURE_MODE` to desired value.
+2. Keep `FORCE_SINGLE_AGENT=FALSE`.
+3. Restart backend.
+4. Verify logs include `[architecture_telemetry]` with expected `architecture_mode`.
+
+### Per-request override
+
+Send optional `architecture_mode` in chat request payload:
+
+- `"single_agent"`
+- `"shadow_multi_agent_v1"`
+- `"multi_agent_v1"`
+
+If `FORCE_SINGLE_AGENT=TRUE`, request override is ignored by design.
+
+## Emergency Rollback
+
+Use the kill switch:
+
+1. Set `FORCE_SINGLE_AGENT=TRUE`.
+2. Restart backend.
+3. Verify new requests log `architecture_mode=single_agent`.
+4. Keep this state until incident is resolved.
+
+## Verification Checklist
+
+- Mode resolves according to the priority order.
+- Kill switch overrides all request/default values.
+- Streaming response schema remains unchanged.
+- Architecture telemetry is emitted with:
+  - `architecture_mode`
+  - `orchestrator_used`
+  - `worker_count`
+  - `retry_count`
+  - `latency_ms`
+  - `token_total`