SurfSense/plans/backend/revamp/01-capabilities.md

Domain ① — Capabilities (CI pivot revamp · WIP)

WIP design doc. Part of the Phase 4 → end revamp (the old "pipeline"). Scope guardrail: Phases 1–3 are SHIPPED and FIXED — the rename, and the proprietary crawler/proxy/billing/stealth/captcha stack. This domain builds on them and never modifies them.

Role in the universe

FIXED foundation (done):  Acquisition (crawler moat) · Metering (03c) · Identity/Tenancy
OUR scope (this revamp):  ▶ Capabilities ◀ → Access → Intelligence + Timeline → Triggers

Capabilities sits directly on the fixed foundation and is consumed by Access (the doors) and the Intelligence hot loop. It is the first domain because everything above it calls into it.

Purpose

Turn the fixed Acquisition engine (+ a new Google Maps actor) into a small set of typed, callable verbs that every door (chat / REST / MCP) and the Intelligence loop consume identically.

This is the core pivot of old Phase 4: replace "connector that ingests into the KB" with "capability that returns data." A capability is a stateless function you call and get data back from — no SearchSourceConnector row to configure, no KB write, no schedule attached.

Foundation it builds on (do not touch)

• WebCrawlerConnector.crawl_url(url) -> CrawlOutcome — single-URL fetch; {content, metadata, crawler_type} on SUCCESS; billable predicate single-sourced as status == SUCCESS. (app/proprietary/web_crawler/.)
• WebCrawlCreditService (03c) — per-success metering + the bill= seam.
• Proxy / stealth / captcha tiers — all behind crawl_url; callers never see the tier.

The verb set (MVP) — namespaced, nothing top-level

Two namespaces: web.* (generic crawler product) and maps.* (the one platform actor). Future platforms slot in as their own namespace (linkedin.*, amazon.*, …).

Verb	Input → Output	Mode	Executes over	Bills (03c)
web.scrape(urls[])	→ [{url, status, content, metadata}]	inline-or-job	loop crawl_url	per success
web.discover(query, top_k)	→ [{url, title, snippet, provider}]	inline	search providers (SearXNG/Linkup/Baidu)	per search (or free — open)
maps.search(query, location)	→ [place]	job	Maps actor (new, proprietary)	per place
maps.place(place_id|url)	→ place (structured)	inline	Maps actor (new, proprietary)	1
maps.reviews(place)	→ [review] (paged)	job	Maps actor (new, proprietary)	per page

web.scrape — one array-friendly verb (no separate "batch")

Scraping always welcomes an array of URLs. There is no separate scrape_batch. One verb, one mental model: "give me URLs, I give you per-URL content."

Execution mode — a property of the result, not two verbs

A capability does not split into sync/async variants. Every call returns a uniform result envelope:

{ status: "completed" | "pending", job_id?: str, progress?: {done, total}, results?: [...] , error?: ... }

• inline (fast path): small/fast input → finishes in-request → envelope returns completed with results inline → zero polling. (single page, one place, a search)
• job (slow path): large/slow input → envelope returns pending + job_id → caller polls a status endpoint until completed, then reads results. (batch scrape, Maps search/reviews)

So "sync" is simply a job that finished instantly. Same verb, same response shape, no magic branching for the caller to reason about.

• Threshold (how many URLs / how heavy before it goes async) is configurable.
• A caller may pass async: true to force a job even for small input (agents that never want to block).
• Only infra this requires: a thin job record (id, status, progress, result_ref) + the existing Celery workers. Not a resurrected pipeline_runs. Sync verbs need none of it.

The capability registry — single source of truth

One registry entry per verb:

Capability {
  name            # dotted, e.g. "web.scrape", "maps.search"
  input_schema    # Pydantic
  output_schema   # Pydantic
  mode            # can-complete-inline?  +  job-capable?
  executor        # the async fn (wraps Acquisition / Maps actor)
  billing_unit    # how Metering charges this call
}

The three doors are generated from the registry, not hand-written three times:

• chat tool ← entry (tool def + handler)
• REST route ← entry (path + request/response models)
• MCP tool ← entry (MCP tool schema + handler)

…and the Intelligence hot loop calls the same executor directly. Add a verb once → it lights up on every surface; the I/O contract cannot drift between surfaces.

Where it lives / license boundary

• New Apache-2 package app/capabilities/ (registry, schemas, executors, the thin job store).
• It imports from the proprietary Acquisition engine but never moves into it.
• The Maps extractor logic is proprietary (app/proprietary/...), consumed by the maps.* executors — same boundary rule Phase 3 set (Apache code may import from proprietary, not move into it).

Billing integration — open to the billing service (not hardcoded per verb)

Billing is not baked into each verb. A capability only declares a billing_unit in its registry entry; charging is delegated to the billing service, so adding/repricing a unit is a billing-service concern, not a capability rewrite. 03c's WebCrawlCreditService is the first provider; new units register with the same service.

• web.scrape → per SUCCESS (existing web_crawl unit).
• maps.* → register a new per-place / per-page unit with the billing service (same wallet).
• captcha attempts → existing per-attempt web_crawl_captcha unit (unchanged).
• web.discover → register a per-search unit (or mark free) with the billing service.

The point: the registry says "this verb bills unit X"; the billing service owns what unit X costs and how it settles. Verbs stay pure; pricing stays pluggable.

The Maps actor (clarification)

• Google-Maps-agnostic — works for any place type (restaurants, gyms, hotels, retail, …).
• Returns typed structured objects ({name, rating, review_count, hours, price_level, …}), not raw markdown — which is exactly what the Intelligence/Timeline layer needs to diff reliably.
• The "restaurant" use case is an Intelligence-domain wedge (one Tracker), not a constraint on the capability. The capability never knows what decision it serves.

Relationship to the drafted Phase 4

• 04b (source-discovery) → absorbed as web.discover + relocating Linkup/Baidu keys from per-connector config to platform env (SearXNG already env-based). Tavily/Serper dropped.
• 04a (connector taxonomy + MCP routing) → demoted to backward-compat hygiene. The capability registry is the new spine, so the Type-1/Type-2 connector registry is no longer central. 04a's "turn off branded connectors" stays a small, separable cleanup for the legacy chat agent; the BYO-MCP_CONNECTOR routing fix belongs to the Access/Conversation domain (it lets the agent use the user's external tools — a different concept from our capabilities).

Deferred (not in MVP)

• Recursive site crawl (follow links across a whole site) — doesn't exist today; the "crawl a topic" story is web.discover → web.scrape.
• A generic web.extract (LLM-structured extraction as a standalone verb) — structured extraction lives in the Intelligence loop for MVP; promote to a capability later if devs want it.
• Additional platform namespaces (linkedin.*, amazon.*, …).

Locked decisions

1. Verbs: web.scrape · web.discover · maps.search · maps.place · maps.reviews. Namespaced; nothing top-level.
2. One array-friendly web.scrape (no separate batch verb).
3. Execution mode = result property (uniform completed/pending envelope), not separate verbs.
4. Build the thin job model now (Maps search / batch scrape force it).
5. One capability registry → generates chat/REST/MCP doors + feeds the Intelligence executor.
6. app/capabilities/ Apache-2; Maps extractor proprietary; billing delegated to the billing service (verbs declare a billing_unit; 03c is the first provider, new units register there).
7. Connector → capability is a replacement for these data sources, not an extension.

Open questions (carry forward)

• Default async threshold for web.scrape.
• The Google Maps actor is net-new (not part of shipped Phases 1–3) — designed/built as a separate effort (incl. sourcing legality). maps.* verbs are contracts against it; this domain doesn't block on it.