Phase 4 — Capabilities (the scraper APIs)

Build first; every other phase calls this registry. Sibling: 05-access.md (the doors). Reuses Phases 3a/3b/3c (shipped): WebCrawlerConnector.crawl_url -> CrawlOutcome, the proxy/stealth/ captcha tiers, and the WebCrawlCreditService billing seam. Locate code by symbol/grep.

Objective

Expose the Acquisition engine as a small set of typed, stateless scraper verbs (web.*, maps.*) that return cleaned, AI-ready structured data. One capability = one function you call and get data back from.

Verb set (MVP)

Two namespaces: web.* (generic crawler) and maps.* (Google Maps actor). Future platforms slot in as their own namespace.

Verb	Input → Output (cleaned)	Executes over	Billing unit
`web.scrape(urls[])`	`[{url, status, content, metadata}]`	loop `crawl_url`	per success
`web.discover(query, top_k)`	`[{url, title, snippet, provider}]`	search providers	per search / free
`maps.search(query, location)`	`[place]`	Maps actor	per place
`maps.place(place_id\|url)`	`place`	Maps actor	1
`maps.reviews(place)`	`[review]` (paged)	Maps actor	per page

maps.* returns typed structured objects ({name, rating, review_count, hours, price_level, …}). web.scrape takes a URL array.

Execution

Each verb is a direct async call that returns its cleaned result. Slow verbs (large web.scrape arrays, maps.reviews paging) are bounded/streamed at the endpoint.

Registry

One entry per verb, and the single source of truth the doors (05) and the agent (07) read from:

Capability {
  name            # dotted, e.g. "web.scrape", "maps.search"
  input_schema    # Pydantic
  output_schema   # Pydantic (cleaned, AI-ready)
  executor        # async fn; wraps Acquisition / Maps actor; returns data directly
  billing_unit    # how the billing service charges this call
}

Adding a verb once lights it up on every door.

Billing

A verb declares a billing_unit; charging is delegated to the billing service (03c first provider).

web.scrape → per SUCCESS (web_crawl unit).
maps.* → new per-place / per-page unit registered with the billing service.
captcha attempts → existing per-attempt web_crawl_captcha unit (03d).
web.discover → per-search unit (or free).

Location

New Apache-2 package app/capabilities/ (registry, schemas, executors) — open-source and self-hostable. Executors import the proprietary Acquisition engine and Maps extractor (app/proprietary/…).

Work items

Registry package: Capability dataclass + registry, imported by 05 and 07.
web.scrape executor: loop crawl_url over the URL array → per-URL cleaned rows; web_crawl unit.
web.discover executor: wrap the search-provider core (SearXNG/Linkup/Baidu, env-keyed); its unit.
maps.* contracts: input/output schemas + executor stubs against the Maps actor.
Output normalization: each executor returns AI-ready structured output.
Billing seam: honor billing_unit via the billing service.

Tests

web.scrape([a,b,c]) returns one cleaned row per URL inline; partial failures don't fail the batch.
Each SUCCESS bills one web_crawl unit; EMPTY/FAILED free; disabling the flag no-ops.
A door and the agent hit the same executor for a verb.
web.discover returns {url,title,snippet,provider}; self-disables when no provider key is set.
maps.* returns typed structured objects against fixtures.

Out of scope

Doors → 05. Keep-watching mode → 06. Agent → 07.
Google Maps actor build (incl. sourcing legality) — net-new, separate effort; maps.* are contracts.
Additional platform namespaces; recursive site crawl; a generic web.extract verb.

Open questions

web.discover metered vs free.
How large web.scrape / maps.reviews responses are bounded/streamed.

3.9 KiB Raw Blame History