mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-06 22:12:12 +02:00
90 lines
3.9 KiB
Markdown
90 lines
3.9 KiB
Markdown
|
# Phase 4 — Capabilities (the scraper APIs)
|
||
|
|
|
||
|
|
> Build first; every other phase calls this registry. Sibling: `05-access.md` (the doors).
|
||
|
|
> Reuses Phases 3a/3b/3c (shipped): `WebCrawlerConnector.crawl_url -> CrawlOutcome`, the proxy/stealth/
|
||
|
|
> captcha tiers, and the `WebCrawlCreditService` billing seam. Locate code by symbol/grep.
|
||
|
|
|
||
|
|
## Objective
|
||
|
|
|
||
|
|
Expose the Acquisition engine as a small set of typed, stateless scraper verbs (`web.*`, `maps.*`) that
|
||
|
|
return cleaned, AI-ready structured data. One capability = one function you call and get data back from.
|
||
|
|
|
||
|
|
## Verb set (MVP)
|
||
|
|
|
||
|
|
Two namespaces: `web.*` (generic crawler) and `maps.*` (Google Maps actor). Future platforms slot in as
|
||
|
|
their own namespace.
|
||
|
|
|
||
|
|
| Verb | Input → Output (cleaned) | Executes over | Billing unit |
|
||
|
|
|------|--------------------------|---------------|--------------|
|
||
|
|
| `web.scrape(urls[])` | `[{url, status, content, metadata}]` | loop `crawl_url` | per success |
|
||
|
|
| `web.discover(query, top_k)` | `[{url, title, snippet, provider}]` | search providers | per search / free |
|
||
|
|
| `maps.search(query, location)` | `[place]` | Maps actor | per place |
|
||
|
|
| `maps.place(place_id\|url)` | `place` | Maps actor | 1 |
|
||
|
|
| `maps.reviews(place)` | `[review]` (paged) | Maps actor | per page |
|
||
|
|
|
||
|
|
`maps.*` returns typed structured objects (`{name, rating, review_count, hours, price_level, …}`).
|
||
|
|
`web.scrape` takes a URL array.
|
||
|
|
|
||
|
|
## Execution
|
||
|
|
|
||
|
|
Each verb is a direct async call that returns its cleaned result. Slow verbs (large `web.scrape` arrays,
|
||
|
|
`maps.reviews` paging) are bounded/streamed at the endpoint.
|
||
|
|
|
||
|
|
## Registry
|
||
|
|
|
||
|
|
One entry per verb, and the single source of truth the doors (`05`) and the agent (`07`) read from:
|
||
|
|
|
||
|
|
```
|
||
|
|
Capability {
|
||
|
|
name # dotted, e.g. "web.scrape", "maps.search"
|
||
|
|
input_schema # Pydantic
|
||
|
|
output_schema # Pydantic (cleaned, AI-ready)
|
||
|
|
executor # async fn; wraps Acquisition / Maps actor; returns data directly
|
||
|
|
billing_unit # how the billing service charges this call
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
Adding a verb once lights it up on every door.
|
||
|
|
|
||
|
|
## Billing
|
||
|
|
|
||
|
|
A verb declares a `billing_unit`; charging is delegated to the billing service (`03c` first provider).
|
||
|
|
|
||
|
|
- `web.scrape` → per `SUCCESS` (`web_crawl` unit).
|
||
|
|
- `maps.*` → new per-place / per-page unit registered with the billing service.
|
||
|
|
- captcha attempts → existing per-attempt `web_crawl_captcha` unit (`03d`).
|
||
|
|
- `web.discover` → per-search unit (or free).
|
||
|
|
|
||
|
|
## Location
|
||
|
|
|
||
|
|
New Apache-2 package `app/capabilities/` (registry, schemas, executors) — open-source and self-hostable.
|
||
|
|
Executors import the proprietary Acquisition engine and Maps extractor (`app/proprietary/…`).
|
||
|
|
|
||
|
|
## Work items
|
||
|
|
|
||
|
|
1. Registry package: `Capability` dataclass + registry, imported by `05` and `07`.
|
||
|
|
2. `web.scrape` executor: loop `crawl_url` over the URL array → per-URL cleaned rows; `web_crawl` unit.
|
||
|
|
3. `web.discover` executor: wrap the search-provider core (SearXNG/Linkup/Baidu, env-keyed); its unit.
|
||
|
|
4. `maps.*` contracts: input/output schemas + executor stubs against the Maps actor.
|
||
|
|
5. Output normalization: each executor returns AI-ready structured output.
|
||
|
|
6. Billing seam: honor `billing_unit` via the billing service.
|
||
|
|
|
||
|
|
## Tests
|
||
|
|
|
||
|
|
- `web.scrape([a,b,c])` returns one cleaned row per URL inline; partial failures don't fail the batch.
|
||
|
|
- Each `SUCCESS` bills one `web_crawl` unit; `EMPTY`/`FAILED` free; disabling the flag no-ops.
|
||
|
|
- A door and the agent hit the same executor for a verb.
|
||
|
|
- `web.discover` returns `{url,title,snippet,provider}`; self-disables when no provider key is set.
|
||
|
|
- `maps.*` returns typed structured objects against fixtures.
|
||
|
|
|
||
|
|
## Out of scope
|
||
|
|
|
||
|
|
- Doors → `05`. Keep-watching mode → `06`. Agent → `07`.
|
||
|
|
- Google Maps actor build (incl. sourcing legality) — net-new, separate effort; `maps.*` are contracts.
|
||
|
|
- Additional platform namespaces; recursive site crawl; a generic `web.extract` verb.
|
||
|
|
|
||
|
|
## Open questions
|
||
|
|
|
||
|
|
- `web.discover` metered vs free.
|
||
|
|
- How large `web.scrape` / `maps.reviews` responses are bounded/streamed.
|