mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-10 22:32:16 +02:00
125 lines
6.5 KiB
Markdown
125 lines
6.5 KiB
Markdown
|
# Web Crawler Engine
|
||
|
|
|
||
|
|
Proprietary crawling engine (licensed separately from the Apache-2.0 project
|
||
|
|
root — see `app/proprietary/LICENSE`). Single framework (Scrapling) for
|
||
|
|
fetching, Trafilatura for HTML → markdown extraction. Callers import only from
|
||
|
|
`__init__.py`: `WebCrawlerConnector` / `crawl_url` for one page, `crawl_site`
|
||
|
|
for depth-bounded multi-page crawls, both returning the same outcome contract.
|
||
|
|
|
||
|
|
## Module map
|
||
|
|
|
||
|
|
| Module | Role |
|
||
|
|
|---|---|
|
||
|
|
| `connector.py` | Single-URL crawl: tiered fetch ladder, extraction, escalation heuristics |
|
||
|
|
| `site_crawler.py` | Multi-page crawl: Scrapling `CrawlerEngine` frontier over the connector |
|
||
|
|
| `url_policy.py` | Link record extraction and categorization (nav/social/contact/document) |
|
||
|
|
| `captcha.py` | Captcha detection, token harvesting, and injection page-actions |
|
||
|
|
| `stealth.py` | Stealth/anti-bot configuration for the StealthyFetcher tier |
|
||
|
|
| `testbench/` | Live-site regression bench (own README) |
|
||
|
|
|
||
|
|
Contact extraction (`extract_contacts`) lives in `app/utils/crawl/contacts.py`
|
||
|
|
because non-proprietary callers use it too.
|
||
|
|
|
||
|
|
## The fetch ladder
|
||
|
|
|
||
|
|
Every crawl walks the same escalation ladder until one tier produces usable
|
||
|
|
content; callers see only the resulting `CrawlOutcome`, never the tier:
|
||
|
|
|
||
|
|
1. **AsyncFetcher** — static HTTP, TLS-impersonated, cheap. Handles most pages.
|
||
|
|
2. **DynamicFetcher** — full browser (thread), for JS-rendered content.
|
||
|
|
3. **StealthyFetcher** — patchright Chromium with anti-bot + Cloudflare
|
||
|
|
solving and captcha handling, the expensive last resort.
|
||
|
|
|
||
|
|
Success alone does not stop the ladder — two content-quality heuristics can
|
||
|
|
force escalation or re-extraction:
|
||
|
|
|
||
|
|
### Thin-page (JS-shell) escalation
|
||
|
|
|
||
|
|
A static fetch can "succeed" on an SPA that server-renders only a hero
|
||
|
|
paragraph and hydrates everything else client-side (a16z.com/team ships 4.2 MB
|
||
|
|
of HTML that extracts to 597 chars). A result is tagged `thin_static` and
|
||
|
|
escalated to the browser tier when **both** hold:
|
||
|
|
|
||
|
|
- raw HTML ≥ 1 MB (`_JS_SHELL_MIN_HTML_BYTES`), and
|
||
|
|
- extracted content < 2.5 KB (`_JS_SHELL_MAX_CONTENT_CHARS`).
|
||
|
|
|
||
|
|
Calibrated on live pages: true shells shipped ≥ 3.4 MB with < 0.05 % text;
|
||
|
|
every healthy page was under ~650 KB. Semi-shells (~150 KB, e.g.
|
||
|
|
ycombinator.com/people) intentionally stay on static — their server-rendered
|
||
|
|
link records still carry the roster. Upgrade path: hydration-marker sniffing
|
||
|
|
instead of size thresholds.
|
||
|
|
|
||
|
|
### Lossy-extraction repair (currency-guarded)
|
||
|
|
|
||
|
|
Trafilatura sometimes drops structured content (pricing cards, tables). We
|
||
|
|
can't detect every loss, but currency amounts are a cheap, high-precision
|
||
|
|
tripwire: if the raw HTML's visible text contains a currency amount
|
||
|
|
(`_CURRENCY_AMOUNT_RE`) and the extracted markdown doesn't, re-extract with
|
||
|
|
`favor_recall=True`; if the amount is still missing, fall back to a sanitized
|
||
|
|
`markdownify` of the whole `<body>`.
|
||
|
|
|
||
|
|
## Link records and contacts
|
||
|
|
|
||
|
|
`url_policy.extract_link_records` returns categorized links with anchor-text
|
||
|
|
provenance — these records, not the markdown, are the primary source for
|
||
|
|
roster/directory answers (names survive in link records even when extraction
|
||
|
|
drops them). `extract_contacts` harvests emails, phones, and social profiles
|
||
|
|
country-agnostically (global social-host list, `unquote()` applied to
|
||
|
|
percent-encoded `mailto:`/`tel:` hrefs — both here and in `url_policy`).
|
||
|
|
|
||
|
|
## Multi-page crawls
|
||
|
|
|
||
|
|
`crawl_site` uses Scrapling's spider engine for the traversal only (frontier,
|
||
|
|
dedupe, same-site scope, `includeUrlPatterns`/`excludeUrlPatterns` regex
|
||
|
|
filtering); every fetch still goes through `crawl_url`, so the ladder, proxy
|
||
|
|
rotation, and captcha handling are reused unchanged. Each `CrawlPage` carries
|
||
|
|
provenance (depth, referrer).
|
||
|
|
|
||
|
|
## Agent tooling layer (outside this package)
|
||
|
|
|
||
|
|
- The main chat agent has `scrape_webpage`; the `web_crawler` subagent has the
|
||
|
|
`web.crawl` capability (single URL or site mode).
|
||
|
|
- Tool outputs over the 40k-char cap (`RUN_OUTPUT_CHAR_CAP` in
|
||
|
|
`app/capabilities/core/runs.py`) are stored as JSONL runs; agents page them
|
||
|
|
with `read_run` (line paging + `char_offset` for giant single items), grep
|
||
|
|
them with `search_run` (returns excerpts around matches), and export them
|
||
|
|
deterministically with `export_run` (JSONL → CSV → workspace document, with
|
||
|
|
filtering and dedupe). Prompts live in
|
||
|
|
`app/agents/chat/multi_agent_chat/subagents/`.
|
||
|
|
|
||
|
|
## Session learnings (agent E2E hardening, Jul 2026)
|
||
|
|
|
||
|
|
Natural-language tasks run end-to-end through the multi-agent chat surfaced
|
||
|
|
these; each fix has a matching unit test:
|
||
|
|
|
||
|
|
1. **Search discovers — the crawler reads.** The agent initially summarized
|
||
|
|
from SERP snippets instead of crawling the pages it found. Routing guidance
|
||
|
|
(`main_agent/system_prompt/prompts/routing.md`) now tells it to crawl every
|
||
|
|
URL whose full content would improve the answer, executing bounded fan-out
|
||
|
|
without asking permission.
|
||
|
|
2. **Success alone is not enough** — content-quality tripwires (thin-page,
|
||
|
|
currency-loss) must gate the ladder, because a "successful" fetch can carry
|
||
|
|
an empty shell or a lossy extraction. Tests:
|
||
|
|
`tests/unit/proprietary/web_crawler/test_connector.py`.
|
||
|
|
3. **Full datasets become files, not chat.** LLMs are bad data pipes:
|
||
|
|
transcribing a 486-row roster through the model loses rows and burns
|
||
|
|
tokens. `export_run` converts the stored run to CSV in code and saves it to
|
||
|
|
the workspace KB. Tests: `tests/unit/capabilities/test_run_truncation.py`.
|
||
|
|
4. **Truncation needs an escape hatch the model will actually use.** Large
|
||
|
|
items defeated line-based paging until `read_run` grew `char_offset` and
|
||
|
|
`search_run` grew match excerpts; subagent prompts explicitly list the
|
||
|
|
readers and forbid re-running tools to "see more".
|
||
|
|
5. **Shared budgets starve precise queries.** In the Reddit scraper, one noisy
|
||
|
|
search consumed the whole `maxItems` cap before precise phrasings returned;
|
||
|
|
the fix fair-shares the budget across concurrent searches and de-dupes
|
||
|
|
across them (`tests/unit/platforms/reddit/test_search_budget.py`). The same
|
||
|
|
failure shape applies to any multi-query fan-out with a shared collector cap.
|
||
|
|
6. **Subagents must not hand back work they can do.** The universal output
|
||
|
|
contract (`subagents/shared/snippets/output_contract_base.md`) now requires:
|
||
|
|
if `next_step` is a call to one of the subagent's own tools (paging a run,
|
||
|
|
re-running with better parameters), execute it instead of returning
|
||
|
|
`partial`.
|
||
|
|
7. **Sizing caps to the ask.** When a task requests N items, tool caps
|
||
|
|
(`max_items`, findings limits in output contracts) must be set above N or
|
||
|
|
the task is unwinnable by construction; prompts now say so.
|