bdacea8b6e
Wire captchatools as the StealthyFetcher-tier page_action to detect, harvest (egressing from the crawl's own proxy IP), inject, and submit reCAPTCHA v2/v3 and hCaptcha tokens. Opt-in and off by default (zero attempts, zero cost). Licensing split: - Apache-2 app/utils/captcha/ holds the generic, vendor-agnostic config (CaptchaConfig + captcha_enabled() = flag AND key present). - Proprietary app/proprietary/web_crawler/captcha.py holds the bypass logic (detect/harvest/inject) plus a process-wide solver latch that halts solving on unrecoverable errors (no balance / bad key). Crawler: CrawlOutcome gains captcha_attempts/captcha_solved, surfaced via a per-call captcha_state dict threaded crawl_url -> _crawl_with_stealthy(_sync) and stamped onto every stealth terminal outcome. The stealth tier captures the proxy once and reuses it for both the fetch and the solver (IP-coherence). Billing: WebCrawlCreditService gains captcha_billing_enabled, captcha_solves_to_micros, charge_captcha, and a generic check_balance, sharing a single _apply_debit path. The indexer accumulates attempts (even on failed crawls), runs a combined crawl+captcha pre-flight, and posts a per-attempt owner charge as usage_type="web_crawl_captcha". The captcha worst-case is only reserved when solving is actually enabled, so a solving-off deployment is never blocked for captcha that can never run. Both chat scrape tools fold attempts into the current turn before the success/fail branch. Fully config-driven prices; no migration. New unit tests cover the config, factory (detection/latch/timeout/cap), credit service, indexer wiring, and the chat fold. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|---|---|---|
| .. | ||
| e2e | ||
| fixtures | ||
| integration | ||
| unit | ||
| utils | ||
| __init__.py | ||
| conftest.py | ||
| README.md | ||
Tests
How the backend test suite is organized and the conventions to follow when adding tests.
Layout: type-first, module-mirrored
Tests are split by type at the top level, and each type mirrors the app/ module tree inside:
tests/
├── conftest.py # global fixtures + DATABASE_URL pinning
├── unit/ # pure logic: no DB, no app, no network
│ └── notifications/
│ ├── api/test_transform.py
│ └── service/
│ ├── messages/test_connector_indexing.py
│ └── test_metadata.py
└── integration/ # real PostgreSQL (pgvector)
├── conftest.py # async engine, transactional db_session, db_user, ...
└── notifications/
├── conftest.py # module-scoped fixtures (e.g. transactional client)
└── test_*_handler.py
To find a feature's tests, look under tests/<type>/<same path as app/>.
Unit vs integration
@pytest.mark.unit— pure, fast, no I/O. Test behavior through a public function's inputs/outputs.@pytest.mark.integration— requires a real database. Run withAUTH_TYPE=LOCAL.
Maximize logic covered by unit tests; keep integration tests for what genuinely needs the DB (persistence, SQL filters, scoping, HTTP wiring).
Principles
- Behavior, not implementation. Assert observable outputs (returned values, persisted rows, HTTP responses), never private helpers. Tests should survive a refactor.
- Functional core / imperative shell. Put pure decision logic in a side-effect-free module (e.g.
app/notifications/service/messages/) so it is unit-testable; keep the persistence shell thin and cover it with a few integration tests. - One responsibility per test file, mirroring the slice it covers.
- Mock only at system boundaries (external APIs, brokers), never internal collaborators. Prefer dependency overrides and the transactional
db_sessionover mocks.
Fixtures
conftest.py is scoped to its directory and below. Keep truly global fixtures in tests/conftest.py; put module-specific fixtures in that module's conftest.py so a DB fixture never loads for a pure unit test.
For API integration tests, override get_async_session and get_auth_context to ride the test's transactional db_session (see tests/integration/notifications/conftest.py): rows seeded in the test and rows read via the endpoint share one transaction that rolls back automatically.
Import mode
The suite uses --import-mode=importlib with pythonpath = ["."] (see pyproject.toml). This lets test files share basenames across modules (e.g. many test_api.py) without __init__.py boilerplate; new test directories do not need an __init__.py.
Running
# fast unit tests
uv run pytest -m unit
# integration (needs Postgres + pgvector)
AUTH_TYPE=LOCAL uv run pytest -m integration
# a single module's tests
uv run pytest tests/unit/notifications