- Added Business Source License 1.1 details to the proprietary LICENSE file.
- Clarified terms of use, including production use restrictions and transition to Apache License after four years.
- Updated README.md to reflect the new licensing structure and its implications for users.
This change ensures clear communication of licensing terms for proprietary components within the SurfSense project.
Move app/scrapers -> app/proprietary/scrapers/youtube to sit alongside the existing proprietary web_crawler/platforms namespace, updating all external imports (routes, tests, e2e script, README). Internal imports were relative so are unchanged.
Also parallelize playlist per-video resolution: page video ids sequentially, then resolve the heavy watch-page fetches concurrently via fan_out (~150 videos ~70s, down from a few minutes). Items stream in completion order; sort by the order field for playlist order.
Co-authored-by: Cursor <cursoragent@cursor.com>
Wire captchatools as the StealthyFetcher-tier page_action to detect, harvest
(egressing from the crawl's own proxy IP), inject, and submit reCAPTCHA v2/v3
and hCaptcha tokens. Opt-in and off by default (zero attempts, zero cost).
Licensing split:
- Apache-2 app/utils/captcha/ holds the generic, vendor-agnostic config
(CaptchaConfig + captcha_enabled() = flag AND key present).
- Proprietary app/proprietary/web_crawler/captcha.py holds the bypass logic
(detect/harvest/inject) plus a process-wide solver latch that halts solving
on unrecoverable errors (no balance / bad key).
Crawler: CrawlOutcome gains captcha_attempts/captcha_solved, surfaced via a
per-call captcha_state dict threaded crawl_url -> _crawl_with_stealthy(_sync)
and stamped onto every stealth terminal outcome. The stealth tier captures the
proxy once and reuses it for both the fetch and the solver (IP-coherence).
Billing: WebCrawlCreditService gains captcha_billing_enabled,
captcha_solves_to_micros, charge_captcha, and a generic check_balance, sharing
a single _apply_debit path. The indexer accumulates attempts (even on failed
crawls), runs a combined crawl+captcha pre-flight, and posts a per-attempt
owner charge as usage_type="web_crawl_captcha". The captcha worst-case is only
reserved when solving is actually enabled, so a solving-off deployment is never
blocked for captcha that can never run. Both chat scrape tools fold attempts
into the current turn before the success/fail branch.
Fully config-driven prices; no migration. New unit tests cover the config,
factory (detection/latch/timeout/cap), credit service, indexer wiring, and the
chat fold.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add a config-driven crawl meter (1 USD / 1000 successes by default, retunable
via WEB_CRAWL_MICROS_PER_SUCCESS) mirroring EtlCreditService:
- WebCrawlCreditService (gate -> pre-check -> post-charge) on the unified wallet
- webcrawler indexer: pre-flight credit block + post-charge of the workspace
owner per success, with a web_crawl TokenUsage audit row
- chat scrape_webpage (main + research): fold each successful scrape into the
active turn's token accumulator so it settles with the chat turn
- WEB_CRAWL_CREDIT_BILLING_ENABLED / WEB_CRAWL_MICROS_PER_SUCCESS config + env
- unit tests (service) + integration tests (indexer wiring)
- scripts/e2e_phase3_crawl_billing.py: manual functional e2e (3a/3b/3c)
Co-authored-by: Cursor <cursoragent@cursor.com>
Add CustomProxyProvider (single endpoint or rotating pool via Scrapling ProxyRotator), registered as 'custom' alongside anonymous_proxies and selectable via PROXY_PROVIDER. Adds is_pool_backed to the ProxyProvider ABC + a zero-arg package helper. The web crawler does a bounded one-per-tier is_proxy_error rotation-retry gated on is_pool_backed() (single-endpoint providers no-op). Config/.env.example gain CUSTOM_PROXY_URL(S). Zero-arg getter contract unchanged for all consumers. Documents the proprietary boundary test (generic proxy infra stays Apache-2). Tests: provider, registry, crawler rotation (16).
Co-authored-by: Cursor <cursoragent@cursor.com>
- Standardized the web crawler to use Scrapling exclusively, removing Firecrawl entirely.
- Updated the crawler's location to `app/proprietary/web_crawler/connector.py` under a non-Apache-2 license boundary.
- Refactored the `WebCrawlerConnector` to eliminate the Firecrawl API key dependency, simplifying the interface for crawling URLs.
- Adjusted related components to accommodate the new structure and ensure successful crawl outcomes are properly handled.
- Updated documentation to reflect these changes and the new implementation status.
Align the .env.example operational runbook with the renamed Redis key
surfsense:spawn_paused:<workspace_id> (carve-out 3). All other carve-out
decisions are applied inline: KEEP enum values 'SEARCH_SPACE', Celery task names,
OTel/metric key "search_space.id", and historical alembic migrations; RENAME the
Redis/event/LangGraph keys and the default seed name "My Workspace".
Apply the same rename to surfsense_backend/tests: workspace_id fields/vars,
Workspace* classes/schemas, table name searchspaces -> workspaces in raw SQL,
and the API URL spellings -> /workspaces. Preserves the carve-out wire literals
tests assert (Celery task names, OTel key "search_space.id").
Hard cutover of the HTTP surface: collapse the three legacy spellings
(/searchspaces, /search-spaces, /search-space/{id}) onto canonical
/workspaces/{workspace_id}/..., rename the gateway field-setter sub-actions to
singular /workspace, rename search_spaces_routes.py -> workspaces_routes.py and the
search_spaces_router -> workspaces_router include, and flip search_space_id ->
workspace_id in bodies/params. No alias routers: the old URLs now 404 by design.
Full app constructs with zero legacy path spellings.
Rename schemas/search_space.py -> schemas/workspace.py, the SearchSpace* classes
-> Workspace* (incl. UserSearchSpaceAccess -> UserWorkspaceAccess), and the
search_space_id / search_space_name fields -> workspace_id / workspace_name across
all schema modules. Under hard cutover the serialized JSON keys change outright
(no alias). Re-exports in schemas/__init__.py updated.
Flip search_space_id -> workspace_id and the workspace relationship/back_populates
in the automations, file_storage, notifications, and podcasts persistence models,
and drop the Phase 1 Column("workspace_id", ...) shim. Full SQLAlchemy mapper
configuration now passes (db.py + satellites consistent).
Flip the symbolic name in db.py: classes SearchSpace/Role/Membership/Invite ->
Workspace*, attributes search_space_id -> workspace_id (and owner_*), relationship
attrs + back_populates pairs, and drop the Phase 1 Column("workspace_id", ...) shim
now that attribute name == column name. Enum values 'SEARCH_SPACE' and the
SearchSourceConnector class are intentionally untouched (carve-outs).
Part of the atomic SearchSpace -> Workspace rename; the suite goes green only once
Waves B-F land (no half-renamed steady state).
Trim the header to a one-line summary and relocate the "never raw DROP/CREATE
PUBLICATION" invariant (bug #1355, migration 116) to the apply_publication call
site where a refactor would actually occur. Comment/docstring only; no behavior
change.
- Increased ACCESS_TOKEN_LIFETIME_SECONDS from 30 minutes to 60 minutes for improved session duration.
- Introduced getRequestHeadersWithCurrentDesktopAuth function to streamline authorization header management across fetch requests in the chat page.
Phase 1 (rename DB) commit 3: single Alembic migration (down_revision=169)
that physically renames the SearchSpace schema to WorkSpace to match the
already-shimmed ORM. Renames 4 tables, 24 columns
(search_space_id->workspace_id, owner_search_space_id->owner_workspace_id),
37 named/auto FK/PK/unique constraints, 25 indexes, and 4 sequences. Guarded
forms (ALTER ... IF EXISTS / DO-block RENAME CONSTRAINT) tolerate
runtime-created and absent objects.
Zero publication is reconciled via the blessed path: neutralize the four
column-list tables (documents/automations/new_chat_threads/podcasts) from the
publication, rename, then apply_publication() re-emits SET TABLE with the new
workspace_id lists (no raw DROP/CREATE PUBLICATION, per migration 116).
Downgrade fully reverses the renames and restores the old search_space_id
publication shape via a hardcoded SET TABLE, never importing the live module.
Targets the existing-deployment upgrade path (169 -> 170). The from-scratch
alembic path is a separate, pre-existing concern (0_initial uses create_all).
Verified on a realistic legacy@169 DB (with publication): upgrade clean,
zero_publication --verify passes, autogenerate EMPTY on a create_all DB,
and upgrade->downgrade->upgrade round-trips (old shape + old publication
restored on downgrade). Single alembic head = 170.
Phase 1 (rename DB) commit 2: flip the search_space_id entries in the
canonical Zero publication column lists (documents, automations,
new_chat_threads, podcasts) to the renamed physical column workspace_id,
so ALTER PUBLICATION ... SET TABLE matches the post-rename schema.
No test references the publication shape; functional verification via
`python -m app.zero_publication --verify` runs after migration 170
builds/renames the publication against a live DB.
- Changed default AUTH_TYPE in backend configuration to "LOCAL".
- Updated frontend environment configuration to reflect the new default for packaged clients.
- Adjusted runtime authentication resolution to use "LOCAL" as the fallback value.
Phase 1 (rename DB) commit 1i: rename the SearchSpace table to workspaces
and flip all 24 inbound FK target strings (20 in db.py + the 4 satellite
models), plus the raw-SQL searchspaces references in the obsidian and
google-unification integration fixtures.
This completes the ORM half of Phase 1. The SearchSpace class name,
relationship/back_populates attribute names, and the /searchspaces API
route URLs are intentionally left for Phase 2.
Verified: unit 2375 passed/1 skip, integration 346 passed (baseline
parity); create_all builds every table with the workspaces name and all
FKs resolving to workspaces.id.
Phase 1 (rename DB) commit 1h: rename the roles table and flip both
inbound FKs (memberships.role_id, invites.role_id), plus the matching
raw-SQL cleanup in the obsidian plugin test.
Phase 1 (rename DB) commit 1g: rename the memberships table (no inbound
FKs reference it) and update the matching raw-SQL cleanup in the obsidian
plugin test. Relationship attribute names stay until Phase 2.
Phase 1 (rename DB) commit 1f: rename the invites table and flip the one
inbound FK (memberships.invited_by_invite_id). The invites row's own
role_id -> search_space_roles FK is left for the roles rename commit.
Phase 1 (rename DB) commit 1d: attribute->physical column shim on the
notifications satellite model, plus the inbox-list index column-ref and
its name aligned to workspace (ix_notifications_user_workspace_created).
Phase 1 (rename DB) commit 1b: same attribute->physical column shim as
1a, applied to the document_files satellite model. ORM attribute name
unchanged; FK target string left for the searchspaces rename commit.
Phase 1 (rename DB) commit 1a: shim every search_space_id /
owner_search_space_id column to the renamed physical column
(workspace_id / owner_workspace_id) via Column("workspace_id", ...),
keeping the ORM attribute name unchanged so existing callers are
untouched. Flip all Table-level references that resolve by column key
(inner __table_args__ strings, the ck_connections_scope_owner CHECK
text, and the _INDEX_DEFINITIONS runtime DDL) plus the searchspace-named
constraint/index names to workspace_id. Update the three raw-SQL test
fixtures that referenced the physical column.
Note: SQLAlchemy defaults a column's key to its name, so passing an
explicit "workspace_id" name moves the Table.c key to workspace_id;
Table-level string refs must therefore change in this phase (the
opposite of the original plan's finding 1). The ORM attribute, the
SearchSpace class, relationships, table names, and FK target strings are
intentionally left for later commits/Phase 2.
Verified: unit 2375 passed/1 skip, integration 346 passed (baseline
parity); create_all builds every table with workspace_id.
