Wire captchatools as the StealthyFetcher-tier page_action to detect, harvest
(egressing from the crawl's own proxy IP), inject, and submit reCAPTCHA v2/v3
and hCaptcha tokens. Opt-in and off by default (zero attempts, zero cost).
Licensing split:
- Apache-2 app/utils/captcha/ holds the generic, vendor-agnostic config
(CaptchaConfig + captcha_enabled() = flag AND key present).
- Proprietary app/proprietary/web_crawler/captcha.py holds the bypass logic
(detect/harvest/inject) plus a process-wide solver latch that halts solving
on unrecoverable errors (no balance / bad key).
Crawler: CrawlOutcome gains captcha_attempts/captcha_solved, surfaced via a
per-call captcha_state dict threaded crawl_url -> _crawl_with_stealthy(_sync)
and stamped onto every stealth terminal outcome. The stealth tier captures the
proxy once and reuses it for both the fetch and the solver (IP-coherence).
Billing: WebCrawlCreditService gains captcha_billing_enabled,
captcha_solves_to_micros, charge_captcha, and a generic check_balance, sharing
a single _apply_debit path. The indexer accumulates attempts (even on failed
crawls), runs a combined crawl+captcha pre-flight, and posts a per-attempt
owner charge as usage_type="web_crawl_captcha". The captcha worst-case is only
reserved when solving is actually enabled, so a solving-off deployment is never
blocked for captcha that can never run. Both chat scrape tools fold attempts
into the current turn before the success/fail branch.
Fully config-driven prices; no migration. New unit tests cover the config,
factory (detection/latch/timeout/cap), credit service, indexer wiring, and the
chat fold.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add a config-driven crawl meter (1 USD / 1000 successes by default, retunable
via WEB_CRAWL_MICROS_PER_SUCCESS) mirroring EtlCreditService:
- WebCrawlCreditService (gate -> pre-check -> post-charge) on the unified wallet
- webcrawler indexer: pre-flight credit block + post-charge of the workspace
owner per success, with a web_crawl TokenUsage audit row
- chat scrape_webpage (main + research): fold each successful scrape into the
active turn's token accumulator so it settles with the chat turn
- WEB_CRAWL_CREDIT_BILLING_ENABLED / WEB_CRAWL_MICROS_PER_SUCCESS config + env
- unit tests (service) + integration tests (indexer wiring)
- scripts/e2e_phase3_crawl_billing.py: manual functional e2e (3a/3b/3c)
Co-authored-by: Cursor <cursoragent@cursor.com>
Add CustomProxyProvider (single endpoint or rotating pool via Scrapling ProxyRotator), registered as 'custom' alongside anonymous_proxies and selectable via PROXY_PROVIDER. Adds is_pool_backed to the ProxyProvider ABC + a zero-arg package helper. The web crawler does a bounded one-per-tier is_proxy_error rotation-retry gated on is_pool_backed() (single-endpoint providers no-op). Config/.env.example gain CUSTOM_PROXY_URL(S). Zero-arg getter contract unchanged for all consumers. Documents the proprietary boundary test (generic proxy infra stays Apache-2). Tests: provider, registry, crawler rotation (16).
Co-authored-by: Cursor <cursoragent@cursor.com>
- Standardized the web crawler to use Scrapling exclusively, removing Firecrawl entirely.
- Updated the crawler's location to `app/proprietary/web_crawler/connector.py` under a non-Apache-2 license boundary.
- Refactored the `WebCrawlerConnector` to eliminate the Firecrawl API key dependency, simplifying the interface for crawling URLs.
- Adjusted related components to accommodate the new structure and ensure successful crawl outcomes are properly handled.
- Updated documentation to reflect these changes and the new implementation status.
Align the .env.example operational runbook with the renamed Redis key
surfsense:spawn_paused:<workspace_id> (carve-out 3). All other carve-out
decisions are applied inline: KEEP enum values 'SEARCH_SPACE', Celery task names,
OTel/metric key "search_space.id", and historical alembic migrations; RENAME the
Redis/event/LangGraph keys and the default seed name "My Workspace".
Apply the same rename to surfsense_backend/tests: workspace_id fields/vars,
Workspace* classes/schemas, table name searchspaces -> workspaces in raw SQL,
and the API URL spellings -> /workspaces. Preserves the carve-out wire literals
tests assert (Celery task names, OTel key "search_space.id").
Hard cutover of the HTTP surface: collapse the three legacy spellings
(/searchspaces, /search-spaces, /search-space/{id}) onto canonical
/workspaces/{workspace_id}/..., rename the gateway field-setter sub-actions to
singular /workspace, rename search_spaces_routes.py -> workspaces_routes.py and the
search_spaces_router -> workspaces_router include, and flip search_space_id ->
workspace_id in bodies/params. No alias routers: the old URLs now 404 by design.
Full app constructs with zero legacy path spellings.
Rename schemas/search_space.py -> schemas/workspace.py, the SearchSpace* classes
-> Workspace* (incl. UserSearchSpaceAccess -> UserWorkspaceAccess), and the
search_space_id / search_space_name fields -> workspace_id / workspace_name across
all schema modules. Under hard cutover the serialized JSON keys change outright
(no alias). Re-exports in schemas/__init__.py updated.
Flip search_space_id -> workspace_id and the workspace relationship/back_populates
in the automations, file_storage, notifications, and podcasts persistence models,
and drop the Phase 1 Column("workspace_id", ...) shim. Full SQLAlchemy mapper
configuration now passes (db.py + satellites consistent).
Flip the symbolic name in db.py: classes SearchSpace/Role/Membership/Invite ->
Workspace*, attributes search_space_id -> workspace_id (and owner_*), relationship
attrs + back_populates pairs, and drop the Phase 1 Column("workspace_id", ...) shim
now that attribute name == column name. Enum values 'SEARCH_SPACE' and the
SearchSourceConnector class are intentionally untouched (carve-outs).
Part of the atomic SearchSpace -> Workspace rename; the suite goes green only once
Waves B-F land (no half-renamed steady state).
Trim the header to a one-line summary and relocate the "never raw DROP/CREATE
PUBLICATION" invariant (bug #1355, migration 116) to the apply_publication call
site where a refactor would actually occur. Comment/docstring only; no behavior
change.
- Increased ACCESS_TOKEN_LIFETIME_SECONDS from 30 minutes to 60 minutes for improved session duration.
- Introduced getRequestHeadersWithCurrentDesktopAuth function to streamline authorization header management across fetch requests in the chat page.
Phase 1 (rename DB) commit 3: single Alembic migration (down_revision=169)
that physically renames the SearchSpace schema to WorkSpace to match the
already-shimmed ORM. Renames 4 tables, 24 columns
(search_space_id->workspace_id, owner_search_space_id->owner_workspace_id),
37 named/auto FK/PK/unique constraints, 25 indexes, and 4 sequences. Guarded
forms (ALTER ... IF EXISTS / DO-block RENAME CONSTRAINT) tolerate
runtime-created and absent objects.
Zero publication is reconciled via the blessed path: neutralize the four
column-list tables (documents/automations/new_chat_threads/podcasts) from the
publication, rename, then apply_publication() re-emits SET TABLE with the new
workspace_id lists (no raw DROP/CREATE PUBLICATION, per migration 116).
Downgrade fully reverses the renames and restores the old search_space_id
publication shape via a hardcoded SET TABLE, never importing the live module.
Targets the existing-deployment upgrade path (169 -> 170). The from-scratch
alembic path is a separate, pre-existing concern (0_initial uses create_all).
Verified on a realistic legacy@169 DB (with publication): upgrade clean,
zero_publication --verify passes, autogenerate EMPTY on a create_all DB,
and upgrade->downgrade->upgrade round-trips (old shape + old publication
restored on downgrade). Single alembic head = 170.
Phase 1 (rename DB) commit 2: flip the search_space_id entries in the
canonical Zero publication column lists (documents, automations,
new_chat_threads, podcasts) to the renamed physical column workspace_id,
so ALTER PUBLICATION ... SET TABLE matches the post-rename schema.
No test references the publication shape; functional verification via
`python -m app.zero_publication --verify` runs after migration 170
builds/renames the publication against a live DB.
- Changed default AUTH_TYPE in backend configuration to "LOCAL".
- Updated frontend environment configuration to reflect the new default for packaged clients.
- Adjusted runtime authentication resolution to use "LOCAL" as the fallback value.
Phase 1 (rename DB) commit 1i: rename the SearchSpace table to workspaces
and flip all 24 inbound FK target strings (20 in db.py + the 4 satellite
models), plus the raw-SQL searchspaces references in the obsidian and
google-unification integration fixtures.
This completes the ORM half of Phase 1. The SearchSpace class name,
relationship/back_populates attribute names, and the /searchspaces API
route URLs are intentionally left for Phase 2.
Verified: unit 2375 passed/1 skip, integration 346 passed (baseline
parity); create_all builds every table with the workspaces name and all
FKs resolving to workspaces.id.
Phase 1 (rename DB) commit 1h: rename the roles table and flip both
inbound FKs (memberships.role_id, invites.role_id), plus the matching
raw-SQL cleanup in the obsidian plugin test.
Phase 1 (rename DB) commit 1g: rename the memberships table (no inbound
FKs reference it) and update the matching raw-SQL cleanup in the obsidian
plugin test. Relationship attribute names stay until Phase 2.
Phase 1 (rename DB) commit 1f: rename the invites table and flip the one
inbound FK (memberships.invited_by_invite_id). The invites row's own
role_id -> search_space_roles FK is left for the roles rename commit.
Phase 1 (rename DB) commit 1d: attribute->physical column shim on the
notifications satellite model, plus the inbox-list index column-ref and
its name aligned to workspace (ix_notifications_user_workspace_created).
Phase 1 (rename DB) commit 1b: same attribute->physical column shim as
1a, applied to the document_files satellite model. ORM attribute name
unchanged; FK target string left for the searchspaces rename commit.
Phase 1 (rename DB) commit 1a: shim every search_space_id /
owner_search_space_id column to the renamed physical column
(workspace_id / owner_workspace_id) via Column("workspace_id", ...),
keeping the ORM attribute name unchanged so existing callers are
untouched. Flip all Table-level references that resolve by column key
(inner __table_args__ strings, the ck_connections_scope_owner CHECK
text, and the _INDEX_DEFINITIONS runtime DDL) plus the searchspace-named
constraint/index names to workspace_id. Update the three raw-SQL test
fixtures that referenced the physical column.
Note: SQLAlchemy defaults a column's key to its name, so passing an
explicit "workspace_id" name moves the Table.c key to workspace_id;
Table-level string refs must therefore change in this phase (the
opposite of the original plan's finding 1). The ORM attribute, the
SearchSpace class, relationships, table names, and FK target strings are
intentionally left for later commits/Phase 2.
Verified: unit 2375 passed/1 skip, integration 346 passed (baseline
parity); create_all builds every table with workspace_id.
Migration 168 used bare add_column/drop_column, so `alembic upgrade head`
from an empty DB failed at 168 ("column revoked_at already exists"): the
0_initial_schema migration bootstraps via Base.metadata.create_all (the live
ORM shape), which already includes the hardened refresh_tokens columns, and
168's unguarded ALTERs then collided.
Guard every step (ADD COLUMN IF NOT EXISTS, DROP COLUMN IF EXISTS, existence-
checked backfill) so 168 no-ops on already-hardened (create_all) databases
while still fully transforming legacy is_revoked-shape databases. Matches the
guarded style already used by migrations 92/149/169. Net schema effect is
unchanged. Restores a clean from-scratch `alembic upgrade head` to head.
- Added logging for JSON parsing failures in structured output to aid in debugging.
- Updated the TranscriptTurn model to ignore unknown keys, improving flexibility for future model variations.
- Updated main-agent middleware to clarify that both filesystem reads/writes and knowledge-base retrieval are handled by the `knowledge_base` subagent.
- Introduced `_forward_mention_pins` function to carry `@`-mention pins into subagent state.
- Revised system prompts to reflect the new retrieval method and ensure proper citation handling.
- Removed the `search_knowledge_base` tool and its related tests, consolidating functionality under the `task` tool.
- Enhanced documentation to guide usage of the new retrieval approach and citation practices.
The document saving logic used a hardcoded character slice ([:4000]) for
the document summary content fed to the embedder. For UTF-8 documents
(e.g., Arabic, Chinese, Japanese), characters above U+007F take multiple
bytes but count as 1 character in a slice, potentially producing text
that exceeds the token limit of the embedding model.
Replaced the arbitrary slice with runcate_for_embedding(), which safely
bounds the text using the embedding model's actual tokenizer.
When a codeBlock is nested inside an indented structure like a
bulletListItem, the \_render_block\ function prepends the block's
indentation \prefix\ to every line.
However, for codeBlock elements, only the fence markers (the opening
and closing \\\) should carry the block indentation. Interior code
lines must not have the prefix prepended, because markdown parsers
treat leading spaces inside a code fence as part of the code content.
This fix removes the prefix from interior code lines to prevent
code snippets stored in notes from gaining spurious whitespace.
When the Unstructured API returns a Table element without text_as_html
in its metadata (e.g. local install or free-tier API), the lambda was
raising KeyError: 'text_as_html', crashing the entire document
indexing pipeline for any file containing tables.
Guard the key access with .get() and fall back to the plain extracted
text content (x) so the pipeline continues and the table content is
still indexed, just without HTML formatting.
The RAG/citation/context redesign in ADR 0001 is implemented and validated
(KB + web on the unified [n] citation spine, pull-based retrieval, eager path
retired). Drop the ADR and the one stale docstring reference to it.
The pull-based KB design (on-demand search_knowledge_base tool + pre-injected
workspace tree) fully replaced the old eager retrieval path. Remove its last
remnants:
- Delete KnowledgePriorityMiddleware (knowledge_search.py) and its tests.
- Drop the kb_priority state field + reducer default; trim
KbContextProjectionMiddleware to project only workspace_tree_text.
- Remove the now-dead feature flags enable_kb_priority_preinjection and
enable_kb_planner_runnable across backend (flags, route schema, tests,
env examples) and frontend (settings toggle, zod schema).
- Scrub <priority_documents> and stale KnowledgePriorityMiddleware references
from prompts, docstrings, and the ADR.
No functional change: nothing wrote kb_priority and neither flag gated live
behavior after the cutover. Full backend suite green (pre-existing unrelated
failures aside).
The legacy system_prompt_composer fragments and its default_system_instructions
wrapper were no longer referenced by any live code path (the main-agent prompt
builder owns composition now). Delete the whole orphaned tree and its test.
The model frequently writes citations glued to the preceding word
(docs[17]); the (?<!\w) lookbehind (added to dodge arr[1] array indexing)
silently skipped these, leaving raw [n] that fails to render and reads
like array access. Drop the lookbehind so glued citations resolve; genuine
code/array syntax stays protected by the existing code-region carve-out and
unresolved ordinals still drop harmlessly.
