Add CustomProxyProvider (single endpoint or rotating pool via Scrapling ProxyRotator), registered as 'custom' alongside anonymous_proxies and selectable via PROXY_PROVIDER. Adds is_pool_backed to the ProxyProvider ABC + a zero-arg package helper. The web crawler does a bounded one-per-tier is_proxy_error rotation-retry gated on is_pool_backed() (single-endpoint providers no-op). Config/.env.example gain CUSTOM_PROXY_URL(S). Zero-arg getter contract unchanged for all consumers. Documents the proprietary boundary test (generic proxy infra stays Apache-2). Tests: provider, registry, crawler rotation (16).
Co-authored-by: Cursor <cursoragent@cursor.com>
- Standardized the web crawler to use Scrapling exclusively, removing Firecrawl entirely.
- Updated the crawler's location to `app/proprietary/web_crawler/connector.py` under a non-Apache-2 license boundary.
- Refactored the `WebCrawlerConnector` to eliminate the Firecrawl API key dependency, simplifying the interface for crawling URLs.
- Adjusted related components to accommodate the new structure and ensure successful crawl outcomes are properly handled.
- Updated documentation to reflect these changes and the new implementation status.
The chat composers submit on Enter without checking whether an IME
composition is active. For Japanese/Chinese/Korean input, pressing Enter
to confirm a conversion fires the submit handler and sends a half-typed
message. Guard the Enter handlers with e.nativeEvent.isComposing in the
main chat editor, the anonymous free composer, and the mention/prompt
picker key navigation.
Add a top status banner (SHIPPED, branch/PR, phase tip commit) and an
Implementation record (deviations, carve-outs as-shipped, re-runnable
verification gates, deploy caveats) to 02-rename-backend.md.
Add a top status banner (SHIPPED, branch/PR, phase tip commit) and an
Implementation record (deviations, carve-outs as-shipped, re-runnable
verification gates with last results) to 01-rename-db.md.
Align the .env.example operational runbook with the renamed Redis key
surfsense:spawn_paused:<workspace_id> (carve-out 3). All other carve-out
decisions are applied inline: KEEP enum values 'SEARCH_SPACE', Celery task names,
OTel/metric key "search_space.id", and historical alembic migrations; RENAME the
Redis/event/LangGraph keys and the default seed name "My Workspace".
Apply the same rename to surfsense_backend/tests: workspace_id fields/vars,
Workspace* classes/schemas, table name searchspaces -> workspaces in raw SQL,
and the API URL spellings -> /workspaces. Preserves the carve-out wire literals
tests assert (Celery task names, OTel key "search_space.id").
Hard cutover of the HTTP surface: collapse the three legacy spellings
(/searchspaces, /search-spaces, /search-space/{id}) onto canonical
/workspaces/{workspace_id}/..., rename the gateway field-setter sub-actions to
singular /workspace, rename search_spaces_routes.py -> workspaces_routes.py and the
search_spaces_router -> workspaces_router include, and flip search_space_id ->
workspace_id in bodies/params. No alias routers: the old URLs now 404 by design.
Full app constructs with zero legacy path spellings.
Rename schemas/search_space.py -> schemas/workspace.py, the SearchSpace* classes
-> Workspace* (incl. UserSearchSpaceAccess -> UserWorkspaceAccess), and the
search_space_id / search_space_name fields -> workspace_id / workspace_name across
all schema modules. Under hard cutover the serialized JSON keys change outright
(no alias). Re-exports in schemas/__init__.py updated.
Flip search_space_id -> workspace_id and the workspace relationship/back_populates
in the automations, file_storage, notifications, and podcasts persistence models,
and drop the Phase 1 Column("workspace_id", ...) shim. Full SQLAlchemy mapper
configuration now passes (db.py + satellites consistent).
Flip the symbolic name in db.py: classes SearchSpace/Role/Membership/Invite ->
Workspace*, attributes search_space_id -> workspace_id (and owner_*), relationship
attrs + back_populates pairs, and drop the Phase 1 Column("workspace_id", ...) shim
now that attribute name == column name. Enum values 'SEARCH_SPACE' and the
SearchSourceConnector class are intentionally untouched (carve-outs).
Part of the atomic SearchSpace -> Workspace rename; the suite goes green only once
Waves B-F land (no half-renamed steady state).
Trim the header to a one-line summary and relocate the "never raw DROP/CREATE
PUBLICATION" invariant (bug #1355, migration 116) to the apply_publication call
site where a refactor would actually occur. Comment/docstring only; no behavior
change.
- Increased ACCESS_TOKEN_LIFETIME_SECONDS from 30 minutes to 60 minutes for improved session duration.
- Introduced getRequestHeadersWithCurrentDesktopAuth function to streamline authorization header management across fetch requests in the chat page.
- Introduced fetchSession function to streamline session fetching logic.
- Updated useSession to handle 401 errors by refreshing the session when necessary.
- Modified getDesktopAccessToken to accept options for forced token refresh, improving desktop authentication flow.
- Refactored fetchZeroContext to include a buildHeaders function for better header management.
- Added a request function to handle 401 errors and refresh sessions as needed.
- Improved overall session management for desktop authentication.
- Implemented session refresh logic in the fetchWithTurnCancellingRetry function to handle 401 errors more gracefully.
- Added a new import for refreshSession utility to facilitate session renewal.
Phase 1 (rename DB) commit 3: single Alembic migration (down_revision=169)
that physically renames the SearchSpace schema to WorkSpace to match the
already-shimmed ORM. Renames 4 tables, 24 columns
(search_space_id->workspace_id, owner_search_space_id->owner_workspace_id),
37 named/auto FK/PK/unique constraints, 25 indexes, and 4 sequences. Guarded
forms (ALTER ... IF EXISTS / DO-block RENAME CONSTRAINT) tolerate
runtime-created and absent objects.
Zero publication is reconciled via the blessed path: neutralize the four
column-list tables (documents/automations/new_chat_threads/podcasts) from the
publication, rename, then apply_publication() re-emits SET TABLE with the new
workspace_id lists (no raw DROP/CREATE PUBLICATION, per migration 116).
Downgrade fully reverses the renames and restores the old search_space_id
publication shape via a hardcoded SET TABLE, never importing the live module.
Targets the existing-deployment upgrade path (169 -> 170). The from-scratch
alembic path is a separate, pre-existing concern (0_initial uses create_all).
Verified on a realistic legacy@169 DB (with publication): upgrade clean,
zero_publication --verify passes, autogenerate EMPTY on a create_all DB,
and upgrade->downgrade->upgrade round-trips (old shape + old publication
restored on downgrade). Single alembic head = 170.
Phase 1 (rename DB) commit 2: flip the search_space_id entries in the
canonical Zero publication column lists (documents, automations,
new_chat_threads, podcasts) to the renamed physical column workspace_id,
so ALTER PUBLICATION ... SET TABLE matches the post-rename schema.
No test references the publication shape; functional verification via
`python -m app.zero_publication --verify` runs after migration 170
builds/renames the publication against a live DB.
- Changed default AUTH_TYPE in backend configuration to "LOCAL".
- Updated frontend environment configuration to reflect the new default for packaged clients.
- Adjusted runtime authentication resolution to use "LOCAL" as the fallback value.
Phase 1 (rename DB) commit 1i: rename the SearchSpace table to workspaces
and flip all 24 inbound FK target strings (20 in db.py + the 4 satellite
models), plus the raw-SQL searchspaces references in the obsidian and
google-unification integration fixtures.
This completes the ORM half of Phase 1. The SearchSpace class name,
relationship/back_populates attribute names, and the /searchspaces API
route URLs are intentionally left for Phase 2.
Verified: unit 2375 passed/1 skip, integration 346 passed (baseline
parity); create_all builds every table with the workspaces name and all
FKs resolving to workspaces.id.
