Commit graph

827 commits

Author SHA1 Message Date
DESKTOP-RTLN3BA\$punk
4a727a2c57 chore: linting 2026-07-24 14:30:46 -07:00
Anish Sarkar
a2d5467023 Merge remote-tracking branch 'upstream/dev' into fix/models 2026-07-25 00:32:12 +05:30
DESKTOP-RTLN3BA\$punk
dbedf0cfa5 feat(backend): integrate PostHog analytics for enhanced observability
- Added PostHog configuration options to .env.example files for both Docker and Surfsense backend.
- Introduced PostHog dependency in pyproject.toml.
- Implemented analytics middleware to capture various events across the application, including user authentication, automation runs, and API requests.
- Enhanced existing routes and services to emit analytics events, providing insights into user interactions and system performance.
- Ensured graceful shutdown of analytics clients in worker processes and application lifecycles.
2026-07-22 22:16:28 -07:00
Rohan Verma
110102c3c0
Merge pull request #1609 from AnishSarkar22/fix/tabs
refactor(tabs): pointer-based tabs with live react-query resolution, workspace limits, and UI polish
2026-07-22 14:53:29 -07:00
Rohan Verma
08d431454f
Merge pull request #1617 from CREDO23/feature-okf
[Feat] Serve the knowledge base in Open Knowledge Format (OKF)
2026-07-21 15:39:59 -07:00
Anish Sarkar
9fd5d3571c Merge remote-tracking branch 'upstream/dev' into fix/tabs 2026-07-22 00:02:26 +05:30
CREDO23
6a72051e06 content-negotiate OKF on GET document 2026-07-21 19:58:52 +02:00
CREDO23
ca317c4686 Merge remote-tracking branch 'upstream/dev' into feature-walmart-scraper
# Conflicts:
#	docker/.env.example
#	surfsense_backend/app/capabilities/core/billing.py
#	surfsense_backend/app/capabilities/core/types.py
#	surfsense_backend/app/config/__init__.py
#	surfsense_mcp/mcp_server/features/scrapers/__init__.py
#	surfsense_web/content/docs/connectors/index.mdx
#	surfsense_web/content/docs/connectors/native/index.mdx
#	surfsense_web/content/docs/how-to/mcp-server.mdx
2026-07-21 02:42:41 +02:00
CREDO23
a3f8a7e741 feat(walmart): register routes and MCP tools 2026-07-19 08:19:22 +02:00
CREDO23
91aa265afb Merge remote-tracking branch 'upstream/dev' into feature-indeed-jobs-scraper
# Conflicts:
#	README.es.md
#	README.hi.md
#	README.md
#	README.pt-BR.md
#	README.zh-CN.md
#	surfsense_backend/tests/unit/capabilities/google_maps/test_registry.py
#	surfsense_backend/tests/unit/capabilities/reddit/test_registry.py
#	surfsense_backend/tests/unit/capabilities/youtube/test_registry.py
#	surfsense_mcp/mcp_server/features/scrapers/__init__.py
#	surfsense_web/content/docs/connectors/index.mdx
#	surfsense_web/content/docs/connectors/native/index.mdx
#	surfsense_web/content/docs/connectors/native/meta.json
#	surfsense_web/content/docs/how-to/mcp-server.mdx
#	surfsense_web/lib/connectors-marketing/index.ts
#	surfsense_web/lib/playground/catalog.ts
2026-07-18 22:11:30 +02:00
DESKTOP-RTLN3BA\$punk
c0ebb62fb2 chore: linting 2026-07-17 14:39:39 -07:00
Anish Sarkar
38b784fbac feat(workspaces): implement workspace limits feature with backend integration and UI updates 2026-07-16 15:44:49 +05:30
Rohan Verma
d1d839a491
Merge pull request #1604 from AnishSarkar22/feat/amazon-scraper
feat(amazon): add public multi-marketplace product scraping capability
2026-07-15 16:29:03 -07:00
Anish Sarkar
21a7a0a0b0 feat(amazon): register scrape capability 2026-07-15 01:57:32 +05:30
CREDO23
ad49e9004e feat: register indeed namespace on routes 2026-07-14 21:49:23 +02:00
Anish Sarkar
2f048d87bc feat(model-connections): stamp own-model readiness and derive setup stage 2026-07-14 00:14:51 +05:30
Anish Sarkar
561bc76a9a feat(workspaces): return llm_setup on workspace create 2026-07-12 20:29:31 +05:30
Anish Sarkar
ff676519ea feat(model-connections): derive workspace LLM setup status server-side 2026-07-12 20:29:26 +05:30
Anish Sarkar
e38ca19b18 Merge remote-tracking branch 'upstream/dev' into feat/instagram-scraper 2026-07-11 04:31:11 +05:30
Anish Sarkar
929efe8152 feat(instagram): register instagram.* capability namespace 2026-07-09 16:01:49 +05:30
CREDO23
2943d8b23c feat(tiktok): tiktok.scrape capability + billing wire-up 2026-07-08 18:21:44 +02:00
Anish Sarkar
cda232101f feat(workspace): remove AI file sorting feature and related components 2026-07-06 10:50:20 +05:30
CREDO23
800c0e55fd feat(backend): expose semantic document search endpoint 2026-07-06 02:29:19 +02:00
DESKTOP-RTLN3BA\$punk
ab747e7a49 feat(agents): consolidate connectors under mcp_discovery; route web search through google_search
MCP consolidation:
- Route all MCP-capable connectors (Slack, Jira, Linear, ClickUp, Airtable,
  Notion, Confluence, interim Gmail/Calendar, custom MCP) through a single
  `mcp_discovery` subagent. Drive/OneDrive/Dropbox stay native to enrich the KB.
- Deprecate Discord/Teams/Luma: no viable official MCP server.

Google-only web search:
- Remove the main-agent `web_search` tool and the SearXNG platform service;
  all public web search now flows through the `google_search` subagent via task().
- Deprecate the Tavily/SearXNG/Linkup/Baidu search connectors (HTTP 410 on
  create, "Deprecated" badge); guide heavy users to the custom MCP connector.
- Remove web search from anonymous chat (pure Q&A).
- Tear SearXNG out of docker compose + install scripts; drop tavily-python
  and linkup-sdk deps and their config/env vars.

Fix:
- metrics._package_version() now swallows any metadata lookup failure. A
  malformed editable-install distribution with no `Version` field raised
  KeyError deep in importlib.metadata, and since it runs on every
  record_subagent_invoke_duration call it was crashing every task()
  delegation. Verified end-to-end against live GPT-5.4.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-04 21:06:04 -07:00
DESKTOP-RTLN3BA\$punk
ff2e5f390f feat(reddit): implement Reddit scraping subagent and associated capabilities
- Added a new `reddit` subagent to scrape structured data from Reddit posts, comments, and users.
- Introduced `reddit.scrape` capability for fetching data using URLs and search queries.
- Implemented tools for scraping and parsing Reddit data, including handling pagination and rate limits.
- Created input/output models for the Reddit scraper to define request and response structures.
- Added documentation for the new Reddit scraping functionality and its usage.
- Integrated the Reddit subagent into the existing multi-agent chat framework.
2026-07-04 17:31:11 -07:00
CREDO23
a6332e23e9 feat(google_search): wire capability into REST door 2026-07-04 20:49:58 +02:00
CREDO23
0844dd581f refactor(routes): drop duplicate /google-maps/* routes (use capability doors) 2026-07-03 17:56:59 +02:00
CREDO23
f31823f765 feat(google_maps): add scrape + reviews capability verbs 2026-07-03 17:56:43 +02:00
CREDO23
d5d673384f Merge upstream/ci_mvp (google maps scrapers) into feature-ci_phase-4-7
Resolve conflicts against the new native google-maps actor + repo-wide
ruff-format pass:
- Keep legacy webcrawler KB indexer + its test deleted (modify/delete).
- test_validators: keep WEBCRAWLER case removed (validator gone).
- test_fetch_resilience: keep platforms.youtube import path (our reorg).
- Relocate google_maps actor + tests scrapers/ -> platforms/ to match the
  reorg convention (youtube already there); rewrite imports + fixture paths.
- Add missing __init__.py across the capabilities/ test subtree so duplicate
  test basenames get unique module paths under importlib mode.

Note: google_maps fixture-backed tests error on ci_mvp too (fixtures/*.json
never committed upstream) - pre-existing, out of scope here.
2026-07-03 12:37:12 +02:00
CREDO23
817ec0e9f4 refactor(youtube): remove legacy KB ingestion; use actor get_youtube_video_id 2026-07-03 12:00:22 +02:00
CREDO23
ab6be6cbda refactor(connectors): remove legacy web-crawler KB indexer paths (keep enums) 2026-07-03 11:53:33 +02:00
CREDO23
62cb0efb44 refactor(youtube): drop duplicate POST /youtube/scrape+comments routes (use capability doors) 2026-07-03 11:38:56 +02:00
CREDO23
3312a442f3 refactor(proprietary): move scrapers/ -> platforms/ (one subpackage per platform) 2026-07-03 11:01:58 +02:00
DESKTOP-RTLN3BA\$punk
7185079bd6 feat(native-connector): added google maps places & reviews scrapers 2026-07-02 21:58:24 -07:00
CREDO23
0b2dba7a57 feat(capabilities): expose youtube.* verbs on the REST door 2026-07-02 20:18:53 +02:00
CREDO23
af3e70ea56 Merge upstream/ci_mvp: add YouTube scraper actor under app/proprietary
Resolved modify/delete conflicts on the old 'revamp phases 4-7/' plan docs
by keeping our deletion (superseded by flattened plans/backend/00-*.md).
2026-07-02 20:01:11 +02:00
CREDO23
6bd0d640e6 refactor(capabilities): group framework into core/, keep verbs top-level 2026-07-02 19:54:20 +02:00
DESKTOP-RTLN3BA\$punk
0445c646c6 refactor(native-connector): relocate youtube scraper under app/proprietary + parallelize playlists
Move app/scrapers -> app/proprietary/scrapers/youtube to sit alongside the existing proprietary web_crawler/platforms namespace, updating all external imports (routes, tests, e2e script, README). Internal imports were relative so are unchanged.

Also parallelize playlist per-video resolution: page video ids sequentially, then resolve the heavy watch-page fetches concurrently via fan_out (~150 videos ~70s, down from a few minutes). Items stream in completion order; sort by the order field for playlist order.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-02 02:34:04 -07:00
DESKTOP-RTLN3BA\$punk
99378d7b63 feat(native-connector): added youtube scrapers 2026-07-02 01:10:31 -07:00
CREDO23
b1e49f21f2 feat(capabilities): mount capability REST routes 2026-07-01 17:53:42 +02:00
Anish Sarkar
0df51c60f3 refactor(models): remove manual model registration backend 2026-06-27 03:46:15 +05:30
CREDO23
56826a63bc refactor(routes): rename to workspace + consolidate URL spellings (Phase 2 Wave E)
Hard cutover of the HTTP surface: collapse the three legacy spellings
(/searchspaces, /search-spaces, /search-space/{id}) onto canonical
/workspaces/{workspace_id}/..., rename the gateway field-setter sub-actions to
singular /workspace, rename search_spaces_routes.py -> workspaces_routes.py and the
search_spaces_router -> workspaces_router include, and flip search_space_id ->
workspace_id in bodies/params. No alias routers: the old URLs now 404 by design.
Full app constructs with zero legacy path spellings.
2026-06-26 18:35:08 +02:00
Rohan Verma
94fdb8a113
Merge pull request #1539 from CREDO23/improve-chat-agent-context-and-citations
[FEAT] Unified [n] citation registry for KB + web, pull-based retrieval
2026-06-25 13:34:52 -07:00
Rohan Verma
96e42a1003
Merge pull request #1536 from CREDO23/feature-mention-chat-in-chat
[Feat] Chat : Reference past chats via @-mention as read-only context
2026-06-25 13:32:25 -07:00
CREDO23
2beafbdec8 agent: retire eager KB priority/planner path and its dead flags
The pull-based KB design (on-demand search_knowledge_base tool + pre-injected
workspace tree) fully replaced the old eager retrieval path. Remove its last
remnants:

- Delete KnowledgePriorityMiddleware (knowledge_search.py) and its tests.
- Drop the kb_priority state field + reducer default; trim
  KbContextProjectionMiddleware to project only workspace_tree_text.
- Remove the now-dead feature flags enable_kb_priority_preinjection and
  enable_kb_planner_runnable across backend (flags, route schema, tests,
  env examples) and frontend (settings toggle, zod schema).
- Scrub <priority_documents> and stale KnowledgePriorityMiddleware references
  from prompts, docstrings, and the ADR.

No functional change: nothing wrote kb_priority and neither flag gated live
behavior after the cutover. Full backend suite green (pre-existing unrelated
failures aside).
2026-06-25 18:37:14 +02:00
Anish Sarkar
2e33ba7723 chore: fix linting 2026-06-25 04:31:36 +05:30
Anish Sarkar
d6bffa6f07 chore: fix linting 2026-06-25 04:31:22 +05:30
Anish Sarkar
e5aded5a65 feat(oauth): migrate Google OAuth account IDs to use 'sub' and enhance user resolution logic 2026-06-25 03:18:42 +05:30
Anish Sarkar
62c7efb216 fix(auth):enforce session auth cutover 2026-06-24 03:55:39 +05:30
CREDO23
857f1bb279 feat(chat): inject referenced-chat context into the new-chat agent input
Thread mentioned_thread_ids from the route through the orchestrator into
input-state assembly, resolve them for the requesting user, and append
the rendered referenced-chat block to the agent's query context.
2026-06-23 17:04:32 +02:00