Anish Sarkar
e8f8eeab27
fix(instagram): improve error handling for Instagram access blocks in scraper
...
Updated the fan_out function to handle InstagramAccessBlockedError more gracefully. Instead of raising the error directly, it now puts the error into the results queue to prevent deadlocks. This change ensures that the consumer can handle access block scenarios without interrupting the processing of other jobs.
2026-07-09 19:08:18 +05:30
Anish Sarkar
6384409b1d
test(instagram): enhance fetch resilience tests for login redirects
...
Added tests to handle scenarios where a 200 status response is redirected to a login page, ensuring proper rotation of IPs and raising errors when persistent login redirects occur. Updated the _FakeSession class to support login wall detection.
2026-07-09 18:29:29 +05:30
Anish Sarkar
d800ca33a5
feat(instagram): handle soft login wall in fetch logic
...
Added detection for Instagram's soft login wall, which returns a 200 status with login HTML. Implemented a new function to identify login redirects and adjusted the fetch logic to treat these cases similarly to 401/403 responses.
2026-07-09 18:29:17 +05:30
Anish Sarkar
7309ca5ca0
test(instagram): add manual e2e scraper script
2026-07-09 16:03:29 +05:30
Anish Sarkar
e151346fbe
test(chat): include Instagram in subagent composition test
2026-07-09 16:03:23 +05:30
Anish Sarkar
458d223ef8
test(billing): cover Instagram item and comment billing
2026-07-09 16:02:54 +05:30
Anish Sarkar
2e66e714e2
test(instagram): add capability unit tests
2026-07-09 16:02:48 +05:30
Anish Sarkar
9da136070d
test(instagram): add platform-layer unit tests
2026-07-09 16:02:41 +05:30
Anish Sarkar
98eab8c4b5
feat(chat): add Instagram builtin subagent
2026-07-09 16:02:05 +05:30
Anish Sarkar
929efe8152
feat(instagram): register instagram.* capability namespace
2026-07-09 16:01:49 +05:30
Anish Sarkar
7ea474be93
feat(instagram): add details capability verb
2026-07-09 16:01:38 +05:30
Anish Sarkar
38a26b1ccf
feat(instagram): add comments capability verb
2026-07-09 14:58:58 +05:30
Anish Sarkar
ff55537bce
feat(instagram): add scrape capability verb
2026-07-09 14:58:54 +05:30
Anish Sarkar
eccc142974
docs(instagram): add platform scraper README
2026-07-09 14:58:47 +05:30
Anish Sarkar
6270250f74
feat(instagram): add fan-out orchestrator with deterministic early-stop cleanup
2026-07-09 14:58:38 +05:30
Anish Sarkar
b1eff478fd
feat(instagram): add proxy-aware fetch with warm-up and rotate-on-block
2026-07-09 14:58:32 +05:30
Anish Sarkar
2d29001004
feat(instagram): add web-JSON parsers
2026-07-09 14:58:27 +05:30
Anish Sarkar
8dc8ad6a36
feat(instagram): add URL classifier and normalizer
2026-07-09 14:58:15 +05:30
Anish Sarkar
842a0e666d
feat(instagram): add platform input/output schemas
2026-07-09 14:58:11 +05:30
Anish Sarkar
51c2dfdfe6
feat(config): add Instagram per-item and per-comment scrape rates
2026-07-09 14:57:59 +05:30
Anish Sarkar
43660ee51b
feat(billing): map Instagram units to rate keys and display nouns
2026-07-09 14:57:53 +05:30
Anish Sarkar
f204db7bce
feat(billing): add INSTAGRAM_ITEM and INSTAGRAM_COMMENT units
2026-07-09 14:57:44 +05:30
Anish Sarkar
8486a4f914
feat(constants): add Instagram to SUBAGENT_TO_REQUIRED_CONNECTOR_MAP for enhanced multi-agent chat capabilities
2026-07-09 14:08:23 +05:30
Anish Sarkar
3bd864ed95
feat(env): add Instagram scraping configuration parameters to .env.example
2026-07-09 14:08:10 +05:30
Rohan Verma
3eb485ff31
Merge pull request #1575 from SyntaxSawdust/feature/1354-embedding-base-url
...
Support separate embedding base URL configuration
2026-07-08 17:11:42 -07:00
Rohan Verma
d2ed0bb382
Merge pull request #1570 from pelazas/fix/scheduler-task-map-bookstack
...
fix: dispatch periodic BookStack indexing from the scheduler task_map
2026-07-08 15:07:38 -07:00
DESKTOP-RTLN3BA\$punk
b8155c6c0e
Merge commit ' 1ae927a7db' into dev
2026-07-08 15:05:11 -07:00
CREDO23
ba375ea3ee
feat(tiktok): emit graceful ErrorItem for blocked/empty listings
...
Profile and search feeds are trust-gated: an anonymous headless session
gets an empty item_list (profile) or no results XHR (search), while
hashtag feeds load. A zero-item listing now yields one honest ErrorItem
(errorCode="no_items") instead of vanishing silently, and ErrorItems are
excluded from billing so a blocked target is surfaced but never charged.
2026-07-08 23:14:50 +02:00
CREDO23
5f9472b122
feat(tiktok): advertise tiktok across agent prompts, MCP instructions, docs
2026-07-08 20:30:38 +02:00
CREDO23
ed1c3a1f3d
feat(tiktok): expose tiktok.scrape on MCP, chat subagent, playground
2026-07-08 20:20:20 +02:00
CREDO23
2943d8b23c
feat(tiktok): tiktok.scrape capability + billing wire-up
2026-07-08 18:21:44 +02:00
CREDO23
9dd39faa50
test(tiktok): pin live fixtures, trim listing debug scaffolding
2026-07-08 17:53:29 +02:00
CREDO23
ea95a0529f
test(tiktok): live e2e smoke script
2026-07-08 17:14:13 +02:00
CREDO23
5a326f3818
fix(tiktok): warm browser listing + drop ctx-var-in-generator
2026-07-08 17:14:13 +02:00
CREDO23
ad8b73477a
feat(tiktok): browser-driven signed listings
2026-07-08 16:37:36 +02:00
CREDO23
daa0856c44
feat(tiktok): blob-first orchestrator + video flow
2026-07-08 16:07:07 +02:00
CREDO23
44b3e640d3
feat(tiktok): rotate-on-block fetch session
2026-07-08 15:51:17 +02:00
CREDO23
5688ab0678
feat(tiktok): scrape input schema
2026-07-08 15:51:17 +02:00
CREDO23
ef66063bfc
feat(tiktok): video normalizer and parser tests
2026-07-08 15:40:50 +02:00
CREDO23
0bbcf99852
feat(tiktok): author normalizer
2026-07-08 15:40:38 +02:00
CREDO23
8840fd2c53
feat(tiktok): rehydration blob extractor
2026-07-08 15:40:21 +02:00
CREDO23
b2d387f994
feat(tiktok): URL target resolver
2026-07-08 15:40:07 +02:00
CREDO23
ef3f9f2e25
feat(tiktok): item output schemas
2026-07-08 15:39:58 +02:00
CREDO23
3c027b9f49
Merge remote-tracking branch 'upstream/main' into dev
...
# Conflicts:
# surfsense_mcp/mcp_server/core/auth/__init__.py
# surfsense_mcp/mcp_server/core/auth/headers.py
# surfsense_mcp/mcp_server/core/auth/identity.py
# surfsense_mcp/mcp_server/core/auth/middleware.py
# surfsense_mcp/mcp_server/core/transport/__init__.py
# surfsense_mcp/mcp_server/core/transport/http.py
# surfsense_mcp/mcp_server/core/workspace_matching.py
# surfsense_mcp/mcp_server/features/knowledge_base/annotations.py
# surfsense_mcp/mcp_server/features/knowledge_base/document_tools.py
# surfsense_mcp/mcp_server/features/knowledge_base/search_tools.py
# surfsense_mcp/mcp_server/features/scrapers/annotations.py
# surfsense_mcp/mcp_server/features/scrapers/platforms/__init__.py
# surfsense_mcp/mcp_server/features/scrapers/platforms/google_maps.py
# surfsense_mcp/mcp_server/features/scrapers/platforms/google_search.py
# surfsense_mcp/mcp_server/features/scrapers/platforms/reddit.py
# surfsense_mcp/mcp_server/features/scrapers/platforms/web.py
# surfsense_mcp/mcp_server/features/scrapers/platforms/youtube.py
# surfsense_mcp/mcp_server/features/scrapers/run_history.py
# surfsense_mcp/src/surfsense_mcp/__main__.py
# surfsense_mcp/src/surfsense_mcp/features/knowledge_base/__init__.py
# surfsense_mcp/src/surfsense_mcp/features/scrapers/__init__.py
# surfsense_mcp/tests/test_auth_headers.py
# surfsense_mcp/tests/test_request_auth.py
# surfsense_mcp/tests/test_workspace_context.py
# surfsense_web/content/docs/how-to/mcp-server.mdx
# surfsense_web/lib/mcp/clients.ts
2026-07-08 14:52:55 +02:00
Anish Sarkar
7ba77c6b86
feat(capabilities): add documentation URLs to Google Maps, YouTube, and web crawl capabilities
...
- Introduced a `docs_url` field to the Google Maps reviews, scrape, YouTube comments, YouTube scrape, and web crawl capabilities for improved documentation access.
- Simplified descriptions for each capability to enhance clarity and user understanding.
2026-07-08 03:38:08 +05:30
Anish Sarkar
3b9c806c90
feat(capabilities): add documentation URL to capabilities and enhance Reddit scrape capability
...
- Introduced a `docs_url` field to the `Capability` and `CapabilitySummary` classes for better documentation access.
- Updated the `REDDIT_SCRAPE` capability to include a specific documentation link.
- Enhanced the PlaygroundRunner component to display the documentation link when available, improving user guidance.
2026-07-08 03:35:32 +05:30
Dustin Persek
41edac7641
Merge remote-tracking branch 'upstream/dev' into feature/1354-embedding-base-url
...
# Conflicts:
# surfsense_web/content/docs/docker-installation/docker-compose.mdx
# surfsense_web/content/docs/manual-installation.mdx
2026-07-07 14:19:40 -04:00
DESKTOP-RTLN3BA\$punk
b477d33cba
Merge commit ' fa7075fde6' into dev
2026-07-07 00:39:58 -07:00
DESKTOP-RTLN3BA\$punk
9a9a78e7ef
refactor: improve session management and concurrency in Google Search fetcher
...
- Introduced a semaphore to limit concurrent page renders, enhancing resource management.
- Updated session handling to defer browser closure until all in-flight renders are complete.
- Improved comments for clarity on the behavior of concurrent fetches and session lifecycle.
- Cleaned up imports in alembic environment and migration flow scripts for consistency.
2026-07-06 22:33:40 -07:00
DESKTOP-RTLN3BA\$punk
1c9ab207ef
chore: bumped version to 0.0.31
2026-07-06 21:43:15 -07:00
