DESKTOP-RTLN3BA\$punk
1c8a15dfab
refactor: consolidate browser loop management across Google Search, Indeed Jobs, and Reddit platforms
2026-07-24 15:01:01 -07:00
DESKTOP-RTLN3BA\$punk
4a727a2c57
chore: linting
2026-07-24 14:30:46 -07:00
Anish Sarkar
8d1e2a5134
feat(reddit-scraper): add support for community-only subreddit scraping without search queries
2026-07-25 02:55:27 +05:30
Anish Sarkar
2379a6a23f
feat(reddit-schemas): add community name normalization validator to prevent 404 errors
2026-07-25 02:55:20 +05:30
CREDO23
6330d5a789
fix: reject spoofed non-YouTube URLs
2026-07-23 18:52:50 +02:00
CREDO23
ca317c4686
Merge remote-tracking branch 'upstream/dev' into feature-walmart-scraper
...
# Conflicts:
# docker/.env.example
# surfsense_backend/app/capabilities/core/billing.py
# surfsense_backend/app/capabilities/core/types.py
# surfsense_backend/app/config/__init__.py
# surfsense_mcp/mcp_server/features/scrapers/__init__.py
# surfsense_web/content/docs/connectors/index.mdx
# surfsense_web/content/docs/connectors/native/index.mdx
# surfsense_web/content/docs/how-to/mcp-server.mdx
2026-07-21 02:42:41 +02:00
CREDO23
f6970b90e8
fix(walmart): parse category breadcrumb path into breadCrumbs and category
2026-07-19 09:10:03 +02:00
CREDO23
d446aa32ec
feat(walmart): add scrape and review orchestration
2026-07-19 08:19:22 +02:00
CREDO23
661bb8c187
feat(walmart): add JSON parsers
2026-07-19 08:19:22 +02:00
CREDO23
34b6885ba2
feat(walmart): add fetch with proxy rotation and block detection
2026-07-19 08:19:21 +02:00
CREDO23
dd3dba66af
feat(walmart): add __NEXT_DATA__ extraction
2026-07-19 08:19:21 +02:00
CREDO23
be1fd0491f
feat(walmart): add url resolver
2026-07-19 08:19:21 +02:00
CREDO23
a72cc3436c
feat(walmart): add product and review schemas
2026-07-19 08:19:21 +02:00
CREDO23
91aa265afb
Merge remote-tracking branch 'upstream/dev' into feature-indeed-jobs-scraper
...
# Conflicts:
# README.es.md
# README.hi.md
# README.md
# README.pt-BR.md
# README.zh-CN.md
# surfsense_backend/tests/unit/capabilities/google_maps/test_registry.py
# surfsense_backend/tests/unit/capabilities/reddit/test_registry.py
# surfsense_backend/tests/unit/capabilities/youtube/test_registry.py
# surfsense_mcp/mcp_server/features/scrapers/__init__.py
# surfsense_web/content/docs/connectors/index.mdx
# surfsense_web/content/docs/connectors/native/index.mdx
# surfsense_web/content/docs/connectors/native/meta.json
# surfsense_web/content/docs/how-to/mcp-server.mdx
# surfsense_web/lib/connectors-marketing/index.ts
# surfsense_web/lib/playground/catalog.ts
2026-07-18 22:11:30 +02:00
DESKTOP-RTLN3BA\$punk
87b0ba86e4
feat: bumped version to 0.0.34
2026-07-17 19:42:10 -07:00
DESKTOP-RTLN3BA\$punk
2cbe6fc111
feat: bumped version to 0.0.33
2026-07-17 17:23:21 -07:00
DESKTOP-RTLN3BA\$punk
8a5d37a4db
refactor: implement cross-country proxy rotation for Reddit and TikTok scrapers, enhancing resilience against IP blocks and improving search query handling
2026-07-17 16:30:26 -07:00
DESKTOP-RTLN3BA\$punk
c0ebb62fb2
chore: linting
2026-07-17 14:39:39 -07:00
DESKTOP-RTLN3BA\$punk
68d1832f10
refactor: enhance Google search scraping capabilities with warm sticky-IP pool and improved captcha handling
2026-07-17 03:27:43 -07:00
CREDO23
c485fe4c43
fix: scrape indeed first page only, steer subagent to one query
2026-07-15 20:54:37 +02:00
CREDO23
6180b96912
fix: fail fast on gated indeed pagination, keep collected pages
2026-07-15 20:24:22 +02:00
CREDO23
b53f00231d
fix: parse indeed /viewjob detail from window._rootProps
2026-07-15 19:57:19 +02:00
CREDO23
51c7220321
feat: indeed job-detail and enrichment flow
2026-07-15 17:54:38 +02:00
CREDO23
6ee95e5b2c
feat: parse indeed /viewjob detail page
2026-07-15 17:54:38 +02:00
Anish Sarkar
01308055dd
chore: ran linting
2026-07-15 15:48:54 +05:30
Anish Sarkar
6e4678c420
feat(amazon): apply marketplace locale across scraper flows
2026-07-15 15:32:21 +05:30
Anish Sarkar
53ccf575a6
fix(amazon): parse eu and us price decimal formats
2026-07-15 15:32:11 +05:30
Anish Sarkar
82e4272834
feat(amazon): route fetches through marketplace geo and language
2026-07-15 15:32:06 +05:30
Anish Sarkar
952aca3de5
feat(amazon): add marketplace locale mapping
2026-07-15 15:31:53 +05:30
Anish Sarkar
94b16a8334
fix(amazon): update anti-bot detection logic to include soft responses
2026-07-15 13:00:53 +05:30
Anish Sarkar
603c94c80b
feat(amazon): enhance product fetching and parsing logic for better offer retrieval
...
- Updated the fetch function to correctly construct the URL for the all-offers panel.
- Improved the product parser to include additional selectors for price and seller information.
- Added a fallback mechanism to retrieve seller information from the buy-box when not available in the standard link.
2026-07-15 11:04:46 +05:30
Anish Sarkar
118411ebca
docs(amazon): add module status readme
2026-07-15 01:56:34 +05:30
Anish Sarkar
f69a573d10
feat(amazon): expose public module api
2026-07-15 01:56:29 +05:30
Anish Sarkar
0cb15272ea
feat(amazon): add scraper orchestrator flows
2026-07-15 01:56:19 +05:30
Anish Sarkar
165f7464b3
feat(amazon): add proxy-aware fetch with captcha rotate
2026-07-15 01:56:02 +05:30
Anish Sarkar
1b6383f9bc
feat(amazon): add pure html parsers
2026-07-15 01:55:22 +05:30
Anish Sarkar
a0ab303768
feat(amazon): add start-url resolver
2026-07-15 01:55:15 +05:30
Anish Sarkar
4e5b039a05
feat(amazon): add scrape input and output schemas
2026-07-15 01:55:09 +05:30
CREDO23
1d367148d0
feat: indeed_jobs package exports
2026-07-14 21:41:54 +02:00
CREDO23
824041ef7f
feat: indeed scraper orchestrator with pagination and dedupe
2026-07-14 21:41:54 +02:00
CREDO23
44f9f0301c
feat: indeed warmed browser session with rotate-on-block
2026-07-14 21:41:54 +02:00
CREDO23
d87c86dcc4
feat: indeed url classifier and search url builder
2026-07-14 21:41:54 +02:00
CREDO23
01c3f097e6
feat: indeed jobcards blob extraction and job parser
2026-07-14 21:41:54 +02:00
CREDO23
249bba0886
feat: indeed scraper input/output schemas
2026-07-14 21:41:54 +02:00
DESKTOP-RTLN3BA\$punk
2b018c4474
refactor: streamline TikTok and Instagram scraping logic by removing search_queries and enhancing documentation for clarity
2026-07-13 17:11:25 -07:00
DESKTOP-RTLN3BA\$punk
1131da5ed7
feat: bumped version to 0.0.32
2026-07-13 16:29:39 -07:00
Anish Sarkar
e38ca19b18
Merge remote-tracking branch 'upstream/dev' into feat/instagram-scraper
2026-07-11 04:31:11 +05:30
Anish Sarkar
819486ac46
docs(instagram): document PolarisMedia extraction and richer media fields
2026-07-11 03:44:38 +05:30
Anish Sarkar
d3c65a37b1
refactor(instagram): drop unused slug field and mark share links unsupported
2026-07-11 03:44:28 +05:30
Anish Sarkar
4bc3d43b56
refactor(instagram): remove unused share-link redirect resolver
2026-07-11 03:44:23 +05:30
