apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-08 22:22:17 +02:00
Code Issues Projects Releases Packages Wiki Activity

feat(reddit): implement Reddit scraping subagent and associated capabilities

Browse source 
- Added a new `reddit` subagent to scrape structured data from Reddit posts, comments, and users.
- Introduced `reddit.scrape` capability for fetching data using URLs and search queries.
- Implemented tools for scraping and parsing Reddit data, including handling pagination and rate limits.
- Created input/output models for the Reddit scraper to define request and response structures.
- Added documentation for the new Reddit scraping functionality and its usage.
- Integrated the Reddit subagent into the existing multi-agent chat framework.
This commit is contained in:
DESKTOP-RTLN3BA\$punk 2026-07-04 17:31:11 -07:00
parent 701f888b9e
commit ff2e5f390f
35 changed files with 510 additions and 15 deletions
Download patch file Download diff file Expand all files Collapse all files

12
surfsense_backend/scripts/e2e_reddit_scraper.py
View file

@ -1,4 +1,4 @@
"""Manual functional e2e for the Reddit scraper (app/proprietary/scrapers/reddit).
"""Manual functional e2e for the Reddit scraper (app/proprietary/platforms/reddit).
Run from the backend directory:
cd surfsense_backend
@ -18,7 +18,7 @@ It:
Step 3 — run a search query.
Step 4 — scrape a user profile.
Step 5 — dump trimmed raw ``.json`` fixtures into
tests/unit/scrapers/reddit/fixtures/ for the offline parser tests.
tests/unit/platforms/reddit/fixtures/ for the offline parser tests.
"""
import asyncio
@ -36,22 +36,22 @@ for _candidate in (_BACKEND_ROOT / ".env", _BACKEND_ROOT.parent / ".env"):
load_dotenv(_candidate)
break
from app.proprietary.scrapers.reddit import ( # noqa: E402
from app.proprietary.platforms.reddit import ( # noqa: E402
RedditScrapeInput,
scrape_reddit,
)
from app.proprietary.scrapers.reddit.fetch import ( # noqa: E402
from app.proprietary.platforms.reddit.fetch import ( # noqa: E402
fetch_json,
proxy_session,
warm_session,
)
from app.proprietary.scrapers.reddit.parsers import children # noqa: E402
from app.proprietary.platforms.reddit.parsers import children # noqa: E402
_SUBREDDIT = "python"
_SEARCH_TERM = "async web scraping"
_USER = "spez"
_FIXTURE_DIR = _BACKEND_ROOT / "tests" / "unit" / "scrapers" / "reddit" / "fixtures"
_FIXTURE_DIR = _BACKEND_ROOT / "tests" / "unit" / "platforms" / "reddit" / "fixtures"
def _hr(title: str) -> None: