feat(reddit): implement Reddit scraping subagent and associated capabilities

- Added a new `reddit` subagent to scrape structured data from Reddit posts, comments, and users.
- Introduced `reddit.scrape` capability for fetching data using URLs and search queries.
- Implemented tools for scraping and parsing Reddit data, including handling pagination and rate limits.
- Created input/output models for the Reddit scraper to define request and response structures.
- Added documentation for the new Reddit scraping functionality and its usage.
- Integrated the Reddit subagent into the existing multi-agent chat framework.
This commit is contained in:
DESKTOP-RTLN3BA\$punk 2026-07-04 17:31:11 -07:00
parent 701f888b9e
commit ff2e5f390f
35 changed files with 510 additions and 15 deletions

View file

@ -60,17 +60,17 @@ rotation surfaces as `RedditAccessBlockedError` (mirrors google_maps'
## Testing
- Offline unit tests: `tests/unit/scrapers/reddit/` — `test_skeleton.py`
- Offline unit tests: `tests/unit/platforms/reddit/` — `test_skeleton.py`
(schema + URL resolver), `test_parsers.py` (fixture-pinned mapping),
`test_fetch_resilience.py` (warm / rotate / backoff loop with fake sessions,
no network).
- Live e2e (needs network + residential proxy): `scripts/e2e_reddit_scraper.py`
— step 0 is the go/no-go `loid` probe; later steps exercise the flows and dump
trimmed fixtures into `tests/unit/scrapers/reddit/fixtures/`.
trimmed fixtures into `tests/unit/platforms/reddit/fixtures/`.
```bash
cd surfsense_backend
.venv/bin/python -m pytest tests/unit/scrapers/reddit/
.venv/bin/python -m pytest tests/unit/platforms/reddit/
.venv/bin/python scripts/e2e_reddit_scraper.py # live; regenerates fixtures
```