mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-10 22:32:16 +02:00
feat(reddit): implement Reddit scraping subagent and associated capabilities
- Added a new `reddit` subagent to scrape structured data from Reddit posts, comments, and users. - Introduced `reddit.scrape` capability for fetching data using URLs and search queries. - Implemented tools for scraping and parsing Reddit data, including handling pagination and rate limits. - Created input/output models for the Reddit scraper to define request and response structures. - Added documentation for the new Reddit scraping functionality and its usage. - Integrated the Reddit subagent into the existing multi-agent chat framework.
This commit is contained in:
parent
701f888b9e
commit
ff2e5f390f
35 changed files with 510 additions and 15 deletions
|
|
@ -60,17 +60,17 @@ rotation surfaces as `RedditAccessBlockedError` (mirrors google_maps'
|
|||
|
||||
## Testing
|
||||
|
||||
- Offline unit tests: `tests/unit/scrapers/reddit/` — `test_skeleton.py`
|
||||
- Offline unit tests: `tests/unit/platforms/reddit/` — `test_skeleton.py`
|
||||
(schema + URL resolver), `test_parsers.py` (fixture-pinned mapping),
|
||||
`test_fetch_resilience.py` (warm / rotate / backoff loop with fake sessions,
|
||||
no network).
|
||||
- Live e2e (needs network + residential proxy): `scripts/e2e_reddit_scraper.py`
|
||||
— step 0 is the go/no-go `loid` probe; later steps exercise the flows and dump
|
||||
trimmed fixtures into `tests/unit/scrapers/reddit/fixtures/`.
|
||||
trimmed fixtures into `tests/unit/platforms/reddit/fixtures/`.
|
||||
|
||||
```bash
|
||||
cd surfsense_backend
|
||||
.venv/bin/python -m pytest tests/unit/scrapers/reddit/
|
||||
.venv/bin/python -m pytest tests/unit/platforms/reddit/
|
||||
.venv/bin/python scripts/e2e_reddit_scraper.py # live; regenerates fixtures
|
||||
```
|
||||
|
||||
Loading…
Add table
Add a link
Reference in a new issue