Commit graph

5 commits

Author SHA1 Message Date
CREDO23
d5d673384f Merge upstream/ci_mvp (google maps scrapers) into feature-ci_phase-4-7
Resolve conflicts against the new native google-maps actor + repo-wide
ruff-format pass:
- Keep legacy webcrawler KB indexer + its test deleted (modify/delete).
- test_validators: keep WEBCRAWLER case removed (validator gone).
- test_fetch_resilience: keep platforms.youtube import path (our reorg).
- Relocate google_maps actor + tests scrapers/ -> platforms/ to match the
  reorg convention (youtube already there); rewrite imports + fixture paths.
- Add missing __init__.py across the capabilities/ test subtree so duplicate
  test basenames get unique module paths under importlib mode.

Note: google_maps fixture-backed tests error on ci_mvp too (fixtures/*.json
never committed upstream) - pre-existing, out of scope here.
2026-07-03 12:37:12 +02:00
CREDO23
3312a442f3 refactor(proprietary): move scrapers/ -> platforms/ (one subpackage per platform) 2026-07-03 11:01:58 +02:00
DESKTOP-RTLN3BA\$punk
7185079bd6 feat(native-connector): added google maps places & reviews scrapers 2026-07-02 21:58:24 -07:00
DESKTOP-RTLN3BA\$punk
0445c646c6 refactor(native-connector): relocate youtube scraper under app/proprietary + parallelize playlists
Move app/scrapers -> app/proprietary/scrapers/youtube to sit alongside the existing proprietary web_crawler/platforms namespace, updating all external imports (routes, tests, e2e script, README). Internal imports were relative so are unchanged.

Also parallelize playlist per-video resolution: page video ids sequentially, then resolve the heavy watch-page fetches concurrently via fan_out (~150 videos ~70s, down from a few minutes). Items stream in completion order; sort by the order field for playlist order.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-02 02:34:04 -07:00
DESKTOP-RTLN3BA\$punk
99378d7b63 feat(native-connector): added youtube scrapers 2026-07-02 01:10:31 -07:00