Commit graph

7476 commits

Author SHA1 Message Date
CREDO23
d3a6e2cd45 docs(docker): note entrypoint wires xvfb at runtime
The xvfb runtime path is now implemented, so drop the stale 'deferred to Slice B' note.
2026-07-10 17:17:53 +02:00
CREDO23
6400dc5f04 fix(tiktok): gate headful profile feed on CRAWL_HEADED_XVFB_ENABLED
fetch_item_list goes headful only when the flag promises a display; otherwise it stays headless so the browser launch never fails and the empty feed degrades to an ErrorItem. Verified under xvfb-run: real videos returned on a clean proxy IP, graceful degrade on a flagged one.
2026-07-10 17:17:45 +02:00
CREDO23
a551dd7553 feat(docker): start Xvfb when CRAWL_HEADED_XVFB_ENABLED is set
For every browser-running role (api/worker/all, not migrate) the entrypoint brings up a virtual display and exports DISPLAY, registering Xvfb for graceful shutdown. Off keeps browsers headless.
2026-07-10 17:17:36 +02:00
CREDO23
6784503414 feat(crawl): add CRAWL_HEADED_XVFB_ENABLED flag
Single switch promising an Xvfb display so the stealth browser can run headful (TikTok's profile feed is empty to headless Chromium). Defaults FALSE.
2026-07-10 17:17:27 +02:00
CREDO23
44966315f1 docs(tiktok): update e2e stage 2 notes for headful profile feed
Profile videos now return via the headful item_list capture; reword the
stage 2 comments to expect real videos, ErrorItem only as fallback.
2026-07-10 16:45:53 +02:00
CREDO23
c679f2a3ef fix(tiktok): fetch profile video lists headful + dismiss login modal
The profile feed (/api/post/item_list) returns an empty 200 to headless
sessions but serves data headful on the same proxy IP. Run fetch_item_list
headful and dismiss the login modal that blocks mid-scroll.
2026-07-10 16:45:45 +02:00
CREDO23
867e940911 Merge upstream/dev into feature/tiktok-scraper
Sync with the advanced dev tip so the PR compares against current dev
(drops already-merged commits from the diff). No history rewritten.
2026-07-09 20:15:20 +02:00
CREDO23
bcfc3a5c3a test(capabilities): expect export_run in the agent tool manifest
The run-reader set gained export_run (CSV export of a stored run), but this
manifest assertion still expected only read_run/search_run. Add export_run so
the agent-door test reflects the shipped readers.
2026-07-09 20:04:30 +02:00
CREDO23
6b15d610d9 feat(tiktok): surface comments, user_search, trending across all product surfaces
Brings the three newer verbs to parity with tiktok.scrape everywhere it was
wired: MCP tools (+ selfcheck manifest), the chat subagent prompt/description,
the playground catalog, the native docs page, and the SEO marketing page. Also
adds live e2e stages for comments, user search, and trending. Docs/FAQ now
state the real contract: profile metadata is reliable while its video list can
be withheld, and keyword video search is walled (use user search for accounts).
2026-07-09 19:09:00 +02:00
CREDO23
67b5472b9f feat(tiktok): add tiktok.trending verb for the Explore feed
The Explore feed (/api/explore/item_list) is a global trending-video feed
served to anonymous sessions, and it returns the same itemStruct shape as the
other listings — so the verb reuses parse_video, the listing flow, the
TikTokVideoItem output, and the per-video billing meter wholesale. Adds a
browser-capture marker + fetch_trending, a synthetic-target orchestrator entry,
and the tiktok.trending capability, surfaced on the chat subagent.
2026-07-09 18:52:56 +02:00
CREDO23
7723b5b8b6 feat(tiktok): add tiktok.comments verb for public comment scraping
Comments load over a signed /api/comment/list XHR that TikTok serves to
anonymous sessions once the comments panel is opened (unlike profile-video and
general-search feeds), so this is a reliable verb. Given video URLs it returns
CommentItems (text, author, likes, reply counts; replies carry repliesToId),
deduped per video, capped, and degraded to an ErrorItem for empty/withheld
videos or a bad_url ErrorItem for non-video inputs.

Generalizes the browser capture over a pluggable interaction step so the
comments flow (open panel, scroll the panel to paginate) reuses the same
warm+capture scaffolding as listing/user-search. Billed per comment on a new
TIKTOK_COMMENT meter (TIKTOK_MICROS_PER_COMMENT, matching the per-comment
market), surfaced on the chat subagent alongside tiktok.scrape/user_search.
2026-07-09 18:35:26 +02:00
CREDO23
192b6dc31a feat(tiktok): add tiktok.user_search verb for account discovery
Video/general search is login-walled for anonymous sessions, but the Users
tab (/api/search/user) returns public account records without a redirect, so
this exposes the one reliably-unblocked search path. A keyword yields
TikTokProfileItems (name, followers, bio, verification), deduped per query,
capped, and degraded to an ErrorItem when a query is empty/withheld.

Reuses the browser capture (generalized over XHR markers + extractor) and the
shared profile item shape. Billed per account on a new TIKTOK_USER meter
(TIKTOK_MICROS_PER_USER), surfaced on the chat subagent alongside tiktok.scrape.
2026-07-09 18:00:40 +02:00
CREDO23
6652efd035 feat(tiktok): return profile metadata even when video listing is gated
A profile's account data (name, followers, bio, verification) lives in the
page's rehydration blob and loads over plain HTTP without a signed request,
so emit it first and always. The video listing needs a signed item_list XHR
that TikTok withholds from anonymous sessions, so it stays best-effort and
degrades to an ErrorItem. A blocked profile now yields its metadata instead
of only an ErrorItem.
2026-07-09 17:48:19 +02:00
Rohan Verma
3eb485ff31
Merge pull request #1575 from SyntaxSawdust/feature/1354-embedding-base-url
Support separate embedding base URL configuration
2026-07-08 17:11:42 -07:00
Rohan Verma
d2ed0bb382
Merge pull request #1570 from pelazas/fix/scheduler-task-map-bookstack
fix: dispatch periodic BookStack indexing from the scheduler task_map
2026-07-08 15:07:38 -07:00
DESKTOP-RTLN3BA\$punk
b8155c6c0e Merge commit '1ae927a7db' into dev 2026-07-08 15:05:11 -07:00
CREDO23
ba375ea3ee feat(tiktok): emit graceful ErrorItem for blocked/empty listings
Profile and search feeds are trust-gated: an anonymous headless session
gets an empty item_list (profile) or no results XHR (search), while
hashtag feeds load. A zero-item listing now yields one honest ErrorItem
(errorCode="no_items") instead of vanishing silently, and ErrorItems are
excluded from billing so a blocked target is surfaced but never charged.
2026-07-08 23:14:50 +02:00
CREDO23
f731d371ab feat(tiktok): add TikTok SEO landing page and site-wide marketing parity 2026-07-08 20:36:46 +02:00
CREDO23
5f9472b122 feat(tiktok): advertise tiktok across agent prompts, MCP instructions, docs 2026-07-08 20:30:38 +02:00
CREDO23
ea6befdf59 docs(tiktok): native connector page + nav card 2026-07-08 20:21:29 +02:00
CREDO23
ed1c3a1f3d feat(tiktok): expose tiktok.scrape on MCP, chat subagent, playground 2026-07-08 20:20:20 +02:00
CREDO23
2943d8b23c feat(tiktok): tiktok.scrape capability + billing wire-up 2026-07-08 18:21:44 +02:00
CREDO23
9dd39faa50 test(tiktok): pin live fixtures, trim listing debug scaffolding 2026-07-08 17:53:29 +02:00
CREDO23
ea95a0529f test(tiktok): live e2e smoke script 2026-07-08 17:14:13 +02:00
CREDO23
5a326f3818 fix(tiktok): warm browser listing + drop ctx-var-in-generator 2026-07-08 17:14:13 +02:00
CREDO23
ad8b73477a feat(tiktok): browser-driven signed listings 2026-07-08 16:37:36 +02:00
CREDO23
daa0856c44 feat(tiktok): blob-first orchestrator + video flow 2026-07-08 16:07:07 +02:00
CREDO23
44b3e640d3 feat(tiktok): rotate-on-block fetch session 2026-07-08 15:51:17 +02:00
CREDO23
5688ab0678 feat(tiktok): scrape input schema 2026-07-08 15:51:17 +02:00
CREDO23
ef66063bfc feat(tiktok): video normalizer and parser tests 2026-07-08 15:40:50 +02:00
CREDO23
0bbcf99852 feat(tiktok): author normalizer 2026-07-08 15:40:38 +02:00
CREDO23
8840fd2c53 feat(tiktok): rehydration blob extractor 2026-07-08 15:40:21 +02:00
CREDO23
b2d387f994 feat(tiktok): URL target resolver 2026-07-08 15:40:07 +02:00
CREDO23
ef3f9f2e25 feat(tiktok): item output schemas 2026-07-08 15:39:58 +02:00
CREDO23
3c027b9f49 Merge remote-tracking branch 'upstream/main' into dev
# Conflicts:
#	surfsense_mcp/mcp_server/core/auth/__init__.py
#	surfsense_mcp/mcp_server/core/auth/headers.py
#	surfsense_mcp/mcp_server/core/auth/identity.py
#	surfsense_mcp/mcp_server/core/auth/middleware.py
#	surfsense_mcp/mcp_server/core/transport/__init__.py
#	surfsense_mcp/mcp_server/core/transport/http.py
#	surfsense_mcp/mcp_server/core/workspace_matching.py
#	surfsense_mcp/mcp_server/features/knowledge_base/annotations.py
#	surfsense_mcp/mcp_server/features/knowledge_base/document_tools.py
#	surfsense_mcp/mcp_server/features/knowledge_base/search_tools.py
#	surfsense_mcp/mcp_server/features/scrapers/annotations.py
#	surfsense_mcp/mcp_server/features/scrapers/platforms/__init__.py
#	surfsense_mcp/mcp_server/features/scrapers/platforms/google_maps.py
#	surfsense_mcp/mcp_server/features/scrapers/platforms/google_search.py
#	surfsense_mcp/mcp_server/features/scrapers/platforms/reddit.py
#	surfsense_mcp/mcp_server/features/scrapers/platforms/web.py
#	surfsense_mcp/mcp_server/features/scrapers/platforms/youtube.py
#	surfsense_mcp/mcp_server/features/scrapers/run_history.py
#	surfsense_mcp/src/surfsense_mcp/__main__.py
#	surfsense_mcp/src/surfsense_mcp/features/knowledge_base/__init__.py
#	surfsense_mcp/src/surfsense_mcp/features/scrapers/__init__.py
#	surfsense_mcp/tests/test_auth_headers.py
#	surfsense_mcp/tests/test_request_auth.py
#	surfsense_mcp/tests/test_workspace_context.py
#	surfsense_web/content/docs/how-to/mcp-server.mdx
#	surfsense_web/lib/mcp/clients.ts
2026-07-08 14:52:55 +02:00
Anish Sarkar
1ae927a7db
Merge pull request #1588 from AnishSarkar22/fix/ci-ui-changes
feat(api-keys): implement API keys management interface and access control
2026-07-08 10:44:45 +05:30
Anish Sarkar
719c2da525 feat(api-keys): implement API keys management interface and access control
- Added a new WorkspaceApiAccessControl component for managing API key access within workspaces.
- Enhanced the PlaygroundApiKeysPage to display API key creation options and access information.
- Updated PlaygroundIndex and PlaygroundSidebar to include links to the new API keys management section.
- Improved user feedback with alerts and descriptions regarding API key usage and requirements.
2026-07-08 10:43:11 +05:30
Anish Sarkar
7faa58a953
Merge pull request #1587 from AnishSarkar22/fix/ci-ui-changes
feat(playground): improve API Playground navigation and sidebar UX
2026-07-08 10:31:35 +05:30
Anish Sarkar
9a18c7771e refactor(sidebar): replace FolderInput with FilePlus in DocumentsSidebar
- Updated the import of FolderInput to FilePlus for improved icon representation in the sidebar.
- Enhanced the button for document import to align with the new iconography, improving visual consistency.
2026-07-08 10:28:53 +05:30
Anish Sarkar
53d3b304d3 refactor(playground): improve API reference and playground runner UI
- Adjusted text formatting in the API reference for better readability.
- Simplified import statements in PlaygroundRunner for cleaner code.
- Enhanced button styling and layout in PlaygroundRunner for improved user experience.
- Streamlined output display logic in PlaygroundRunner for better clarity.
2026-07-08 10:17:01 +05:30
Anish Sarkar
33e685c6ed feat(sidebar): add isPlaygroundSidebarOpen prop for improved sidebar state management
- Introduced isPlaygroundSidebarOpen prop to Sidebar component for better control of sidebar visibility.
- Updated LayoutShell to manage playground sidebar state, enhancing user experience and navigation consistency.
2026-07-08 10:08:37 +05:30
Anish Sarkar
2fa8b76b56 feat(playground): enhance Playground layout with new sidebar and navigation functions
- Introduced PlaygroundSidebar component for improved navigation within the API Playground.
- Added utility functions for generating navigation items and groups, enhancing modularity and reusability.
- Updated PlaygroundLayoutShell to utilize new navigation functions, streamlining the layout structure.
- Enhanced RoutedSectionShell to support optional desktop navigation, improving responsiveness and user experience.
2026-07-08 10:06:41 +05:30
Anish Sarkar
ba08d6392f
Merge pull request #1585 from AnishSarkar22/fix/ci-ui-changes
feat(ui): Unify notifications, sidebar, and API playground UX
2026-07-08 04:14:49 +05:30
Anish Sarkar
3eb963b8b3 fix(placeholder): update placeholder text for clarity in assistant UI and hero chat demo
- Revised placeholder text to improve readability and user understanding by replacing the dash with a period and adding "Use" for prompts and docs.
2026-07-08 04:09:47 +05:30
Anish Sarkar
248e622478 feat(playground): enhance schema form and artifacts library UI
- Replaced required/optional text with Badge component in schema form for better visual distinction.
- Updated EmptyState in artifacts library to include a more informative and visually appealing layout with an icon and description.
2026-07-08 03:53:41 +05:30
Anish Sarkar
7ba77c6b86 feat(capabilities): add documentation URLs to Google Maps, YouTube, and web crawl capabilities
- Introduced a `docs_url` field to the Google Maps reviews, scrape, YouTube comments, YouTube scrape, and web crawl capabilities for improved documentation access.
- Simplified descriptions for each capability to enhance clarity and user understanding.
2026-07-08 03:38:08 +05:30
Anish Sarkar
3b9c806c90 feat(capabilities): add documentation URL to capabilities and enhance Reddit scrape capability
- Introduced a `docs_url` field to the `Capability` and `CapabilitySummary` classes for better documentation access.
- Updated the `REDDIT_SCRAPE` capability to include a specific documentation link.
- Enhanced the PlaygroundRunner component to display the documentation link when available, improving user guidance.
2026-07-08 03:35:32 +05:30
Anish Sarkar
7268eae7fa refactor(playground): update terminology and enhance UI components for API runs
- Changed label from "Runs" to "API Runs" in PlaygroundLayoutShell and PlaygroundIndex for clarity.
- Improved the API reference UI by adding a ChevronRight icon for expandable sections and enhancing the summary layout.
- Introduced an EndpointCopyButton component for easier copying of API endpoint details, improving user experience.
2026-07-08 03:08:39 +05:30
Anish Sarkar
bcfa3c5ef5 refactor(user-settings): update terminology for auto-reload and credit features
- Replaced instances of "auto-reload" with "top-ups" for consistency and clarity across user settings and pricing sections.
- Enhanced user feedback messages related to credit balance and top-up settings.
- Improved comments and descriptions in the Purchase History and Credit Balance Display components for better understanding.
2026-07-08 02:51:23 +05:30
Anish Sarkar
d54e21e066 feat(runs-table): add alert component for API runs overview
- Introduced an Alert component to provide a concise description of API runs in the workspace.
- Replaced the previous header and description with the new alert for improved clarity and user experience.
2026-07-08 02:37:31 +05:30