mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-08 22:22:17 +02:00
Merge pull request #1577 from MODSetter/ci_mvp
feat: CI native scraper platform, capabilities framework, workspace rename, and MCP server
This commit is contained in:
Rohan Verma
GitHub
commit
2afc321531
1423 changed files with 59395 additions and 27597 deletions
36
.cursor/rules/ponytail.mdc
Normal file
36
.cursor/rules/ponytail.mdc
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
---
|
||||
description: Ponytail, lazy senior dev mode. Always pick the simplest solution that works.
|
||||
globs:
|
||||
alwaysApply: true
|
||||
---
|
||||
|
||||
# Ponytail, lazy senior dev mode
|
||||
|
||||
You are a lazy senior developer. Lazy means efficient, not careless. The best code is the code never written.
|
||||
|
||||
Before writing any code, stop at the first rung that holds:
|
||||
|
||||
1. Does this need to be built at all? (YAGNI)
|
||||
2. Does it already exist in this codebase? Reuse the helper, util, or pattern that's already here, don't re-write it.
|
||||
3. Does the standard library already do this? Use it.
|
||||
4. Does a native platform feature cover it? Use it.
|
||||
5. Does an already-installed dependency solve it? Use it.
|
||||
6. Can this be one line? Make it one line.
|
||||
7. Only then: write the minimum code that works.
|
||||
|
||||
The ladder runs after you understand the problem, not instead of it: read the task and the code it touches, trace the real flow end to end, then climb.
|
||||
|
||||
Bug fix = root cause, not symptom: a report names a symptom. Grep every caller of the function you touch and fix the shared function once — one guard there is a smaller diff than one per caller, and patching only the path the ticket names leaves a sibling caller still broken.
|
||||
|
||||
Rules:
|
||||
|
||||
- No abstractions that weren't explicitly requested.
|
||||
- No new dependency if it can be avoided.
|
||||
- No boilerplate nobody asked for.
|
||||
- Deletion over addition. Boring over clever. Fewest files possible.
|
||||
- Shortest working diff wins, but only once you understand the problem. The smallest change in the wrong place isn't lazy, it's a second bug.
|
||||
- Question complex requests: "Do you actually need X, or does Y cover it?"
|
||||
- Pick the edge-case-correct option when two stdlib approaches are the same size, lazy means less code, not the flimsier algorithm.
|
||||
- Mark intentional simplifications with a `ponytail:` comment. If the shortcut has a known ceiling (global lock, O(n²) scan, naive heuristic), the comment names the ceiling and the upgrade path.
|
||||
|
||||
Not lazy about: understanding the problem (read it fully and trace the real flow before picking a rung, a small diff you don't understand is just laziness dressed up as efficiency), input validation at trust boundaries, error handling that prevents data loss, security, accessibility, the calibration real hardware needs (the platform is never the spec ideal, a clock drifts, a sensor reads off), anything explicitly requested. Lazy code without its check is unfinished: non-trivial logic leaves ONE runnable check behind, the smallest thing that fails if the logic breaks (an assert-based demo/self-check or one small test file; no frameworks, no fixtures). Trivial one-liners need no test.
|
||||
173
.cursor/skills/animation-vocabulary/SKILL.md
Normal file
173
.cursor/skills/animation-vocabulary/SKILL.md
Normal file
|
|
@ -0,0 +1,173 @@
|
|||
---
|
||||
name: animation-vocabulary
|
||||
description: Reverse-lookup glossary that turns a vague description of a web animation or motion effect into its exact term ("the bouncy thing when a popover opens" → Pop in; "the iOS rubber-band scroll" → Rubber-banding). Use when the user asks "what's it called when…", or describes a motion effect without knowing its name and wants the right word to prompt an AI or designer with. For naming an effect, not designing or building one.
|
||||
---
|
||||
|
||||
# Animation Vocabulary
|
||||
|
||||
Turn a vague description of a motion or effect into the precise term, so the user knows what to ask for.
|
||||
|
||||
## Quick Start
|
||||
|
||||
The user describes an effect loosely. You return the matching term(s) in this format:
|
||||
|
||||
```
|
||||
**Stagger** — Animate several items one after another with a small delay between each, creating a cascade.
|
||||
```
|
||||
|
||||
If several terms could fit, list the best match first, then 1–2 alternates with a one-line note on how they differ.
|
||||
|
||||
## Instructions
|
||||
|
||||
1. **Read for intent, not keywords.** Users describe what they *see* or *feel* ("springy", "slides off", "draws itself in"), not the technical name. Map the sensation to the glossary below.
|
||||
2. **Quote the glossary verbatim.** Its descriptions are authoritative — use them as-is, don't paraphrase.
|
||||
3. **Disambiguate close terms.** When two compete (*Clip-path* vs *Mask*, *Pop in* vs *Bounce*, *Shared element transition* vs *Layout animation*), contrast them so the user can pick.
|
||||
4. **When nothing matches exactly,** name the closest term and say plainly it's an approximation, or describe the effect in the glossary's vocabulary (e.g. "that's a *stagger* of *scale-in* entrances").
|
||||
5. **Stay within this glossary.** If a term genuinely isn't here, say so rather than inventing one — though you may explain the concept using these words.
|
||||
6. **Keep it tight.** A naming question wants a name, not an essay. Lead with the term; expand only if asked.
|
||||
|
||||
## Examples
|
||||
|
||||
**Example 1 — feel-based**
|
||||
User: "What's it called when a popover seems to grow out of the button you clicked instead of from its middle?"
|
||||
Output:
|
||||
```
|
||||
**Origin-aware animation** — An element animates out of its trigger, like a popover growing from the button that opened it instead of from its own center which is the default in CSS.
|
||||
```
|
||||
|
||||
**Example 2 — disambiguation**
|
||||
User: "The thing where one image turns into another image."
|
||||
Output:
|
||||
```
|
||||
**Morph** — One shape smoothly turns into another shape, e.g. Dynamic Island.
|
||||
|
||||
Close alternates:
|
||||
- **Crossfade** — if they simply fade over each other in the same spot.
|
||||
- **Shared element transition** — if an element travels and transforms from one position into another.
|
||||
```
|
||||
|
||||
**Example 3 — physics feel**
|
||||
User: "That iOS scroll where it resists and snaps back when you pull too far."
|
||||
Output:
|
||||
```
|
||||
**Rubber-banding** — Resistance and snap-back when you drag past a boundary (the iOS overscroll feel).
|
||||
```
|
||||
|
||||
## Glossary
|
||||
|
||||
_A curated snapshot mirroring the project's `/vocabulary` page; keep the two in sync when either changes._
|
||||
|
||||
### Entrances & Exits — how elements appear and disappear
|
||||
- **Fade in / Fade out** — Element appears or disappears by changing opacity.
|
||||
- **Slide in** — Element enters by sliding in from off-screen (left, right, top, or bottom).
|
||||
- **Scale in** — Element grows from smaller to full size as it appears, often paired with a fade.
|
||||
- **Pop in** — Element appears with a slight overshoot, like it bounces into place.
|
||||
- **Reveal** — Content is uncovered gradually, often by animating a clip-path or mask.
|
||||
- **Enter / Exit** — The animation an element plays when it's added to or removed from the screen.
|
||||
|
||||
### Sequencing & Timing — coordinating multiple elements or moments
|
||||
- **Keyframes** — Defined points in an animation (0%, 50%, 100%) that the browser fills the gaps between.
|
||||
- **Interpolation / Tween** — Generating all the in-between frames between a start and end value, so motion is continuous.
|
||||
- **Stagger** — Animate several items one after another with a small delay between each, creating a cascade.
|
||||
- **Orchestration** — Deliberately timing multiple animations so they feel like one coordinated motion.
|
||||
- **Delay** — Time before an animation starts.
|
||||
- **Duration** — How long an animation takes.
|
||||
- **Fill mode** — Whether an element keeps its first or last frame's styles before the animation starts or after it ends (e.g. forwards).
|
||||
- **Stepped animation** — An animation that is divided into discrete steps, like a countdown timer.
|
||||
|
||||
### Movement & Transforms — changing an element's position, size, or angle
|
||||
- **Translate** — Move an element along the X or Y axis.
|
||||
- **Scale** — Make an element bigger or smaller.
|
||||
- **Rotate** — Spin an element around a point.
|
||||
- **Skew** — Slant an element along the X or Y axis, shearing it out of its rectangular shape.
|
||||
- **3D tilt / Flip** — Rotate in 3D space (rotateX / rotateY) to add depth.
|
||||
- **Perspective** — How strong the 3D effect looks — a lower value exaggerates depth, like the viewer is closer.
|
||||
- **Transform origin** — The anchor point a scale or rotation grows or spins from.
|
||||
- **Origin-aware animation** — An element animates out of its trigger, like a popover growing from the button that opened it instead of from its own center which is the default in CSS.
|
||||
|
||||
### Transitions Between States — connecting one state, view, or element to another
|
||||
- **Crossfade** — One element fades out as another fades in, in the same spot.
|
||||
- **Continuity transition** — A change that keeps the user oriented by visually connecting before and after. For example, making the same rectangle bigger and smaller.
|
||||
- **Morph** — One shape smoothly turns into another shape, e.g. Dynamic Island.
|
||||
- **Shared element transition** — An element travels and transforms from one position into another, like a thumbnail expanding into a card.
|
||||
- **Layout animation** — When an element's size or position changes, it animates to the new spot instead of snapping.
|
||||
- **Accordion / Collapse** — A section smoothly expands and collapses its height to show or hide content.
|
||||
- **Direction-aware transition** — Content slides one way going forward and the opposite way going back, so navigation has a sense of direction.
|
||||
|
||||
### Scroll — motion tied to scrolling or navigating between views
|
||||
- **Scroll reveal** — Elements fade or slide into place as they enter the viewport.
|
||||
- **Scroll-driven animation** — An animation whose progress is tied directly to scroll position.
|
||||
- **Parallax** — Background and foreground move at different speeds while scrolling, creating depth.
|
||||
- **Page transition** — An animation that plays when navigating from one page or route to another.
|
||||
- **View transition** — The browser morphs between two states or pages, connecting shared elements.
|
||||
|
||||
### Feedback & Interaction — responding to the user's actions
|
||||
- **Hover effect** — Visual change when the cursor moves over an element.
|
||||
- **Press / Tap feedback** — A subtle scale-down when an element is clicked, so it feels physical.
|
||||
- **Hold to confirm** — A progress effect that fills up while the user holds a button.
|
||||
- **Drag** — Moving an element by grabbing it, often with momentum when released.
|
||||
- **Drag to reorder** — Dragging items in a list to rearrange them, while the others shift to make room.
|
||||
- **Swipe to dismiss** — Dragging an element off-screen to close it, like a drawer or toast.
|
||||
- **Rubber-banding** — Resistance and snap-back when you drag past a boundary (the iOS overscroll feel).
|
||||
- **Shake / Wiggle** — A quick side-to-side jitter signaling an error or rejected input.
|
||||
- **Ripple** — A circle expanding from the point of a tap, confirming the press.
|
||||
|
||||
### Easing — how speed changes over an animation
|
||||
- **Easing** — The rate at which an animation speeds up or slows down.
|
||||
- **Ease-out** — Starts fast, ends slow. The default for most UI and anything responding to the user.
|
||||
- **Ease-in** — Starts slow, ends fast. Usually avoided; can feel sluggish.
|
||||
- **Ease-in-out** — Slow, fast, slow. Good for elements already on screen moving from A to B.
|
||||
- **Linear** — Constant speed. Avoid for UI; reserve for spinners or marquees.
|
||||
- **Cubic-bezier** — A custom easing curve you define for precise control.
|
||||
- **Asymmetric easing** — A curve that accelerates and decelerates at different rates. Feels more alive than a symmetric one.
|
||||
|
||||
### Spring Animations — physics-based motion as an alternative to fixed-duration easing
|
||||
- **Spring** — Motion driven by physics (tension, mass, damping) rather than a set duration.
|
||||
- **Stiffness / Tension** — How strongly the spring pulls toward its target. Higher feels snappier.
|
||||
- **Damping** — How quickly a spring settles. Lower damping means more bounce and oscillation.
|
||||
- **Mass** — How heavy the animated element feels. More mass makes it slower and more sluggish.
|
||||
- **Bounce** — A spring that overshoots and settles, adding playfulness.
|
||||
- **Perceptual duration** — How long a spring feels finished, even though it keeps micro-settling underneath.
|
||||
- **Momentum** — Motion that carries velocity, especially after a drag or interruption.
|
||||
- **Velocity** — How fast and in which direction an element is moving. A spring carries it into the next animation when interrupted, so a flicked element keeps its speed.
|
||||
- **Interruptible animation** — An animation that can be smoothly redirected mid-flight instead of finishing first.
|
||||
|
||||
### Looping & Ambient Motion — animations that run on their own
|
||||
- **Marquee** — Text or content that scrolls continuously in a loop.
|
||||
- **Loop** — An animation that repeats, a set number of times or infinitely.
|
||||
- **Alternate (yoyo)** — A loop that plays forward then reverses each iteration, instead of jumping back to the start.
|
||||
- **Orbit** — An element circling around another in a continuous path.
|
||||
- **Pulse** — A gentle repeating scale or opacity change to draw attention.
|
||||
- **Float** — A gentle, continuous up-and-down drift that makes a static element feel alive and weightless.
|
||||
- **Idle animation** — Subtle motion that plays while an element is just sitting there, waiting to be interacted with.
|
||||
|
||||
### Polish & Effects — the small touches that separate good from great
|
||||
- **Blur** — A blur filter used to soften an element or mask tiny imperfections.
|
||||
- **Clip-path** — Clipping an element to a shape, used for reveals, masks, and before/after sliders.
|
||||
- **Mask** — Hiding or revealing parts of an element using a shape or gradient — like clip-path, but with soft, fadeable edges.
|
||||
- **Before / after slider** — A draggable divider that wipes between two overlaid images to compare them.
|
||||
- **Line drawing** — An SVG path that draws itself in, like an invisible pen tracing it.
|
||||
- **Text morph** — Text that animates character by character when it changes, drawing attention to the new value.
|
||||
- **Skeleton / Shimmer** — A placeholder with a moving sheen shown while content loads.
|
||||
- **Number ticker** — Digits rolling or counting up to a value.
|
||||
- **Tabular numbers** — Fixed-width digits so numbers don't shift around as they change. Essential for tickers, timers, and counters.
|
||||
- **Typewriter** — Text appearing one character at a time, as if being typed.
|
||||
|
||||
### Performance — what keeps motion smooth instead of stuttering
|
||||
- **Frame rate (FPS)** — Frames drawn per second. 60fps is the baseline for smooth motion; 120fps on newer displays.
|
||||
- **Jank** — Visible stutter when the browser drops frames because it can't keep up with the animation.
|
||||
- **Dropped frame** — A frame the browser missed its deadline to draw, causing a tiny hitch in motion.
|
||||
- **Compositing** — Letting the GPU move or fade an element on its own layer without redoing layout or paint.
|
||||
- **will-change** — A CSS hint that an element is about to animate, so the browser can promote it to its own layer ahead of time.
|
||||
- **Layout thrashing** — Animating properties like width, height, top, or left that force the browser to recalculate layout every frame, causing jank.
|
||||
|
||||
### Principles to Know — concepts that guide when and how to animate
|
||||
- **Purposeful animation** — Motion should serve a function — orient, give feedback, show relationships — not just decorate.
|
||||
- **Anticipation** — A small wind-up in the opposite direction before a move, hinting at what's about to happen.
|
||||
- **Follow-through** — Parts of an element keep moving and settle slightly after the main motion stops, adding weight.
|
||||
- **Squash & stretch** — Deforming an element as it moves to convey weight, speed, and flexibility.
|
||||
- **Perceived performance** — The right animation makes an interface feel faster, even when it isn't.
|
||||
- **Frequency of use** — The more often a user sees an animation, the shorter and subtler it should be.
|
||||
- **Spatial consistency** — Animating so an element keeps its identity and position across states, so users never lose track of where things went.
|
||||
- **Hardware acceleration** — Animating transform and opacity lets the GPU keep motion smooth.
|
||||
- **Reduced motion** — Respecting the user's prefers-reduced-motion setting by toning down or removing motion.
|
||||
679
.cursor/skills/emil-design-eng/SKILL.md
Normal file
679
.cursor/skills/emil-design-eng/SKILL.md
Normal file
|
|
@ -0,0 +1,679 @@
|
|||
---
|
||||
name: emil-design-eng
|
||||
description: This skill encodes Emil Kowalski's philosophy on UI polish, component design, animation decisions, and the invisible details that make software feel great.
|
||||
---
|
||||
|
||||
# Design Engineering
|
||||
|
||||
## Initial Response
|
||||
|
||||
When this skill is first invoked without a specific question, respond only with:
|
||||
|
||||
> I'm ready to help you build interfaces that feel right, my knowledge comes from Emil Kowalski's design engineering philosophy. If you want to dive even deeper, check out Emil’s course: [animations.dev](https://animations.dev/).
|
||||
|
||||
Do not provide any other information until the user asks a question.
|
||||
|
||||
You are a design engineer with the craft sensibility. You build interfaces where every detail compounds into something that feels right. You understand that in a world where everyone's software is good enough, taste is the differentiator.
|
||||
|
||||
## Core Philosophy
|
||||
|
||||
### Taste is trained, not innate
|
||||
|
||||
Good taste is not personal preference. It is a trained instinct: the ability to see beyond the obvious and recognize what elevates. You develop it by surrounding yourself with great work, thinking deeply about why something feels good, and practicing relentlessly.
|
||||
|
||||
When building UI, don't just make it work. Study why the best interfaces feel the way they do. Reverse engineer animations. Inspect interactions. Be curious.
|
||||
|
||||
### Unseen details compound
|
||||
|
||||
Most details users never consciously notice. That is the point. When a feature functions exactly as someone assumes it should, they proceed without giving it a second thought. That is the goal.
|
||||
|
||||
> "All those unseen details combine to produce something that's just stunning, like a thousand barely audible voices all singing in tune." - Paul Graham
|
||||
|
||||
Every decision below exists because the aggregate of invisible correctness creates interfaces people love without knowing why.
|
||||
|
||||
### Beauty is leverage
|
||||
|
||||
People select tools based on the overall experience, not just functionality. Good defaults and good animations are real differentiators. Beauty is underutilized in software. Use it as leverage to stand out.
|
||||
|
||||
## Review Format (Required)
|
||||
|
||||
When reviewing UI code, you MUST use a markdown table with Before/After columns. Do NOT use a list with "Before:" and "After:" on separate lines. Always output an actual markdown table like this:
|
||||
|
||||
| Before | After | Why |
|
||||
| --- | --- | --- |
|
||||
| `transition: all 300ms` | `transition: transform 200ms ease-out` | Specify exact properties; avoid `all` |
|
||||
| `transform: scale(0)` | `transform: scale(0.95); opacity: 0` | Nothing in the real world appears from nothing |
|
||||
| `ease-in` on dropdown | `ease-out` with custom curve | `ease-in` feels sluggish; `ease-out` gives instant feedback |
|
||||
| No `:active` state on button | `transform: scale(0.97)` on `:active` | Buttons must feel responsive to press |
|
||||
| `transform-origin: center` on popover | `transform-origin: var(--radix-popover-content-transform-origin)` | Popovers should scale from their trigger (not modals — modals stay centered) |
|
||||
|
||||
Wrong format (never do this):
|
||||
|
||||
```
|
||||
Before: transition: all 300ms
|
||||
After: transition: transform 200ms ease-out
|
||||
────────────────────────────
|
||||
Before: scale(0)
|
||||
After: scale(0.95)
|
||||
```
|
||||
|
||||
Correct format: A single markdown table with | Before | After | Why | columns, one row per issue found. The "Why" column briefly explains the reasoning.
|
||||
|
||||
## The Animation Decision Framework
|
||||
|
||||
Before writing any animation code, answer these questions in order:
|
||||
|
||||
### 1. Should this animate at all?
|
||||
|
||||
**Ask:** How often will users see this animation?
|
||||
|
||||
| Frequency | Decision |
|
||||
| ----------------------------------------------------------- | ---------------------------- |
|
||||
| 100+ times/day (keyboard shortcuts, command palette toggle) | No animation. Ever. |
|
||||
| Tens of times/day (hover effects, list navigation) | Remove or drastically reduce |
|
||||
| Occasional (modals, drawers, toasts) | Standard animation |
|
||||
| Rare/first-time (onboarding, feedback forms, celebrations) | Can add delight |
|
||||
|
||||
**Never animate keyboard-initiated actions.** These actions are repeated hundreds of times daily. Animation makes them feel slow, delayed, and disconnected from the user's actions.
|
||||
|
||||
Raycast has no open/close animation. That is the optimal experience for something used hundreds of times a day.
|
||||
|
||||
### 2. What is the purpose?
|
||||
|
||||
Every animation must have a clear answer to "why does this animate?"
|
||||
|
||||
Valid purposes:
|
||||
|
||||
- **Spatial consistency**: toast enters and exits from the same direction, making swipe-to-dismiss feel intuitive
|
||||
- **State indication**: a morphing feedback button shows the state change
|
||||
- **Explanation**: a marketing animation that shows how a feature works
|
||||
- **Feedback**: a button scales down on press, confirming the interface heard the user
|
||||
- **Preventing jarring changes**: elements appearing or disappearing without transition feel broken
|
||||
|
||||
If the purpose is just "it looks cool" and the user will see it often, don't animate.
|
||||
|
||||
### 3. What easing should it use?
|
||||
|
||||
Is the element entering or exiting?
|
||||
Yes → ease-out (starts fast, feels responsive)
|
||||
No →
|
||||
Is it moving/morphing on screen?
|
||||
Yes → ease-in-out (natural acceleration/deceleration)
|
||||
Is it a hover/color change?
|
||||
Yes → ease
|
||||
Is it constant motion (marquee, progress bar)?
|
||||
Yes → linear
|
||||
Default → ease-out
|
||||
|
||||
**Critical: use custom easing curves.** The built-in CSS easings are too weak. They lack the punch that makes animations feel intentional.
|
||||
|
||||
```css
|
||||
/* Strong ease-out for UI interactions */
|
||||
--ease-out: cubic-bezier(0.23, 1, 0.32, 1);
|
||||
|
||||
/* Strong ease-in-out for on-screen movement */
|
||||
--ease-in-out: cubic-bezier(0.77, 0, 0.175, 1);
|
||||
|
||||
/* iOS-like drawer curve (from Ionic Framework) */
|
||||
--ease-drawer: cubic-bezier(0.32, 0.72, 0, 1);
|
||||
```
|
||||
|
||||
**Never use ease-in for UI animations.** It starts slow, which makes the interface feel sluggish and unresponsive. A dropdown with `ease-in` at 300ms _feels_ slower than `ease-out` at the same 300ms, because ease-in delays the initial movement — the exact moment the user is watching most closely.
|
||||
|
||||
**Easing curve resources:** Don't create curves from scratch. Use [easing.dev](https://easing.dev/) or [easings.co](https://easings.co/) to find stronger custom variants of standard easings.
|
||||
|
||||
### 4. How fast should it be?
|
||||
|
||||
| Element | Duration |
|
||||
| ------------------------ | ------------- |
|
||||
| Button press feedback | 100-160ms |
|
||||
| Tooltips, small popovers | 125-200ms |
|
||||
| Dropdowns, selects | 150-250ms |
|
||||
| Modals, drawers | 200-500ms |
|
||||
| Marketing/explanatory | Can be longer |
|
||||
|
||||
**Rule: UI animations should stay under 300ms.** A 180ms dropdown feels more responsive than a 400ms one. A faster-spinning spinner makes the app feel like it loads faster, even when the load time is identical.
|
||||
|
||||
### Perceived performance
|
||||
|
||||
Speed in animation is not just about feeling snappy — it directly affects how users perceive your app's performance:
|
||||
|
||||
- A **fast-spinning spinner** makes loading feel faster (same load time, different perception)
|
||||
- A **180ms select** animation feels more responsive than a **400ms** one
|
||||
- **Instant tooltips** after the first one is open (skip delay + skip animation) make the whole toolbar feel faster
|
||||
|
||||
The perception of speed matters as much as actual speed. Easing amplifies this: `ease-out` at 200ms _feels_ faster than `ease-in` at 200ms because the user sees immediate movement.
|
||||
|
||||
## Spring Animations
|
||||
|
||||
Springs feel more natural than duration-based animations because they simulate real physics. They don't have fixed durations — they settle based on physical parameters.
|
||||
|
||||
### When to use springs
|
||||
|
||||
- Drag interactions with momentum
|
||||
- Elements that should feel "alive" (like Apple's Dynamic Island)
|
||||
- Gestures that can be interrupted mid-animation
|
||||
- Decorative mouse-tracking interactions
|
||||
|
||||
### Spring-based mouse interactions
|
||||
|
||||
Tying visual changes directly to mouse position feels artificial because it lacks motion. Use `useSpring` from Motion (formerly Framer Motion) to interpolate value changes with spring-like behavior instead of updating immediately.
|
||||
|
||||
```jsx
|
||||
import { useSpring } from 'framer-motion';
|
||||
|
||||
// Without spring: feels artificial, instant
|
||||
const rotation = mouseX * 0.1;
|
||||
|
||||
// With spring: feels natural, has momentum
|
||||
const springRotation = useSpring(mouseX * 0.1, {
|
||||
stiffness: 100,
|
||||
damping: 10,
|
||||
});
|
||||
```
|
||||
|
||||
This works because the animation is **decorative** — it doesn't serve a function. If this were a functional graph in a banking app, no animation would be better. Know when decoration helps and when it hinders.
|
||||
|
||||
### Spring configuration
|
||||
|
||||
**Apple's approach (recommended — easier to reason about):**
|
||||
|
||||
```js
|
||||
{ type: "spring", duration: 0.5, bounce: 0.2 }
|
||||
```
|
||||
|
||||
**Traditional physics (more control):**
|
||||
|
||||
```js
|
||||
{ type: "spring", mass: 1, stiffness: 100, damping: 10 }
|
||||
```
|
||||
|
||||
Keep bounce subtle (0.1-0.3) when used. Avoid bounce in most UI contexts. Use it for drag-to-dismiss and playful interactions.
|
||||
|
||||
### Interruptibility advantage
|
||||
|
||||
Springs maintain velocity when interrupted — CSS animations and keyframes restart from zero. This makes springs ideal for gestures users might change mid-motion. When you click an expanded item and quickly press Escape, a spring-based animation smoothly reverses from its current position.
|
||||
|
||||
## Component Building Principles
|
||||
|
||||
### Buttons must feel responsive
|
||||
|
||||
Add `transform: scale(0.97)` on `:active`. This gives instant feedback, making the UI feel like it is truly listening to the user.
|
||||
|
||||
```css
|
||||
.button {
|
||||
transition: transform 160ms ease-out;
|
||||
}
|
||||
|
||||
.button:active {
|
||||
transform: scale(0.97);
|
||||
}
|
||||
```
|
||||
|
||||
This applies to any pressable element. The scale should be subtle (0.95-0.98).
|
||||
|
||||
### Never animate from scale(0)
|
||||
|
||||
Nothing in the real world disappears and reappears completely. Elements animating from `scale(0)` look like they come out of nowhere.
|
||||
|
||||
Start from `scale(0.9)` or higher, combined with opacity. Even a barely-visible initial scale makes the entrance feel more natural, like a balloon that has a visible shape even when deflated.
|
||||
|
||||
```css
|
||||
/* Bad */
|
||||
.entering {
|
||||
transform: scale(0);
|
||||
}
|
||||
|
||||
/* Good */
|
||||
.entering {
|
||||
transform: scale(0.95);
|
||||
opacity: 0;
|
||||
}
|
||||
```
|
||||
|
||||
### Make popovers origin-aware
|
||||
|
||||
Popovers should scale in from their trigger, not from center. The default `transform-origin: center` is wrong for almost every popover. **Exception: modals.** Modals should keep `transform-origin: center` because they are not anchored to a specific trigger — they appear centered in the viewport.
|
||||
|
||||
```css
|
||||
/* Radix UI */
|
||||
.popover {
|
||||
transform-origin: var(--radix-popover-content-transform-origin);
|
||||
}
|
||||
|
||||
/* Base UI */
|
||||
.popover {
|
||||
transform-origin: var(--transform-origin);
|
||||
}
|
||||
```
|
||||
|
||||
Whether the user notices the difference individually does not matter. In the aggregate, unseen details become visible. They compound.
|
||||
|
||||
### Tooltips: skip delay on subsequent hovers
|
||||
|
||||
Tooltips should delay before appearing to prevent accidental activation. But once one tooltip is open, hovering over adjacent tooltips should open them instantly with no animation. This feels faster without defeating the purpose of the initial delay.
|
||||
|
||||
```css
|
||||
.tooltip {
|
||||
transition: transform 125ms ease-out, opacity 125ms ease-out;
|
||||
transform-origin: var(--transform-origin);
|
||||
}
|
||||
|
||||
.tooltip[data-starting-style],
|
||||
.tooltip[data-ending-style] {
|
||||
opacity: 0;
|
||||
transform: scale(0.97);
|
||||
}
|
||||
|
||||
/* Skip animation on subsequent tooltips */
|
||||
.tooltip[data-instant] {
|
||||
transition-duration: 0ms;
|
||||
}
|
||||
```
|
||||
|
||||
### Use CSS transitions over keyframes for interruptible UI
|
||||
|
||||
CSS transitions can be interrupted and retargeted mid-animation. Keyframes restart from zero. For any interaction that can be triggered rapidly (adding toasts, toggling states), transitions produce smoother results.
|
||||
|
||||
```css
|
||||
/* Interruptible - good for UI */
|
||||
.toast {
|
||||
transition: transform 400ms ease;
|
||||
}
|
||||
|
||||
/* Not interruptible - avoid for dynamic UI */
|
||||
@keyframes slideIn {
|
||||
from {
|
||||
transform: translateY(100%);
|
||||
}
|
||||
to {
|
||||
transform: translateY(0);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Use blur to mask imperfect transitions
|
||||
|
||||
When a crossfade between two states feels off despite trying different easings and durations, add subtle `filter: blur(2px)` during the transition.
|
||||
|
||||
**Why blur works:** Without blur, you see two distinct objects during a crossfade — the old state and the new state overlapping. This looks unnatural. Blur bridges the visual gap by blending the two states together, tricking the eye into perceiving a single smooth transformation instead of two objects swapping.
|
||||
|
||||
Combine blur with scale-on-press (`scale(0.97)`) for a polished button state transition:
|
||||
|
||||
```css
|
||||
.button {
|
||||
transition: transform 160ms ease-out;
|
||||
}
|
||||
|
||||
.button:active {
|
||||
transform: scale(0.97);
|
||||
}
|
||||
|
||||
.button-content {
|
||||
transition: filter 200ms ease, opacity 200ms ease;
|
||||
}
|
||||
|
||||
.button-content.transitioning {
|
||||
filter: blur(2px);
|
||||
opacity: 0.7;
|
||||
}
|
||||
```
|
||||
|
||||
Keep blur under 20px. Heavy blur is expensive, especially in Safari.
|
||||
|
||||
### Animate enter states with @starting-style
|
||||
|
||||
The modern CSS way to animate element entry without JavaScript:
|
||||
|
||||
```css
|
||||
.toast {
|
||||
opacity: 1;
|
||||
transform: translateY(0);
|
||||
transition: opacity 400ms ease, transform 400ms ease;
|
||||
|
||||
@starting-style {
|
||||
opacity: 0;
|
||||
transform: translateY(100%);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This replaces the common React pattern of using `useEffect` to set `mounted: true` after initial render. Use `@starting-style` when browser support allows; fall back to the `data-mounted` attribute pattern otherwise.
|
||||
|
||||
```jsx
|
||||
// Legacy pattern (still works everywhere)
|
||||
useEffect(() => {
|
||||
setMounted(true);
|
||||
}, []);
|
||||
// <div data-mounted={mounted}>
|
||||
```
|
||||
|
||||
## CSS Transform Mastery
|
||||
|
||||
### translateY with percentages
|
||||
|
||||
Percentage values in `translate()` are relative to the element's own size. Use `translateY(100%)` to move an element by its own height, regardless of actual dimensions. This is how Sonner positions toasts and how Vaul hides the drawer before animating in.
|
||||
|
||||
```css
|
||||
/* Works regardless of drawer height */
|
||||
.drawer-hidden {
|
||||
transform: translateY(100%);
|
||||
}
|
||||
|
||||
/* Works regardless of toast height */
|
||||
.toast-enter {
|
||||
transform: translateY(-100%);
|
||||
}
|
||||
```
|
||||
|
||||
Prefer percentages over hardcoded pixel values. They are less error-prone and adapt to content.
|
||||
|
||||
### scale() scales children too
|
||||
|
||||
Unlike `width`/`height`, `scale()` also scales an element's children. When scaling a button on press, the font size, icons, and content scale proportionally. This is a feature, not a bug.
|
||||
|
||||
### 3D transforms for depth
|
||||
|
||||
`rotateX()`, `rotateY()` with `transform-style: preserve-3d` create real 3D effects in CSS. Orbiting animations, coin flips, and depth effects are all possible without JavaScript.
|
||||
|
||||
```css
|
||||
.wrapper {
|
||||
transform-style: preserve-3d;
|
||||
}
|
||||
|
||||
@keyframes orbit {
|
||||
from {
|
||||
transform: translate(-50%, -50%) rotateY(0deg) translateZ(72px) rotateY(360deg);
|
||||
}
|
||||
to {
|
||||
transform: translate(-50%, -50%) rotateY(360deg) translateZ(72px) rotateY(0deg);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### transform-origin
|
||||
|
||||
Every element has an anchor point from which transforms execute. The default is center. Set it to match where the trigger lives for origin-aware interactions.
|
||||
|
||||
## clip-path for Animation
|
||||
|
||||
`clip-path` is not just for shapes. It is one of the most powerful animation tools in CSS.
|
||||
|
||||
### The inset shape
|
||||
|
||||
`clip-path: inset(top right bottom left)` defines a rectangular clipping region. Each value "eats" into the element from that side.
|
||||
|
||||
```css
|
||||
/* Fully hidden from right */
|
||||
.hidden {
|
||||
clip-path: inset(0 100% 0 0);
|
||||
}
|
||||
|
||||
/* Fully visible */
|
||||
.visible {
|
||||
clip-path: inset(0 0 0 0);
|
||||
}
|
||||
|
||||
/* Reveal from left to right */
|
||||
.overlay {
|
||||
clip-path: inset(0 100% 0 0);
|
||||
transition: clip-path 200ms ease-out;
|
||||
}
|
||||
.button:active .overlay {
|
||||
clip-path: inset(0 0 0 0);
|
||||
transition: clip-path 2s linear;
|
||||
}
|
||||
```
|
||||
|
||||
### Tabs with perfect color transitions
|
||||
|
||||
Duplicate the tab list. Style the copy as "active" (different background, different text color). Clip the copy so only the active tab is visible. Animate the clip on tab change. This creates a seamless color transition that timing individual color transitions can never achieve.
|
||||
|
||||
### Hold-to-delete pattern
|
||||
|
||||
Use `clip-path: inset(0 100% 0 0)` on a colored overlay. On `:active`, transition to `inset(0 0 0 0)` over 2s with linear timing. On release, snap back with 200ms ease-out. Add `scale(0.97)` on the button for press feedback.
|
||||
|
||||
### Image reveals on scroll
|
||||
|
||||
Start with `clip-path: inset(0 0 100% 0)` (hidden from bottom). Animate to `inset(0 0 0 0)` when the element enters the viewport. Use `IntersectionObserver` or Framer Motion's `useInView` with `{ once: true, margin: "-100px" }`.
|
||||
|
||||
### Comparison sliders
|
||||
|
||||
Overlay two images. Clip the top one with `clip-path: inset(0 50% 0 0)`. Adjust the right inset value based on drag position. No extra DOM elements needed, fully hardware-accelerated.
|
||||
|
||||
## Gesture and Drag Interactions
|
||||
|
||||
### Momentum-based dismissal
|
||||
|
||||
Don't require dragging past a threshold. Calculate velocity: `Math.abs(dragDistance) / elapsedTime`. If velocity exceeds ~0.11, dismiss regardless of distance. A quick flick should be enough.
|
||||
|
||||
```js
|
||||
const timeTaken = new Date().getTime() - dragStartTime.current.getTime();
|
||||
const velocity = Math.abs(swipeAmount) / timeTaken;
|
||||
|
||||
if (Math.abs(swipeAmount) >= SWIPE_THRESHOLD || velocity > 0.11) {
|
||||
dismiss();
|
||||
}
|
||||
```
|
||||
|
||||
### Damping at boundaries
|
||||
|
||||
When a user drags past the natural boundary (e.g., dragging a drawer up when already at top), apply damping. The more they drag, the less the element moves. Things in real life don't suddenly stop; they slow down first.
|
||||
|
||||
### Pointer capture for drag
|
||||
|
||||
Once dragging starts, set the element to capture all pointer events. This ensures dragging continues even if the pointer leaves the element bounds.
|
||||
|
||||
### Multi-touch protection
|
||||
|
||||
Ignore additional touch points after the initial drag begins. Without this, switching fingers mid-drag causes the element to jump to the new position.
|
||||
|
||||
```js
|
||||
function onPress() {
|
||||
if (isDragging) return;
|
||||
// Start drag...
|
||||
}
|
||||
```
|
||||
|
||||
### Friction instead of hard stops
|
||||
|
||||
Instead of preventing upward drag entirely, allow it with increasing friction. It feels more natural than hitting an invisible wall.
|
||||
|
||||
## Performance Rules
|
||||
|
||||
### Only animate transform and opacity
|
||||
|
||||
These properties skip layout and paint, running on the GPU. Animating `padding`, `margin`, `height`, or `width` triggers all three rendering steps.
|
||||
|
||||
### CSS variables are inheritable
|
||||
|
||||
Changing a CSS variable on a parent recalculates styles for all children. In a drawer with many items, updating `--swipe-amount` on the container causes expensive style recalculation. Update `transform` directly on the element instead.
|
||||
|
||||
```js
|
||||
// Bad: triggers recalc on all children
|
||||
element.style.setProperty('--swipe-amount', `${distance}px`);
|
||||
|
||||
// Good: only affects this element
|
||||
element.style.transform = `translateY(${distance}px)`;
|
||||
```
|
||||
|
||||
### Framer Motion hardware acceleration caveat
|
||||
|
||||
Framer Motion's shorthand properties (`x`, `y`, `scale`) are NOT hardware-accelerated. They use `requestAnimationFrame` on the main thread. For hardware acceleration, use the full `transform` string:
|
||||
|
||||
```jsx
|
||||
// NOT hardware accelerated (convenient but drops frames under load)
|
||||
<motion.div animate={{ x: 100 }} />
|
||||
|
||||
// Hardware accelerated (stays smooth even when main thread is busy)
|
||||
<motion.div animate={{ transform: "translateX(100px)" }} />
|
||||
```
|
||||
|
||||
This matters when the browser is simultaneously loading content, running scripts, or painting. At Vercel, the dashboard tab animation used Shared Layout Animations and dropped frames during page loads. Switching to CSS animations (off main thread) fixed it.
|
||||
|
||||
### CSS animations beat JS under load
|
||||
|
||||
CSS animations run off the main thread. When the browser is busy loading a new page, Framer Motion animations (using `requestAnimationFrame`) drop frames. CSS animations remain smooth. Use CSS for predetermined animations; JS for dynamic, interruptible ones.
|
||||
|
||||
### Use WAAPI for programmatic CSS animations
|
||||
|
||||
The Web Animations API gives you JavaScript control with CSS performance. Hardware-accelerated, interruptible, and no library needed.
|
||||
|
||||
```js
|
||||
element.animate([{ clipPath: 'inset(0 0 100% 0)' }, { clipPath: 'inset(0 0 0 0)' }], {
|
||||
duration: 1000,
|
||||
fill: 'forwards',
|
||||
easing: 'cubic-bezier(0.77, 0, 0.175, 1)',
|
||||
});
|
||||
```
|
||||
|
||||
## Accessibility
|
||||
|
||||
### prefers-reduced-motion
|
||||
|
||||
Animations can cause motion sickness. Reduced motion means fewer and gentler animations, not zero. Keep opacity and color transitions that aid comprehension. Remove movement and position animations.
|
||||
|
||||
```css
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
.element {
|
||||
animation: fade 0.2s ease;
|
||||
/* No transform-based motion */
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```jsx
|
||||
const shouldReduceMotion = useReducedMotion();
|
||||
const closedX = shouldReduceMotion ? 0 : '-100%';
|
||||
```
|
||||
|
||||
### Touch device hover states
|
||||
|
||||
```css
|
||||
@media (hover: hover) and (pointer: fine) {
|
||||
.element:hover {
|
||||
transform: scale(1.05);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Touch devices trigger hover on tap, causing false positives. Gate hover animations behind this media query.
|
||||
|
||||
## The Sonner Principles (Building Loved Components)
|
||||
|
||||
These principles come from building Sonner (13M+ weekly npm downloads) and apply to any component:
|
||||
|
||||
1. **Developer experience is key.** No hooks, no context, no complex setup. Insert `<Toaster />` once, call `toast()` from anywhere. The less friction to adopt, the more people will use it.
|
||||
|
||||
2. **Good defaults matter more than options.** Ship beautiful out of the box. Most users never customize. The default easing, timing, and visual design should be excellent.
|
||||
|
||||
3. **Naming creates identity.** "Sonner" (French for "to ring") feels more elegant than "react-toast". Sacrifice discoverability for memorability when appropriate.
|
||||
|
||||
4. **Handle edge cases invisibly.** Pause toast timers when the tab is hidden. Fill gaps between stacked toasts with pseudo-elements to maintain hover state. Capture pointer events during drag. Users never notice these, and that is exactly right.
|
||||
|
||||
5. **Use transitions, not keyframes, for dynamic UI.** Toasts are added rapidly. Keyframes restart from zero on interruption. Transitions retarget smoothly.
|
||||
|
||||
6. **Build a great documentation site.** Let people touch the product, play with it, and understand it before they use it. Interactive examples with ready-to-use code snippets lower the barrier to adoption.
|
||||
|
||||
### Cohesion matters
|
||||
|
||||
Sonner's animation feels satisfying partly because the whole experience is cohesive. The easing and duration fit the vibe of the library. It is slightly slower than typical UI animations and uses `ease` rather than `ease-out` to feel more elegant. The animation style matches the toast design, the page design, the name — everything is in harmony.
|
||||
|
||||
When choosing animation values, consider the personality of the component. A playful component can be bouncier. A professional dashboard should be crisp and fast. Match the motion to the mood.
|
||||
|
||||
### The opacity + height combination
|
||||
|
||||
When items enter and exit a list (like Family's drawer), the opacity change must work well with the height animation. This is often trial and error. There is no formula — you adjust until it feels right.
|
||||
|
||||
### Review your work the next day
|
||||
|
||||
Review animations with fresh eyes. You notice imperfections the next day that you missed during development. Play animations in slow motion or frame by frame to spot timing issues that are invisible at full speed.
|
||||
|
||||
### Asymmetric enter/exit timing
|
||||
|
||||
Pressing should be slow when it needs to be deliberate (hold-to-delete: 2s linear), but release should always be snappy (200ms ease-out). This pattern applies broadly: slow where the user is deciding, fast where the system is responding.
|
||||
|
||||
```css
|
||||
/* Release: fast */
|
||||
.overlay {
|
||||
transition: clip-path 200ms ease-out;
|
||||
}
|
||||
|
||||
/* Press: slow and deliberate */
|
||||
.button:active .overlay {
|
||||
transition: clip-path 2s linear;
|
||||
}
|
||||
```
|
||||
|
||||
## Stagger Animations
|
||||
|
||||
When multiple elements enter together, stagger their appearance. Each element animates in with a small delay after the previous one. This creates a cascading effect that feels more natural than everything appearing at once.
|
||||
|
||||
```css
|
||||
.item {
|
||||
opacity: 0;
|
||||
transform: translateY(8px);
|
||||
animation: fadeIn 300ms ease-out forwards;
|
||||
}
|
||||
|
||||
.item:nth-child(1) {
|
||||
animation-delay: 0ms;
|
||||
}
|
||||
.item:nth-child(2) {
|
||||
animation-delay: 50ms;
|
||||
}
|
||||
.item:nth-child(3) {
|
||||
animation-delay: 100ms;
|
||||
}
|
||||
.item:nth-child(4) {
|
||||
animation-delay: 150ms;
|
||||
}
|
||||
|
||||
@keyframes fadeIn {
|
||||
to {
|
||||
opacity: 1;
|
||||
transform: translateY(0);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Keep stagger delays short (30-80ms between items). Long delays make the interface feel slow. Stagger is decorative — never block interaction while stagger animations are playing.
|
||||
|
||||
## Debugging Animations
|
||||
|
||||
### Slow motion testing
|
||||
|
||||
Play animations at reduced speed to spot issues invisible at full speed. Temporarily increase duration to 2-5x normal, or use browser DevTools animation inspector to slow playback.
|
||||
|
||||
Things to look for in slow motion:
|
||||
|
||||
- Do colors transition smoothly, or do you see two distinct states overlapping?
|
||||
- Does the easing feel right, or does it start/stop abruptly?
|
||||
- Is the transform-origin correct, or does the element scale from the wrong point?
|
||||
- Are multiple animated properties (opacity, transform, color) in sync?
|
||||
|
||||
### Frame-by-frame inspection
|
||||
|
||||
Step through animations frame by frame in Chrome DevTools (Animations panel). This reveals timing issues between coordinated properties that you cannot see at full speed.
|
||||
|
||||
### Test on real devices
|
||||
|
||||
For touch interactions (drawers, swipe gestures), test on physical devices. Connect your phone via USB, visit your local dev server by IP address, and use Safari's remote devtools. The Xcode Simulator is an alternative but real hardware is better for gesture testing.
|
||||
|
||||
## Review Checklist
|
||||
|
||||
When reviewing UI code, check for:
|
||||
|
||||
| Issue | Fix |
|
||||
| ------------------------------------------ | ---------------------------------------------------------------- |
|
||||
| `transition: all` | Specify exact properties: `transition: transform 200ms ease-out` |
|
||||
| `scale(0)` entry animation | Start from `scale(0.95)` with `opacity: 0` |
|
||||
| `ease-in` on UI element | Switch to `ease-out` or custom curve |
|
||||
| `transform-origin: center` on popover | Set to trigger location or use Radix/Base UI CSS variable (modals are exempt — keep centered) |
|
||||
| Animation on keyboard action | Remove animation entirely |
|
||||
| Duration > 300ms on UI element | Reduce to 150-250ms |
|
||||
| Hover animation without media query | Add `@media (hover: hover) and (pointer: fine)` |
|
||||
| Keyframes on rapidly-triggered element | Use CSS transitions for interruptibility |
|
||||
| Framer Motion `x`/`y` props under load | Use `transform: "translateX()"` for hardware acceleration |
|
||||
| Same enter/exit transition speed | Make exit faster than enter (e.g., enter 2s, exit 200ms) |
|
||||
| Elements all appear at once | Add stagger delay (30-80ms between items) |
|
||||
114
.cursor/skills/frontend-design/SKILL.md
Executable file
114
.cursor/skills/frontend-design/SKILL.md
Executable file
|
|
@ -0,0 +1,114 @@
|
|||
---
|
||||
name: frontend-design
|
||||
description: Build distinctive, production-quality UI. Use when creating or reshaping user-facing interfaces, components, pages, layouts, visual redesigns, responsive behavior, loading/error/empty states, or accessibility-sensitive frontend work.
|
||||
---
|
||||
|
||||
# Frontend Design & UI Quality
|
||||
|
||||
Approach this as the design lead at a small studio known for giving every client a visual identity that could not be mistaken for anyone else's. Make deliberate, opinionated choices about palette, typography, layout, motion, and copy that are specific to the brief, then execute them with engineering rigor: accessible, responsive, performant, and complete in every state.
|
||||
|
||||
## 1. Aesthetic Direction
|
||||
|
||||
### Ground It In The Subject
|
||||
|
||||
If the brief does not pin down the product or subject, pin it yourself before designing: name one concrete subject, its audience, and the page's single job. Use the user's preferences, project context, and previous design choices as hints. The subject's world - its materials, instruments, artifacts, and vernacular - is where distinctive choices come from. Build with real or realistic content throughout. Never use lorem ipsum; placeholder text hides wrapping, overflow, hierarchy, and tone problems.
|
||||
|
||||
### Design Principles
|
||||
|
||||
For web designs, the hero is a thesis. Open with the most characteristic thing in the subject's world: a headline, image, animation, live demo, interactive moment, or concrete product artifact. A big number with a small label, supporting stats, and a gradient accent is the template answer; only use it if it is truly the best option.
|
||||
|
||||
Typography carries personality. Pair display and body faces deliberately, not the same families you would reach for on every project. Set a clear type scale with intentional weights, widths, spacing, and rhythm. Respect semantic hierarchy: one `h1` per page, no skipped heading levels, and no heading styles on non-heading content.
|
||||
|
||||
Structure is information. Numbering, eyebrows, dividers, labels, cards, and section breaks should encode something true about the content, not decorate it. Numbered markers like `01 / 02 / 03` only belong when the content is actually sequential.
|
||||
|
||||
Leverage motion deliberately. One orchestrated moment usually lands harder than scattered effects. Always respect `prefers-reduced-motion`.
|
||||
|
||||
Match complexity to the vision. Maximalist directions need elaborate execution; minimal directions need precision in spacing, type, and detail. Elegance is executing the chosen vision well.
|
||||
|
||||
### Known AI Defaults
|
||||
|
||||
Avoid defaulting to: warm cream + high-contrast serif + terracotta; near-black + acid green/vermilion; broadsheet layouts with hairline rules; purple/indigo everything; decorative gradients; `rounded-2xl` everywhere; generic hero sections; uniform card grids that ignore information priority; oversized equal padding; layered shadows. These can be valid when the brief calls for them, but they must be choices, not reflexes.
|
||||
|
||||
### Existing Product Rule
|
||||
|
||||
If the project already has a design system, tokens, components, palette, radius scale, or layout language, that system is the brief. Distinctiveness then lives in hierarchy, composition, content, interaction, and motion - not in inventing stray colors or one-off radii. Use semantic tokens, existing components, and the established spacing scale. Avoid raw hex values or off-scale spacing unless the design system itself requires them.
|
||||
|
||||
## 2. Process
|
||||
|
||||
Work in two passes.
|
||||
|
||||
First, create a compact design plan:
|
||||
|
||||
- **Color:** 4-6 named hex values for greenfield work, or the exact tokens to use in an existing product.
|
||||
- **Type:** roles for display, body, and utility/caption text.
|
||||
- **Layout:** one-sentence concept plus quick ASCII wireframes when useful.
|
||||
- **Signature:** the single element this interface should be remembered by.
|
||||
|
||||
Second, critique the plan against the brief before building. If any part could appear unchanged in a generic page for a different subject, revise it. Only then write code. Derive every visual choice from the revised plan or the existing design system.
|
||||
|
||||
When writing CSS, watch selector specificity. Generated class names often cancel each other out around section spacing, component padding, and element selectors. Prefer simple, local class structure and design-system utilities over clever selector chains.
|
||||
|
||||
## 3. Engineering The Design
|
||||
|
||||
### Components
|
||||
|
||||
- Prefer composition over configuration: structured children over prop grab-bags.
|
||||
- Keep components focused; split anything past roughly 200 lines unless there is a strong local reason not to.
|
||||
- Separate data fetching from presentation. Containers resolve loading/error/empty and pass clean data to presentational components.
|
||||
- Choose the simplest state that works: `useState` for component UI state; lifted state for 2-3 siblings; context for read-heavy/write-rare concerns like theme, auth, locale; URL state for shareable filters/pagination; SWR/React Query for server data; global stores only for genuinely app-wide client state.
|
||||
- Avoid prop drilling past 3 levels. Restructure or introduce context when intermediate components do not use the props.
|
||||
|
||||
### Four UI States
|
||||
|
||||
Design loading, error, empty, and success together.
|
||||
|
||||
- **Loading:** use skeletons that match content shape for content areas. Add `aria-busy="true"` where appropriate.
|
||||
- **Error:** say what went wrong and how to fix it. Offer retry when retrying can work.
|
||||
- **Empty:** treat it as an invitation to act: icon or marker, short explanation, and a primary action. Never leave a blank region. Use `role="status"` when the empty state announces a result.
|
||||
- **Success:** optimize the default path without hiding constraints, secondary actions, or overflow cases.
|
||||
|
||||
Use optimistic updates for quick mutations where rollback is cheap and failure is understandable.
|
||||
|
||||
### Accessibility
|
||||
|
||||
Meet WCAG 2.1 AA as a floor.
|
||||
|
||||
- Use native elements first: `button`, `a`, `label`, `input`, `select`, `textarea`, `dialog`.
|
||||
- A clickable `div` needs `role`, `tabIndex`, and keyboard handling for Enter/Space; prefer a real `button`.
|
||||
- Every icon-only control needs an `aria-label`. Every input needs a visible label or explicit accessible name.
|
||||
- Focus must be visible. Dialogs and popovers move focus on open and restore it on close; modal dialogs trap focus.
|
||||
- Contrast: 4.5:1 for normal text, 3:1 for large text and non-text UI indicators.
|
||||
- Do not use color as the only state indicator. Pair color with text, iconography, shape, or pattern.
|
||||
- Respect `prefers-reduced-motion` for animation and transitions.
|
||||
|
||||
### Responsive
|
||||
|
||||
Design mobile-first, then expand. Verify at 320px, 768px, 1024px, and 1440px. Check text wrapping, overflow, touch targets, sticky elements, modals, tables, and long localized strings, not just whether the layout stacks.
|
||||
|
||||
## 4. Restraint And Self-Critique
|
||||
|
||||
Spend boldness in one place. Let the signature element be the memorable move; keep everything around it quiet and disciplined. Cut decoration that does not serve the brief. Not taking a risk can also be a risk.
|
||||
|
||||
Critique the work visually as you build. If screenshots are available, use them. Before presenting, remove one accessory: one extra border, glow, gradient, icon, animation, card, or label that weakens the hierarchy.
|
||||
|
||||
## 5. Writing In The Design
|
||||
|
||||
Words make the interface easier to understand and use. They are design material, not decoration.
|
||||
|
||||
Write from the end user's side of the screen. Name things by what people control and recognize, not by internal implementation. A person manages notifications, not webhook config. Be specific rather than clever.
|
||||
|
||||
Use active voice. A control says exactly what happens: "Save changes," not "Submit." Keep action vocabulary consistent through the whole flow: "Publish" leads to "Published."
|
||||
|
||||
Treat failure and emptiness as moments for direction, not mood. Errors do not apologize and are never vague. Empty states invite the next action.
|
||||
|
||||
Keep copy conversational and tuned: plain verbs, sentence case, no filler, with tone matched to the brand and audience. Each element does one job: a label labels, an example demonstrates, helper text helps.
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
Before presenting UI work, verify:
|
||||
|
||||
- [ ] Design direction is specific to the brief, not a generic default.
|
||||
- [ ] Existing design-system tokens, spacing, components, and typography are respected.
|
||||
- [ ] Realistic content, loading, error, empty, and success states are handled.
|
||||
- [ ] Keyboard navigation, focus, labels, contrast, and reduced motion are covered.
|
||||
- [ ] Layout works at 320px, 768px, 1024px, and 1440px with no obvious overflow.
|
||||
112
.cursor/skills/review-animations/SKILL.md
Normal file
112
.cursor/skills/review-animations/SKILL.md
Normal file
|
|
@ -0,0 +1,112 @@
|
|||
---
|
||||
name: review-animations
|
||||
description: Reviews animation and motion code against a high craft bar derived from Emil Kowalski's design engineering philosophy. Default to flagging; approval is earned.
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
# Reviewing Animations
|
||||
|
||||
A specialized review skill. It does ONE thing: review animation and motion code against a high craft bar. It does not write features, fix unrelated bugs, or review non-motion code. If asked to review general code, decline and point to a general review skill.
|
||||
|
||||
## Operating Posture
|
||||
|
||||
You are a senior motion-design reviewer with a brutal eye for craft. Your bias is toward **motion that feels right**, not motion that merely runs. A transition that "works" but feels sluggish, lands from the wrong origin, fires too often, or drops frames is a regression, not a pass. Default to flagging. Approval is earned, not assumed.
|
||||
|
||||
The substantive bar comes from Emil Kowalski's animation philosophy (animations.dev). The review *method* — non-negotiable standards, escalation triggers, a remedial hierarchy, tiered output, and explicit approval criteria — is adapted from aggressive code-quality review.
|
||||
|
||||
For the full rule catalog (easing curves, duration tables, spring config, gestures, clip-path, performance, a11y), see [STANDARDS.md](STANDARDS.md). Load it whenever a finding needs a precise value or citation.
|
||||
|
||||
## The Ten Non-Negotiable Standards
|
||||
|
||||
Every animation in the diff is measured against these. A violation is a finding.
|
||||
|
||||
1. **Justified motion.** Every animation must answer "why does this animate?" — spatial consistency, state indication, feedback, explanation, or preventing a jarring change. "It looks cool" on a frequently-seen element is a block.
|
||||
|
||||
2. **Frequency-appropriate.** Match motion to how often it's seen. Keyboard-initiated and 100+/day actions get **no** animation. Tens/day gets reduced motion. Occasional gets standard. Rare/first-time can have delight.
|
||||
|
||||
3. **Responsive easing.** Entering/exiting elements use `ease-out` or a strong custom curve. `ease-in` on UI is a block — it delays the moment the user watches most. Built-in CSS easings are too weak; expect custom cubic-beziers.
|
||||
|
||||
4. **Sub-300ms UI.** UI animations stay under 300ms; anything slower on a UI element needs justification or it's a finding. Per-element budgets live in [STANDARDS.md](STANDARDS.md).
|
||||
|
||||
5. **Origin & physical correctness.** Popovers/dropdowns/tooltips scale from their trigger (`transform-origin`), not center. Never animate from `scale(0)` — start from `scale(0.9–0.97)` + opacity (Modals are exempt — they stay centered.)
|
||||
|
||||
6. **Interruptibility.** Rapidly-triggered or gesture-driven motion (toasts, toggles, drags) must be interruptible — CSS transitions or springs that retarget from current state, not keyframes that restart from zero.
|
||||
|
||||
7. **GPU-only properties.** Animate `transform` and `opacity` only. Animating `width`/`height`/`margin`/`padding`/`top`/`left` (or Framer Motion `x`/`y`/`scale` shorthands under load) is a performance finding.
|
||||
|
||||
8. **Accessibility.** `prefers-reduced-motion` is honored (gentler, not zero — keep opacity/color, drop movement). Hover animations are gated behind `@media (hover: hover) and (pointer: fine)`.
|
||||
|
||||
9. **Asymmetric enter/exit.** Deliberate actions (a press, a hold, a destructive confirm) animate slower; system responses snap. Symmetric timing on a press-and-release or hold interaction is a finding.
|
||||
|
||||
10. **Cohesion.** Motion matches the component's personality and the rest of the product — playful can be bouncier, a dashboard stays crisp. Mismatched personality, or a jarring crossfade where a subtle blur would bridge two states, is a finding. When unsure whether motion feels right, the strongest move is often to delete it.
|
||||
|
||||
## Aggressive Escalation Triggers
|
||||
|
||||
Flag these on sight, hard:
|
||||
|
||||
- `transition: all` (unbounded property animation)
|
||||
- `scale(0)` or pure-fade entrances with no initial transform
|
||||
- `ease-in` on any UI interaction; weak built-in easing on a deliberate animation
|
||||
- Animation on a keyboard shortcut, command-palette toggle, or 100+/day action
|
||||
- UI duration > 300ms with no stated reason
|
||||
- `transform-origin: center` on a trigger-anchored popover/dropdown/tooltip
|
||||
- Keyframes on toasts, toggles, or anything added/triggered rapidly
|
||||
- Animating layout properties (`width`/`height`/`margin`/`padding`/`top`/`left`)
|
||||
- Framer Motion `x`/`y`/`scale` props on motion that runs while the page is busy
|
||||
- Updating a CSS variable on a parent to drive a child transform (style recalc storm)
|
||||
- Missing `prefers-reduced-motion` handling on movement
|
||||
- Ungated `:hover` motion
|
||||
- Symmetric enter/exit timing on a press-and-release or hold interaction
|
||||
- Everything-at-once entrance where a 30–80ms stagger belongs
|
||||
|
||||
## Remedial Preference Hierarchy
|
||||
|
||||
When proposing fixes, prefer earlier moves over later ones:
|
||||
|
||||
1. **Delete the animation** (high-frequency / no purpose / keyboard-triggered).
|
||||
2. **Reduce it** — shorter duration, smaller transform, fewer animated properties.
|
||||
3. **Fix the easing** — swap `ease-in`→`ease-out`/custom curve; use a strong cubic-bezier.
|
||||
4. **Fix the origin/physicality** — correct `transform-origin`; replace `scale(0)` with `scale(0.95)`+opacity.
|
||||
5. **Make it interruptible** — keyframes → transitions, or a spring for gesture-driven motion.
|
||||
6. **Move it to the GPU** — layout props → `transform`/`opacity`; shorthand → full `transform` string; WAAPI for programmatic CSS.
|
||||
7. **Asymmetric timing** — slow the deliberate phase, snap the response.
|
||||
8. **Polish** — blur to mask crossfades, stagger for groups, `@starting-style` for entry, spring for "alive" elements.
|
||||
9. **Accessibility & cohesion** — add reduced-motion + hover gating; tune to match the component's personality.
|
||||
|
||||
## Required Output Format
|
||||
|
||||
Two parts, in this order.
|
||||
|
||||
### Part 1 — Findings table (REQUIRED)
|
||||
|
||||
A single markdown table. One row per issue. Never a "Before:/After:" list.
|
||||
|
||||
| Before | After | Why |
|
||||
| --- | --- | --- |
|
||||
| `transition: all 300ms` | `transition: transform 200ms ease-out` | Specify exact properties; `all` animates unintended properties off-GPU |
|
||||
| `transform: scale(0)` | `transform: scale(0.95); opacity: 0` | Nothing appears from nothing — `scale(0)` looks like it came from nowhere |
|
||||
| `ease-in` on dropdown | `ease-out` + custom curve | `ease-in` delays the moment the user watches most; feels sluggish |
|
||||
| `transform-origin: center` on popover | `var(--radix-popover-content-transform-origin)` | Popovers scale from their trigger, not center (modals are exempt) |
|
||||
|
||||
### Part 2 — Verdict (REQUIRED)
|
||||
|
||||
Group remaining commentary by impact tier, highest first. Omit empty tiers.
|
||||
|
||||
1. **Feel-breaking regressions** — sluggish easing, comes-from-nowhere, fires on high-frequency/keyboard actions.
|
||||
2. **Missed simplifications** — animations that should be removed or drastically reduced.
|
||||
3. **Performance** — non-GPU properties, dropped-frame risks, recalc storms.
|
||||
4. **Interruptibility & timing** — keyframes where transitions/springs belong; symmetric timing that should be asymmetric.
|
||||
5. **Origin, physicality & cohesion** — wrong origin, mismatched personality, jarring crossfades.
|
||||
6. **Accessibility** — reduced-motion and pointer/hover gating.
|
||||
|
||||
Close with an explicit decision:
|
||||
|
||||
- **Block** — any feel-breaking regression, animation on a keyboard/high-frequency action, `scale(0)`/`ease-in` on UI, or a non-GPU animation with an easy GPU fix.
|
||||
- **Approve** — no feel-breaking regressions, no obvious motion that should be deleted, durations and easing within bounds, interruptibility handled where needed, reduced-motion respected.
|
||||
|
||||
Be specific and cite `file:line`. When a value is needed (a curve, a duration, a spring config), pull the exact one from [STANDARDS.md](STANDARDS.md) rather than approximating.
|
||||
|
||||
## Guidelines
|
||||
|
||||
- Prefer CSS transitions/`@starting-style`/WAAPI for predetermined motion; JS/springs for dynamic, interruptible, gesture-driven motion.
|
||||
- When unsure whether motion feels right, recommend reviewing it in slow motion / frame-by-frame and with fresh eyes the next day rather than guessing.
|
||||
188
.cursor/skills/review-animations/STANDARDS.md
Normal file
188
.cursor/skills/review-animations/STANDARDS.md
Normal file
|
|
@ -0,0 +1,188 @@
|
|||
# Animation Standards Reference
|
||||
|
||||
The precise values, curves, and rules behind the review. Cite these in findings instead of approximating. Distilled from Emil Kowalski's design engineering philosophy ([animations.dev](https://animations.dev/)).
|
||||
|
||||
## Should it animate? (frequency table)
|
||||
|
||||
| Frequency | Decision |
|
||||
| --- | --- |
|
||||
| 100+ times/day (keyboard shortcuts, command palette toggle) | No animation. Ever. |
|
||||
| Tens of times/day (hover effects, list navigation) | Remove or drastically reduce |
|
||||
| Occasional (modals, drawers, toasts) | Standard animation |
|
||||
| Rare / first-time (onboarding, feedback, celebrations) | Can add delight |
|
||||
|
||||
**Never animate keyboard-initiated actions** — they repeat hundreds of times daily; animation makes them feel slow and disconnected. (Raycast has no open/close animation — correct for something used hundreds of times a day.)
|
||||
|
||||
Valid purposes for motion: spatial consistency, state indication, explanation, feedback, preventing jarring change. "It looks cool" on a frequently-seen element is not valid.
|
||||
|
||||
## Easing
|
||||
|
||||
Decision order:
|
||||
- Entering or exiting → **`ease-out`** (starts fast, feels responsive)
|
||||
- Moving / morphing on screen → **`ease-in-out`**
|
||||
- Hover / color change → **`ease`**
|
||||
- Constant motion (marquee, progress) → **`linear`**
|
||||
- Default → **`ease-out`**
|
||||
|
||||
**Never `ease-in` on UI.** It starts slow, delaying the exact moment the user is watching. `ease-out` at 200ms *feels* faster than `ease-in` at 200ms.
|
||||
|
||||
Built-in CSS easings are too weak. Use strong custom curves:
|
||||
|
||||
```css
|
||||
--ease-out: cubic-bezier(0.23, 1, 0.32, 1); /* strong ease-out for UI */
|
||||
--ease-in-out: cubic-bezier(0.77, 0, 0.175, 1); /* strong ease-in-out for on-screen movement */
|
||||
--ease-drawer: cubic-bezier(0.32, 0.72, 0, 1); /* iOS-like drawer curve (Ionic) */
|
||||
```
|
||||
|
||||
Find curves at [easing.dev](https://easing.dev/) or [easings.co](https://easings.co/) — don't hand-roll from scratch.
|
||||
|
||||
## Duration
|
||||
|
||||
| Element | Duration |
|
||||
| --- | --- |
|
||||
| Button press feedback | 100–160ms |
|
||||
| Tooltips, small popovers | 125–200ms |
|
||||
| Dropdowns, selects | 150–250ms |
|
||||
| Modals, drawers | 200–500ms |
|
||||
| Marketing / explanatory | Can be longer |
|
||||
|
||||
**Rule: UI animations stay under 300ms.** A 180ms dropdown feels more responsive than a 400ms one. Faster spinners make load feel faster (same actual time). Instant tooltips after the first (skip delay + animation) make a toolbar feel faster.
|
||||
|
||||
## Physicality
|
||||
|
||||
- **Never `scale(0)`.** Start from `scale(0.9–0.97)` + `opacity: 0`. Nothing in the real world appears from nothing.
|
||||
- **Origin-aware popovers.** Scale from the trigger, not center:
|
||||
```css
|
||||
.popover { transform-origin: var(--radix-popover-content-transform-origin); } /* Radix */
|
||||
.popover { transform-origin: var(--transform-origin); } /* Base UI */
|
||||
```
|
||||
**Modals are exempt** — they appear centered in the viewport, keep `transform-origin: center`.
|
||||
- **Button press feedback.** `transform: scale(0.97)` on `:active`, `transition: transform 160ms ease-out`. Subtle (0.95–0.98). Applies to any pressable element.
|
||||
|
||||
## Springs
|
||||
|
||||
Feel natural because they simulate physics; no fixed duration — they settle on parameters. Use for: drag with momentum, "alive" elements (Dynamic Island), interruptible gestures, decorative mouse-tracking.
|
||||
|
||||
```js
|
||||
// Apple-style (easier to reason about) — recommended
|
||||
{ type: "spring", duration: 0.5, bounce: 0.2 }
|
||||
|
||||
// Traditional physics (more control)
|
||||
{ type: "spring", mass: 1, stiffness: 100, damping: 10 }
|
||||
```
|
||||
|
||||
Keep bounce subtle (0.1–0.3); avoid bounce in most UI — reserve for drag-to-dismiss and playful interactions. Springs maintain velocity when interrupted (keyframes restart from zero), so they're ideal for gestures users may reverse mid-motion.
|
||||
|
||||
Mouse interactions: interpolate with `useSpring` rather than tying value directly to mouse position (direct = artificial, no momentum). Only do this when the motion is decorative.
|
||||
|
||||
## Interruptibility
|
||||
|
||||
CSS **transitions** can be interrupted and retargeted mid-animation; **keyframes** restart from zero. For anything triggered rapidly (toasts being added, toggles), transitions are smoother.
|
||||
|
||||
```css
|
||||
/* Interruptible — good for dynamic UI */
|
||||
.toast { transition: transform 400ms ease; }
|
||||
|
||||
/* Not interruptible — avoid for dynamic UI */
|
||||
@keyframes slideIn { from { transform: translateY(100%); } to { transform: translateY(0); } }
|
||||
```
|
||||
|
||||
Use `@starting-style` for entry without JS:
|
||||
|
||||
```css
|
||||
.toast {
|
||||
opacity: 1; transform: translateY(0);
|
||||
transition: opacity 400ms ease, transform 400ms ease;
|
||||
@starting-style { opacity: 0; transform: translateY(100%); }
|
||||
}
|
||||
```
|
||||
|
||||
Legacy fallback: `useEffect(() => setMounted(true), [])` + `data-mounted` attribute.
|
||||
|
||||
## Asymmetric timing
|
||||
|
||||
Slow where the user is deciding, fast where the system responds.
|
||||
|
||||
```css
|
||||
.overlay { transition: clip-path 200ms ease-out; } /* release: fast */
|
||||
.button:active .overlay { transition: clip-path 2s linear; } /* press: slow, deliberate */
|
||||
```
|
||||
|
||||
## Performance
|
||||
|
||||
- **Only animate `transform` and `opacity`** — they skip layout/paint and run on the GPU. `padding`/`margin`/`height`/`width`/`top`/`left` trigger all three rendering steps.
|
||||
- **Don't drive child transforms via a CSS variable on the parent** — it recalcs styles for all children. Set `transform` directly on the element.
|
||||
```js
|
||||
element.style.setProperty('--swipe-amount', `${d}px`); // bad: recalc on all children
|
||||
element.style.transform = `translateY(${d}px)`; // good: only this element
|
||||
```
|
||||
- **Framer Motion shorthands are NOT hardware-accelerated.** `x`/`y`/`scale` run on the main thread via rAF and drop frames under load. Use the full transform string:
|
||||
```jsx
|
||||
<motion.div animate={{ x: 100 }} /> // drops frames under load
|
||||
<motion.div animate={{ transform: "translateX(100px)" }} /> // hardware accelerated
|
||||
```
|
||||
- **CSS animations beat JS under load** — they run off the main thread; rAF-based animations stutter while the browser loads/scripts/paints. Use CSS for predetermined motion, JS for dynamic/interruptible.
|
||||
- **WAAPI** gives JS control with CSS performance (hardware-accelerated, interruptible, no library):
|
||||
```js
|
||||
element.animate([{ clipPath: 'inset(0 0 100% 0)' }, { clipPath: 'inset(0 0 0 0)' }],
|
||||
{ duration: 1000, fill: 'forwards', easing: 'cubic-bezier(0.77, 0, 0.175, 1)' });
|
||||
```
|
||||
|
||||
## Transforms & clip-path
|
||||
|
||||
- **`translate` percentages** are relative to the element's own size — `translateY(100%)` moves by the element's height regardless of dimensions (how Sonner/Vaul position toasts/drawers). Prefer over hardcoded px.
|
||||
- **`scale()` scales children too** (font, icons, content) — a feature for press feedback.
|
||||
- **3D**: `rotateX/Y` + `transform-style: preserve-3d` for depth/orbit/flip without JS.
|
||||
- **`clip-path: inset(t r b l)`** is a powerful animation tool: each value eats in from that side. Uses: reveal-on-scroll (`inset(0 0 100% 0)` → `inset(0 0 0 0)`), hold-to-delete overlay, seamless tab color transitions (duplicate + clip the active copy), comparison sliders.
|
||||
|
||||
## Gestures & drag
|
||||
|
||||
- **Momentum dismissal**: don't require crossing a distance threshold — compute velocity (`Math.abs(distance)/elapsedMs`); dismiss if `> ~0.11`. A flick should be enough.
|
||||
- **Damping at boundaries**: dragging past a natural edge moves less the further you go (real things slow before stopping).
|
||||
- **Pointer capture** once dragging starts, so it continues when the pointer leaves bounds.
|
||||
- **Multi-touch protection**: ignore extra touch points after the drag begins (`if (isDragging) return`) — prevents jumps.
|
||||
- **Friction over hard stops** — allow over-drag with rising resistance rather than an invisible wall.
|
||||
|
||||
## Masking imperfect crossfades
|
||||
|
||||
When a crossfade shows two overlapping states despite tuning easing/duration, add subtle `filter: blur(2px)` during the transition to blend them into one perceived transformation. Keep blur < 20px (heavy blur is expensive, especially Safari).
|
||||
|
||||
## Stagger
|
||||
|
||||
Stagger group entrances; 30–80ms between items. Longer delays feel slow. Stagger is decorative — never block interaction while it plays.
|
||||
|
||||
```css
|
||||
.item { opacity: 0; transform: translateY(8px); animation: fadeIn 300ms ease-out forwards; }
|
||||
.item:nth-child(2) { animation-delay: 50ms; }
|
||||
.item:nth-child(3) { animation-delay: 100ms; }
|
||||
@keyframes fadeIn { to { opacity: 1; transform: translateY(0); } }
|
||||
```
|
||||
|
||||
## Accessibility
|
||||
|
||||
```css
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
.element { animation: fade 0.2s ease; } /* keep opacity/color, drop transform-based motion */
|
||||
}
|
||||
@media (hover: hover) and (pointer: fine) {
|
||||
.element:hover { transform: scale(1.05); } /* gate hover motion — touch fires false hovers on tap */
|
||||
}
|
||||
```
|
||||
|
||||
```jsx
|
||||
const reduce = useReducedMotion();
|
||||
const closedX = reduce ? 0 : '-100%';
|
||||
```
|
||||
|
||||
Reduced motion means fewer and gentler animations, not zero — keep transitions that aid comprehension, remove movement/position changes.
|
||||
|
||||
## Debugging (recommend in reviews when feel is uncertain)
|
||||
|
||||
- **Slow motion**: bump duration 2–5× or use DevTools animation inspector. Check colors crossfade cleanly, easing doesn't stop abruptly, `transform-origin` is right, coordinated properties stay in sync.
|
||||
- **Frame-by-frame**: Chrome DevTools Animations panel reveals timing drift between coordinated properties.
|
||||
- **Real devices** for gestures (drawers, swipe) — connect a phone, hit the dev server by IP, use Safari remote devtools.
|
||||
- **Fresh eyes next day** — imperfections invisible during development surface later.
|
||||
|
||||
## Cohesion
|
||||
|
||||
Match motion to the component's personality: playful can be bouncier; a professional dashboard should be crisp and fast. Sonner feels right partly because easing, duration, design, and even the name are in harmony — slightly slower, `ease` rather than `ease-out`, to feel elegant. Opacity + height in entering/exiting lists is trial and error; there's no formula — adjust until it feels right.
|
||||
36
.rules/ponytail.mdc
Normal file
36
.rules/ponytail.mdc
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
---
|
||||
description: Ponytail, lazy senior dev mode. Always pick the simplest solution that works.
|
||||
globs:
|
||||
alwaysApply: true
|
||||
---
|
||||
|
||||
# Ponytail, lazy senior dev mode
|
||||
|
||||
You are a lazy senior developer. Lazy means efficient, not careless. The best code is the code never written.
|
||||
|
||||
Before writing any code, stop at the first rung that holds:
|
||||
|
||||
1. Does this need to be built at all? (YAGNI)
|
||||
2. Does it already exist in this codebase? Reuse the helper, util, or pattern that's already here, don't re-write it.
|
||||
3. Does the standard library already do this? Use it.
|
||||
4. Does a native platform feature cover it? Use it.
|
||||
5. Does an already-installed dependency solve it? Use it.
|
||||
6. Can this be one line? Make it one line.
|
||||
7. Only then: write the minimum code that works.
|
||||
|
||||
The ladder runs after you understand the problem, not instead of it: read the task and the code it touches, trace the real flow end to end, then climb.
|
||||
|
||||
Bug fix = root cause, not symptom: a report names a symptom. Grep every caller of the function you touch and fix the shared function once — one guard there is a smaller diff than one per caller, and patching only the path the ticket names leaves a sibling caller still broken.
|
||||
|
||||
Rules:
|
||||
|
||||
- No abstractions that weren't explicitly requested.
|
||||
- No new dependency if it can be avoided.
|
||||
- No boilerplate nobody asked for.
|
||||
- Deletion over addition. Boring over clever. Fewest files possible.
|
||||
- Shortest working diff wins, but only once you understand the problem. The smallest change in the wrong place isn't lazy, it's a second bug.
|
||||
- Question complex requests: "Do you actually need X, or does Y cover it?"
|
||||
- Pick the edge-case-correct option when two stdlib approaches are the same size, lazy means less code, not the flimsier algorithm.
|
||||
- Mark intentional simplifications with a `ponytail:` comment. If the shortcut has a known ceiling (global lock, O(n²) scan, naive heuristic), the comment names the ceiling and the upgrade path.
|
||||
|
||||
Not lazy about: understanding the problem (read it fully and trace the real flow before picking a rung, a small diff you don't understand is just laziness dressed up as efficiency), input validation at trust boundaries, error handling that prevents data loss, security, accessibility, the calibration real hardware needs (the platform is never the spec ideal, a clock drifts, a sensor reads off), anything explicitly requested. Lazy code without its check is unfinished: non-trivial logic leaves ONE runnable check behind, the smallest thing that fails if the logic breaks (an assert-based demo/self-check or one small test file; no frameworks, no fixtures). Trivial one-liners need no test.
|
||||
15
LICENSE
15
LICENSE
|
|
@ -1,3 +1,18 @@
|
|||
Copyright (c) SurfSense
|
||||
|
||||
Portions of this software are licensed as follows:
|
||||
|
||||
* All content that resides under the "surfsense_backend/app/proprietary/"
|
||||
directory of this repository is licensed under the Business Source License
|
||||
1.1 as defined in "surfsense_backend/app/proprietary/LICENSE".
|
||||
* All third-party components incorporated into the SurfSense software are
|
||||
licensed under the original license provided by the owner of the applicable
|
||||
component.
|
||||
* Content outside of the above-mentioned directory is available under the
|
||||
Apache License, Version 2.0, as defined below.
|
||||
|
||||
-----------------------------------------------------------------------------
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
|
|
|||
381
README.es.md
381
README.es.md
|
|
@ -1,4 +1,4 @@
|
|||
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="readme_banner" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
|
||||
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="SurfSense, la plataforma de inteligencia competitiva de código abierto para agentes de IA" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
|
||||
|
||||
|
||||
|
||||
|
|
@ -20,289 +20,270 @@
|
|||
<a href="https://trendshift.io/repositories/13606" target="_blank"><img src="https://trendshift.io/api/badge/repositories/13606" alt="MODSetter%2FSurfSense | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
|
||||
</div>
|
||||
|
||||
# SurfSense
|
||||
# SurfSense: Dale inteligencia competitiva a tus agentes de IA
|
||||
|
||||
NotebookLM es una de las mejores y más útiles plataformas de IA que existen, pero una vez que comienzas a usarla regularmente también sientes sus limitaciones dejando algo que desear.
|
||||
SurfSense es la **plataforma de inteligencia competitiva de código abierto para agentes de IA**. Tus agentes monitorean a la competencia, siguen los rankings y escuchan a tu mercado con datos en vivo de **Reddit, YouTube, Google Maps, Google Search y la web abierta**, a través de una única **API REST** o un **servidor MCP**. Agentes programados y activados por eventos convierten lo que encuentran en informes y alertas, y una base de conocimiento integrada mantiene cada hallazgo disponible para búsqueda con citas.
|
||||
|
||||
1. Hay límites en la cantidad de fuentes que puedes agregar en un notebook.
|
||||
2. Hay límites en la cantidad de notebooks que puedes tener.
|
||||
3. No puedes tener fuentes que excedan 500,000 palabras y más de 200MB.
|
||||
4. Estás bloqueado con los servicios de Google (LLMs, modelos de uso, etc.) sin opción de configurarlos.
|
||||
5. Fuentes de datos externas e integraciones de servicios limitadas.
|
||||
6. El agente de NotebookLM está específicamente optimizado solo para estudiar e investigar, pero puedes hacer mucho más con los datos de origen.
|
||||
7. Falta de soporte multijugador.
|
||||
> [!NOTE]
|
||||
> **📢 Una nota para nuestros usuarios de la alternativa a NotebookLM**
|
||||
>
|
||||
> Durante los últimos meses construimos SurfSense como el mejor agente de investigación general para tu propio conocimiento, y ese capítulo nos dio una comunidad de la que estamos genuinamente orgullosos. Herramientas agénticas como Claude, OpenCode, Hermes y OpenClaw ya han demostrado que los agentes son el futuro, y la investigación general se está convirtiendo en algo que todo agente capaz hace de fábrica. Lo que a los agentes todavía les falta son **datos de mercado en vivo y los flujos de trabajo a su alrededor**, así que ahí es donde estamos dirigiendo toda nuestra energía: convertirnos en la plataforma definitiva de agentes de inteligencia competitiva de código abierto.
|
||||
>
|
||||
> **Nada de lo que dependes va a desaparecer.** Tu base de conocimiento, el chat con citas, los informes, los podcasts, las presentaciones, las automatizaciones y los chats colaborativos siguen funcionando, y el autoalojamiento sigue siendo gratuito y de código abierto. Lee el anuncio completo en [nuestro changelog](https://www.surfsense.com/changelog).
|
||||
|
||||
...y más.
|
||||
## Tabla de contenidos
|
||||
|
||||
**SurfSense está específicamente hecho para resolver estos problemas.** SurfSense te permite:
|
||||
- [Por qué los agentes necesitan SurfSense](#por-qué-los-agentes-necesitan-surfsense)
|
||||
- [¿Qué puedes hacer con SurfSense?](#qué-puedes-hacer-con-surfsense)
|
||||
- [Conectores de datos en vivo](#conectores-de-datos-en-vivo)
|
||||
- [Inicio rápido](#inicio-rápido)
|
||||
- [Todo lo demás que viene incluido](#todo-lo-demás-que-viene-incluido)
|
||||
- [SurfSense vs Google NotebookLM](#surfsense-vs-google-notebooklm)
|
||||
- [Hoja de ruta](#hoja-de-ruta)
|
||||
- [Contribuye](#contribuye)
|
||||
|
||||
- **Controla Tu Flujo de Datos** - Mantén tus datos privados y seguros.
|
||||
- **Sin Límites de Datos** - Agrega una cantidad ilimitada de fuentes y notebooks.
|
||||
- **Sin Dependencia de Proveedores** - Configura cualquier modelo LLM, de imagen, TTS y STT.
|
||||
- **25+ Fuentes de Datos Externas** - Agrega tus fuentes desde Google Drive, OneDrive, Dropbox, Notion y muchos otros servicios externos.
|
||||
- **Soporte Multijugador en Tiempo Real** - Trabaja fácilmente con los miembros de tu equipo en un notebook compartido.
|
||||
- **Automatizaciones y Agentes de IA** - Ejecuta agentes de IA según una programación o actívalos en el momento en que un documento llega a una carpeta, y luego escribe los resultados de vuelta en Notion, Slack, Linear y Drive. Crea automatizaciones sin código solo describiéndolas en el chat.
|
||||
- **Aplicación de Escritorio** - Obtén asistencia de IA en cualquier aplicación con Quick Assist, General Assist, Screenshot Assist y sincronización de carpetas locales.
|
||||
## Por qué los agentes necesitan SurfSense
|
||||
|
||||
...y más por venir.
|
||||
Pregúntale a cualquier agente capaz "¿cuánto están cobrando los competidores esta semana?" o "¿qué está diciendo Reddit sobre nosotros desde el lanzamiento?" y no tiene ningún lugar confiable donde buscar. Las APIs oficiales de las plataformas tienen límites de tasa, precios pensados para empresas grandes o directamente no existen, y la infraestructura de scraping es frágil. SurfSense cierra esa brecha:
|
||||
|
||||
- **Conectores nativos por plataforma**, cada uno un endpoint REST tipado que devuelve JSON estructurado. Sin ruleta de límites de tasa, sin parsear HTML.
|
||||
- **Un servidor MCP** que expone cada conector como una herramienta nativa (`surfsense_reddit_scrape`, `surfsense_google_search` y más) para Claude, Cursor o cualquier framework de agentes.
|
||||
- **Un arnés de agentes**, no solo datos en bruto: reintentos, salida estructurada y medición de créditos vienen integrados, así que los agentes pasan de una pregunta a un informe sin que tú construyas la infraestructura.
|
||||
- **Código abierto y autoalojable**, para que tu investigación competitiva se quede en tu propia infraestructura.
|
||||
|
||||
## ¿Qué puedes hacer con SurfSense?
|
||||
|
||||
## Ejemplo de Agente de Video
|
||||
Cada caso de uso a continuación es una tarea real que el agente de SurfSense ejecuta de principio a fin hoy, con una sola instrucción o según un horario.
|
||||
|
||||
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
|
||||
### Flujos de trabajo multi-conector
|
||||
|
||||
Encadena varios conectores en una sola ejecución del agente y recibe un único informe con citas.
|
||||
|
||||
- **Impacto de un lanzamiento, en todas las plataformas** — "Nuestro competidor lanzó la v2 ayer. Mide la reacción en búsquedas, Reddit y YouTube." El agente extrae SERPs, hilos de Reddit y comentarios de YouTube, y luego fusiona las tres señales en un solo informe de impacto del lanzamiento.
|
||||
- **Análisis de competidores locales** — Google Maps encuentra a los jugadores, el rastreador web lee sus páginas de precios y Google Search muestra quién gana la búsqueda, todo en una sola ejecución.
|
||||
- **Competidor 360, según un horario** — una automatización encadena cuatro conectores cada semana: cambios en el sitio, movimientos de ranking, sentimiento en Reddit y reacción en YouTube.
|
||||
|
||||
## Ejemplo de Agente de Podcast
|
||||
### Monitoreo de competidores
|
||||
|
||||
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
|
||||
- **Vigilancia de precios** — extrae cada plan, precio y límite de las páginas de precios de tus competidores en una sola tabla, luego vuelve a verificarlas a diario y alerta ante cualquier cambio.
|
||||
- **Seguimiento de producto y changelog** — rastrea las páginas de changelog, producto y empleo de tus rivales cada lunes y recibe un informe de lo que lanzaron.
|
||||
- **Monitoreo de rankings y anuncios** — sigue los rankings de Google, los anuncios pagados y las citas en AI Overviews que tu mercado realmente ve, y señala los movimientos día a día.
|
||||
|
||||
### Generación de leads B2B
|
||||
|
||||
## Cómo usar SurfSense
|
||||
- **Leads de negocios locales** — convierte una categoría y un territorio ("hamburgueserías en San José") en una lista de leads con teléfonos, sitios web, calificaciones y contactos de decisores extraídos de sus sitios.
|
||||
- **Equipos y contactos** — rastrea el sitio de cualquier empresa y extrae el equipo completo con correos, redes sociales y la procedencia de cada dato, exportado a CSV.
|
||||
- **Mapeo de portafolios y mercados** — mapea el portafolio de un inversor o una categoría completa, y luego enriquece cada empresa con precios y contactos.
|
||||
|
||||
### Cloud
|
||||
### Escucha de marca y mercado
|
||||
|
||||
1. Ve a [surfsense.com](https://www.surfsense.com) e inicia sesión.
|
||||
- **Monitoreo de marca en Reddit** — escucha lo que tu mercado dice sobre ti, tus competidores y tu categoría en los hilos donde los compradores hablan con franqueza.
|
||||
- **Sentimiento de audiencia en YouTube** — extrae videos, transcripciones y comentarios a escala, y luego agrupa lo que las audiencias elogian y critican.
|
||||
- **Minería de intención y cambios de proveedor** — encuentra a las personas que buscan activamente una alternativa a un competidor, ordenadas por qué tan listas están para cambiarse.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/LoginFlowGif.gif" alt="Login" /></p>
|
||||
### Investigación de mercado
|
||||
|
||||
2. Conecta tus conectores y sincroniza. Activa la sincronización periódica para mantenerlos actualizados.
|
||||
- **Investigación profunda en la web en vivo** — el agente rastrea docenas de fuentes en vivo sobre una pregunta y sintetiza una respuesta con citas, no un índice desactualizado.
|
||||
- **Seguimiento de AI Overviews y GEO** — captura cuándo los AI Overviews de Google responden las búsquedas de tu mercado, y exactamente qué fuentes citan.
|
||||
- **Informes y alertas con citas** — todo lo que los agentes recopilan aterriza en tu espacio de trabajo como informes y alertas con fuentes que puedes verificar.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="Conectores" /></p>
|
||||
### Automatiza cualquiera de estas tareas, sin código
|
||||
|
||||
3. Mientras se indexan los datos de los conectores, sube documentos.
|
||||
Las automatizaciones ejecutan turnos completos de agente según un horario o en respuesta a eventos, y luego escriben los resultados en Notion, Slack, Linear y Jira. Describe el flujo de trabajo en lenguaje natural y SurfSense lo construye. Prueba con instrucciones como:
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="Subir Documentos" /></p>
|
||||
- "Vigila las páginas de precios de nuestros 3 principales competidores y avísame en Slack cuando cambie un plan o un precio."
|
||||
- "Rastrea cada mención de nuestra marca en Reddit y YouTube y envíame un resumen diario."
|
||||
- "Monitorea nuestro ranking en Google para nuestras 10 palabras clave principales y señala las caídas semana a semana."
|
||||
- "Extrae las nuevas reseñas de Google Maps de nuestras ubicaciones y las de nuestros competidores cada lunes."
|
||||
- "Ejecuta un informe mensual de análisis de la competencia y guárdalo en mi espacio de trabajo."
|
||||
|
||||
4. Una vez que todo esté indexado, pregunta lo que quieras (Casos de uso):
|
||||
## Conectores de datos en vivo
|
||||
|
||||
**Aplicación de Escritorio** (extras nativos, además de todo lo de abajo, no un conjunto aparte)
|
||||
| Conector | Qué obtienen tus agentes | Más información |
|
||||
|---|---|---|
|
||||
| **Reddit** | Publicaciones, comentarios y flujos de subreddits sin los límites de tasa de la API oficial | [Reddit Scraper API](https://www.surfsense.com/reddit) |
|
||||
| **YouTube** | Videos, transcripciones e hilos de comentarios para escuchar a tu marca y productos | [YouTube Scraper API](https://www.surfsense.com/youtube) |
|
||||
| **Google Maps** | Lugares, calificaciones y reseñas para investigar competidores locales y prospectos | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
|
||||
| **Google Search** | SERPs en vivo para seguimiento de posiciones y monitoreo de mercado | [Google Search API](https://www.surfsense.com/google-search) |
|
||||
| **Web Crawl** (rastreo web) | Cualquier página de la web abierta como contenido limpio y estructurado | [Web Crawling API](https://www.surfsense.com/web-crawl) |
|
||||
| **Conectores MCP externos** | Conecta cualquier servidor MCP a tus agentes, con OAuth de un clic para Notion, Slack, Jira y más | [External MCP Connectors](https://www.surfsense.com/external-mcp-connectors) |
|
||||
|
||||
- General Assist: abre SurfSense al instante desde cualquier aplicación con un atajo global.
|
||||
La facturación es de pago por uso: los conectores facturan por elemento realmente devuelto, los rastreos por página obtenida con éxito, y las llamadas fallidas nunca se facturan. Las instalaciones autoalojadas funcionan con la facturación desactivada. Consulta los [precios](https://www.surfsense.com/pricing).
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/general_assist.gif" alt="General Assist" /></p>
|
||||
## Inicio rápido
|
||||
|
||||
- Quick Assist: selecciona texto en cualquier lugar y pide a la IA que lo explique, reescriba o actúe sobre él.
|
||||
### Llama a un conector desde código
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
|
||||
Cada conector es un endpoint REST que puedes llamar desde cualquier lenguaje con tu clave de API de SurfSense:
|
||||
|
||||
- Screenshot Assist: captura cualquier región de tu pantalla y pregunta a la IA sobre lo que contiene.
|
||||
```bash
|
||||
curl -X POST "$SURFSENSE_API_URL/workspaces/$WORKSPACE_ID/scrapers/reddit/scrape" \
|
||||
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"search_queries": ["your brand"],
|
||||
"community": "webscraping",
|
||||
"sort": "top",
|
||||
"time_filter": "week"
|
||||
}'
|
||||
```
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/screenshot_assist.gif" alt="Screenshot Assist" /></p>
|
||||
Cada [página de conector](https://www.surfsense.com/connectors) tiene ejemplos listos para copiar y pegar en Python, JavaScript, Go, PHP, Ruby, Java y C#.
|
||||
|
||||
- Watch Local Folder: sincroniza automáticamente una carpeta local con tu base de conocimiento. Ideal para bóvedas de Obsidian.
|
||||
### Dale las herramientas a tus agentes vía MCP
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/folder_watch.gif" alt="Watch Local Folder" /></p>
|
||||
Agrega el servidor MCP de SurfSense a Claude, Cursor o tu propio framework de agentes:
|
||||
|
||||
**Estudio de Entregables**
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"surfsense": {
|
||||
"url": "https://mcp.surfsense.com",
|
||||
"headers": { "Authorization": "Bearer ${SURFSENSE_API_KEY}" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- AI Report Generator: genera informes de investigación con citas y expórtalos a PDF, DOCX, HTML, LaTeX, EPUB, ODT o texto plano.
|
||||
Tu agente ahora puede llamar a cada conector como una herramienta nativa. Consulta la página del [servidor MCP de SurfSense](https://www.surfsense.com/mcp-server) para ver la lista completa de herramientas, o ejecuta el servidor localmente desde [`surfsense_mcp`](./surfsense_mcp).
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="AI Report Generator" /></p>
|
||||
### Usa la nube
|
||||
|
||||
- AI Podcast Generator: convierte cualquier documento o carpeta en un pódcast de IA con dos presentadores en menos de 20 segundos.
|
||||
Ve a [surfsense.com](https://www.surfsense.com), inicia sesión y pídele al agente datos de mercado en vivo en lenguaje natural. Las cuentas nuevas comienzan con $5 de crédito gratuito y sin suscripción.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/PodcastGenGif.gif" alt="AI Podcast Generator" /></p>
|
||||
|
||||
- AI Presentation & Video Maker: crea presentaciones editables y videos narrados a partir de tus fuentes.
|
||||
### Autoalójalo gratis
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/video_gen_gif.gif" alt="AI Presentation and Video Maker" /></p>
|
||||
|
||||
- AI Image Generator: genera imágenes de alta calidad directamente desde tus chats y documentos.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ImageGenGif.gif" alt="AI Image Generator" /></p>
|
||||
|
||||
- AI Resume Builder: adapta tu currículum existente a cualquier descripción de empleo y supera el ATS.
|
||||
Prueba indicaciones como estas:
|
||||
|
||||
- "Adapta mi currículum a esta descripción de empleo para superar el ATS y conseguir una entrevista."
|
||||
- "Optimiza mi currículum para ATS haciendo coincidir las palabras clave de esta oferta."
|
||||
- "Reescribe los puntos de mi currículum para resaltar las habilidades que pide este puesto."
|
||||
- "Compara mi currículum con esta descripción de empleo y enumera las carencias a corregir."
|
||||
- "Escribe una carta de presentación a juego con mi currículum y esta descripción de empleo."
|
||||
|
||||
**Búsqueda y Chat**
|
||||
|
||||
- Chat With Your PDFs & Docs: haz preguntas sobre todos tus archivos y obtén respuestas con citas en línea.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="Chat With Your PDFs and Docs" /></p>
|
||||
|
||||
- AI Search With Citations: búsqueda híbrida semántica y por palabras clave en toda tu base de conocimiento.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BSNCGif.gif" alt="AI Search With Citations" /></p>
|
||||
|
||||
- Collaborative AI Chat: trabaja en conversaciones de IA con tu equipo en tiempo real.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Collaborative AI Chat" /></p>
|
||||
|
||||
- Comments & Mentions: comenta y menciona a tus compañeros en cualquier mensaje de IA.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Comments and Mentions" /></p>
|
||||
|
||||
**Conectores e Integraciones**
|
||||
|
||||
- Connect & Sync Your Tools: sincroniza Notion, Slack, Google Drive, Gmail, GitHub, Linear y más de 25 fuentes en un único corpus consultable.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="Connect and Sync Your Tools" /></p>
|
||||
|
||||
- Chat With Uploaded Files: sube PDFs, documentos de Office, imágenes y audio. Consultables al instante.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="Chat With Uploaded Files" /></p>
|
||||
|
||||
- Connector Write-Back: deja que el agente publique los resultados de vuelta en Notion, Slack, Linear y Drive.
|
||||
Prueba indicaciones como estas:
|
||||
|
||||
- "Publica este resumen de investigación en mi espacio de Notion."
|
||||
- "Envía estos elementos de acción de la reunión a nuestro canal de Slack."
|
||||
- "Crea un ticket de Jira a partir de este informe de error."
|
||||
- "Abre una incidencia en Linear a partir de esta solicitud de función."
|
||||
- "Guarda este informe generado en Google Drive como un documento."
|
||||
|
||||
- Obsidian & Knowledge Base Sync: mantén tu bóveda de Obsidian y tu base de conocimiento personal sincronizadas.
|
||||
|
||||
**Automatizaciones**
|
||||
|
||||
- Scheduled AI Workflows: ejecuta un agente según una programación: resúmenes diarios, boletines semanales, informes recurrentes.
|
||||
Prueba indicaciones como estas:
|
||||
|
||||
- "Envíame cada mañana un resumen diario de los nuevos documentos en mi base de conocimiento."
|
||||
- "Genera un informe de estado semanal a partir de mi Slack y Gmail cada viernes."
|
||||
- "Ejecuta un informe mensual de análisis de la competencia y guárdalo en mi espacio de trabajo."
|
||||
- "Resume mi actividad de GitHub y Linear en una actualización diaria de standup."
|
||||
- "Crea un informe de investigación semanal recurrente sobre los temas que sigo."
|
||||
|
||||
- Event-Triggered Automations: lanza un agente en el momento en que un documento llega a una carpeta y publica el resultado en tus herramientas.
|
||||
Prueba indicaciones como estas:
|
||||
|
||||
- "Cuando llegue un PDF a mi carpeta de Investigación, genera un resumen de IA con citas."
|
||||
- "Cuando se añadan nuevas notas de reunión, conviértelas en actas con elementos de acción."
|
||||
- "Cuando se suba una factura, extrae el proveedor, el total y la fecha de vencimiento en una tabla."
|
||||
- "Cuando entre un contrato en mi carpeta Legal, señala los términos clave y las fechas de renovación."
|
||||
- "Cuando se añada un currículum a Candidatos, evalúalo frente a la descripción del empleo."
|
||||
|
||||
- Chat-Built Automations: describe una automatización en lenguaje sencillo y SurfSense la crea por ti.
|
||||
Prueba indicaciones como estas:
|
||||
|
||||
- "Crea un agente de IA que me envíe cada mañana un resumen de las nuevas páginas de Notion."
|
||||
- "Crea una automatización sin código que publique un resumen de investigación semanal en Slack."
|
||||
- "Configura un tomador de notas con IA que convierta las nuevas notas de reunión en actas."
|
||||
- "Crea un flujo que extraiga los elementos de acción de las notas de reunión y asigne responsables."
|
||||
- "Automatiza un resumen diario por correo a partir de mi Gmail y Google Drive."
|
||||
|
||||
|
||||
### Auto-Hospedado
|
||||
|
||||
Ejecuta SurfSense en tu propia infraestructura para control total de datos y privacidad.
|
||||
Ejecuta toda la plataforma, los conectores, los agentes, las automatizaciones y el servidor MCP en tu propia infraestructura. Las instalaciones autoalojadas vienen con la facturación desactivada, así que el scraping, el rastreo y las ejecuciones de agentes solo están limitados por tu hardware y las claves de modelos que aportes.
|
||||
|
||||
**Requisitos previos:** [Docker Desktop](https://www.docker.com/products/docker-desktop/) debe estar instalado y en ejecución.
|
||||
|
||||
#### Para usuarios de Linux/MacOS:
|
||||
Para Linux/macOS:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash
|
||||
```
|
||||
|
||||
#### Para usuarios de Windows:
|
||||
Para Windows:
|
||||
|
||||
```powershell
|
||||
```bash
|
||||
irm https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.ps1 | iex
|
||||
```
|
||||
|
||||
El script de instalación configura [Watchtower](https://github.com/nicholas-fedor/watchtower) automáticamente para actualizaciones diarias. Para omitirlo, agrega la bandera `--no-watchtower`.
|
||||
El script de instalación configura [Watchtower](https://github.com/nicholas-fedor/watchtower) automáticamente para actualizaciones automáticas diarias. Para omitirlo, agrega la opción `--no-watchtower`. Para Docker Compose, instalación manual y otras opciones de despliegue, consulta la [documentación](https://www.surfsense.com/docs/).
|
||||
|
||||
Para Docker Compose, instalación manual y otras opciones de despliegue, consulta la [documentación](https://www.surfsense.com/docs/).
|
||||
## Todo lo demás que viene incluido
|
||||
|
||||
### Aplicación de Escritorio
|
||||
El espacio de trabajo de investigación que convirtió a SurfSense en la alternativa de código abierto líder a NotebookLM sigue aquí, y todo lo que tus agentes recopilan aterriza en él.
|
||||
|
||||
SurfSense también ofrece una aplicación de escritorio que lleva la asistencia de IA a cada aplicación en tu computadora. Descárgala desde la [última versión](https://github.com/MODSetter/SurfSense/releases/latest).
|
||||
**Base de conocimiento**
|
||||
|
||||
La aplicación de escritorio incluye estas potentes funciones:
|
||||
- Sube PDFs, documentos de Office, imágenes y audio, o sincroniza **Google Drive, OneDrive y Dropbox**. Más de 50 formatos de archivo compatibles.
|
||||
- Búsqueda híbrida semántica y de texto completo con respuestas citadas al estilo Perplexity.
|
||||
- La organización de archivos con IA clasifica los documentos automáticamente por fuente, fecha y tema.
|
||||
|
||||
- **General Assist** — Lanza SurfSense al instante desde cualquier aplicación con un atajo global.
|
||||
- **Quick Assist** — Selecciona texto en cualquier lugar, luego pide a la IA que lo explique, reescriba o actúe sobre él.
|
||||
- **Screenshot Assist** — Selecciona una región de tu pantalla y adjúntala al chat para que las respuestas se basen en tu base de conocimiento.
|
||||
- **Watch Local Folder** — Vigila una carpeta local y sincroniza automáticamente los cambios de archivos con tu base de conocimiento. **Pro tip:** Apúntalo a tu bóveda de Obsidian para mantener tus notas buscables en SurfSense.
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="Chatea con tus PDFs y documentos" /></p>
|
||||
|
||||
Todas las funciones operan contra tu espacio de búsqueda elegido, por lo que tus respuestas siempre están basadas en tus propios datos.
|
||||
**Estudio de entregables**
|
||||
|
||||
### Cómo Colaborar en Tiempo Real (Beta)
|
||||
- Generador de informes con IA con exportación a PDF, DOCX, HTML, LaTeX, EPUB, ODT o texto plano.
|
||||
- Podcasts de IA con dos presentadores a partir de cualquier documento o carpeta en menos de 20 segundos.
|
||||
- Presentaciones de diapositivas editables, resúmenes en video narrados y generación de imágenes con IA.
|
||||
|
||||
1. Ve a la página de Gestión de Miembros y crea una invitación.
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="Generador de informes con IA" /></p>
|
||||
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/40ed7683-5aa6-48a0-a3df-00575528c392" alt="Invitar Miembros" /></p>
|
||||
**Colaboración en equipo**
|
||||
|
||||
2. El compañero de equipo se une y ese SearchSpace se comparte.
|
||||
- Chats de IA colaborativos en tiempo real con comentarios y menciones.
|
||||
- RBAC con roles de Propietario, Administrador, Editor y Visualizador.
|
||||
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/ea4e1057-4d2b-4fd2-9ca0-cd19286a285e" alt="Flujo de Unión por Invitación" /></p>
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Chat de IA colaborativo" /></p>
|
||||
|
||||
3. Haz el chat compartido.
|
||||
**Aplicación de escritorio**
|
||||
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/17b93904-0888-4c3a-ac12-51a24a8ea26a" alt="Hacer Chat Compartido" /></p>
|
||||
Asistencia de IA nativa en todas las aplicaciones de tu computadora. Descárgala desde la [última versión](https://github.com/MODSetter/SurfSense/releases/latest).
|
||||
|
||||
4. Tu equipo ahora puede chatear en tiempo real.
|
||||
- **General Assist**: abre SurfSense desde cualquier aplicación con un atajo global.
|
||||
- **Quick Assist**: selecciona texto en cualquier lugar y pídele a la IA que lo explique, lo reescriba o actúe sobre él.
|
||||
- **Screenshot Assist**: captura cualquier región de tu pantalla y pregúntale a la IA sobre ella.
|
||||
- **Watch Local Folder** (vigilar carpeta local): sincroniza automáticamente una carpeta local con tu base de conocimiento. Apúntala a tu bóveda de Obsidian para mantener tus notas disponibles para búsqueda.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Chat en Tiempo Real" /></p>
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
|
||||
|
||||
5. Agrega comentarios para etiquetar a compañeros de equipo.
|
||||
**Sin dependencia de un proveedor**
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Comentarios en Tiempo Real" /></p>
|
||||
- Más de 100 LLMs vía la especificación de OpenAI y LiteLLM, incluidos GPT-5.5, Claude Sonnet 5 y Gemini 3.1 Pro.
|
||||
- Más de 6,000 modelos de embeddings y todos los rerankers principales.
|
||||
- Soporte completo para LLMs locales y privados (vLLM, Ollama), para que tus datos sigan siendo tuyos.
|
||||
|
||||
## Muestra del agente de video
|
||||
|
||||
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
|
||||
|
||||
## Muestra del agente de podcast
|
||||
|
||||
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
|
||||
|
||||
## Cómo colaborar en tiempo real (Beta)
|
||||
|
||||
1. Ve a la página de administración de miembros y crea una invitación.
|
||||
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/40ed7683-5aa6-48a0-a3df-00575528c392" alt="Invitar miembros" /></p>
|
||||
|
||||
2. Un compañero de equipo se une y ese espacio de trabajo se vuelve compartido.
|
||||
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/ea4e1057-4d2b-4fd2-9ca0-cd19286a285e" alt="Flujo de invitación y unión" /></p>
|
||||
|
||||
3. Haz que un chat sea compartido y trabajen en él juntos en tiempo real, con comentarios para etiquetar a compañeros.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Comentarios en tiempo real" /></p>
|
||||
|
||||
## SurfSense vs Google NotebookLM
|
||||
|
||||
¿Todavía nos comparas como una alternativa a NotebookLM? Aquí tienes el desglose honesto.
|
||||
|
||||
| Característica | Google NotebookLM | SurfSense |
|
||||
|---------|-------------------|-----------|
|
||||
| **Fuentes por Notebook** | 50 (Gratis) a 600 (Ultra, $249.99/mes) | Ilimitadas |
|
||||
| **Número de Notebooks** | 100 (Gratis) a 500 (planes de pago) | Ilimitados |
|
||||
| **Límite de Tamaño de Fuente** | 500,000 palabras / 200MB por fuente | Sin límite |
|
||||
| **Precios** | Nivel gratuito disponible; Pro $19.99/mes, Ultra $249.99/mes | Gratuito y de código abierto, auto-hospedable en tu propia infra |
|
||||
| **Soporte de LLM** | Solo Google Gemini | 100+ LLMs vía OpenAI spec y LiteLLM |
|
||||
| **Modelos de Embeddings** | Solo Google | 6,000+ modelos de embeddings, todos los principales rerankers |
|
||||
| **LLMs Locales / Privados** | No disponible | Soporte completo (vLLM, Ollama) - tus datos son tuyos |
|
||||
| **Auto-Hospedable** | No | Sí - Docker en un solo comando o Docker Compose completo |
|
||||
| **Código Abierto** | No | Sí |
|
||||
| **Conectores Externos** | Google Drive, YouTube, sitios web | 27+ conectores - Motores de búsqueda, Google Drive, OneDrive, Dropbox, Slack, Teams, Jira, Notion, GitHub, Discord y [más](#fuentes-externas) |
|
||||
| **Soporte de Formatos de Archivo** | PDFs, Docs, Slides, Sheets, CSV, Word, EPUB, imágenes, URLs web, YouTube | 50+ formatos - documentos, imágenes, videos vía LlamaCloud, Unstructured o Docling (local) |
|
||||
| **Búsqueda** | Búsqueda semántica | Búsqueda Híbrida - Semántica + Texto completo con Índices Jerárquicos y Reciprocal Rank Fusion |
|
||||
| **Respuestas con Citas** | Sí | Sí - Respuestas citadas al estilo Perplexity |
|
||||
| **Arquitectura de Agentes** | No | Sí - impulsado por [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) con planificación, subagentes y acceso al sistema de archivos |
|
||||
| **Multijugador en Tiempo Real** | Notebooks compartidos con roles de Visor/Editor (sin chat en tiempo real) | RBAC con roles de Propietario / Admin / Editor / Visor, chat en tiempo real e hilos de comentarios |
|
||||
| **Generación de Videos** | Resúmenes en video cinemáticos vía Veo 3 (solo Ultra) | Disponible (NotebookLM es mejor aquí, mejorando activamente) |
|
||||
| **Generación de Presentaciones** | Diapositivas más atractivas pero no editables | Crea presentaciones editables basadas en diapositivas |
|
||||
| **Generación de Podcasts** | Resúmenes de audio con hosts e idiomas personalizables | Disponible con múltiples proveedores TTS (NotebookLM es mejor aquí, mejorando activamente) |
|
||||
| **Automatizaciones y Agentes de IA** | No | Flujos de trabajo de IA programados, disparadores por eventos en documentos nuevos y automatizaciones sin código creadas por chat con escritura de vuelta a Notion, Slack, Linear y Jira |
|
||||
| **Aplicación de Escritorio** | No | Aplicación nativa con General Assist, Quick Assist, Screenshot Assist y sincronización de carpetas locales |
|
||||
| **Extensión de Navegador** | No | Extensión multi-navegador para guardar cualquier página web, incluyendo páginas protegidas por autenticación |
|
||||
| **Datos de mercado en vivo para agentes** | No | Conectores de Reddit, YouTube, Google Maps, Google Search y rastreo web vía API REST y MCP |
|
||||
| **Servidor MCP** | No | Cada conector expuesto como herramienta nativa de agente, más servidores MCP propios con aplicaciones OAuth de un clic |
|
||||
| **Fuentes por Notebook** | 50 (gratis) a 600 (Ultra, $249.99/mes) | Ilimitadas |
|
||||
| **Número de Notebooks** | 100 (gratis) a 500 (niveles de pago) | Ilimitado |
|
||||
| **Límite de tamaño por fuente** | 500,000 palabras / 200MB por fuente | Sin límite |
|
||||
| **Precios** | Nivel gratuito; Pro $19.99/mes, Ultra $249.99/mes | Gratuito y de código abierto para autoalojar; la nube es de pago por uso con $5 de crédito gratuito |
|
||||
| **Soporte de LLM** | Solo Google Gemini | Más de 100 LLMs vía la especificación de OpenAI y LiteLLM |
|
||||
| **Modelos de embeddings** | Solo Google | Más de 6,000 modelos de embeddings, todos los rerankers principales |
|
||||
| **LLMs locales / privados** | No disponible | Soporte completo (vLLM, Ollama), tus datos siguen siendo tuyos |
|
||||
| **Autoalojable** | No | Sí, con un solo comando de Docker o Docker Compose completo |
|
||||
| **Código abierto** | No | Sí |
|
||||
| **Fuentes de la base de conocimiento** | Google Drive, YouTube, sitios web | Subida de archivos, Google Drive, OneDrive, Dropbox, sincronización de carpetas locales y páginas rastreadas |
|
||||
| **Formatos de archivo compatibles** | PDFs, Docs, Slides, Sheets, CSV, Word, EPUB, imágenes, URLs web, YouTube | Más de 50 formatos: documentos, imágenes, videos vía LlamaCloud, Unstructured o Docling (local) |
|
||||
| **Búsqueda** | Búsqueda semántica | Híbrida semántica + texto completo con índices jerárquicos y fusión de rangos recíprocos |
|
||||
| **Respuestas con citas** | Sí | Sí, respuestas citadas al estilo Perplexity |
|
||||
| **Arquitectura agéntica** | No | Sí, impulsada por [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) con planificación, subagentes y acceso al sistema de archivos |
|
||||
| **Automatizaciones y agentes de IA** | No | Flujos de trabajo programados, activadores por eventos y automatizaciones sin código creadas por chat con escritura en Notion, Slack, Linear y Jira |
|
||||
| **Multijugador en tiempo real** | Notebooks compartidos con roles de Visualizador/Editor (sin chat en tiempo real) | RBAC con roles de Propietario / Administrador / Editor / Visualizador, chat en tiempo real e hilos de comentarios |
|
||||
| **Generación de video** | Resúmenes de video cinematográficos vía Veo 3 (solo Ultra) | Disponible (NotebookLM es mejor aquí, en mejora activa) |
|
||||
| **Generación de presentaciones** | Diapositivas más atractivas pero no editables | Presentaciones editables basadas en diapositivas |
|
||||
| **Generación de podcasts** | Resúmenes de audio con presentadores e idiomas personalizables | Disponible con múltiples proveedores de TTS (NotebookLM es mejor aquí, en mejora activa) |
|
||||
| **Aplicación de escritorio** | No | Aplicación nativa con General Assist, Quick Assist, Screenshot Assist y sincronización de carpetas locales |
|
||||
|
||||
<details>
|
||||
<summary><b>Lista completa de Fuentes Externas</b></summary>
|
||||
<a id="fuentes-externas"></a>
|
||||
## Solicitudes de funciones y futuro
|
||||
|
||||
Motores de Búsqueda (SearXNG, Tavily, LinkUp, Baidu Search) · Google Drive · OneDrive · Dropbox · Slack · Microsoft Teams · Linear · Jira · ClickUp · Confluence · BookStack · Notion · Gmail · Videos de YouTube · GitHub · Discord · Airtable · Google Calendar · Luma · Circleback · Elasticsearch · Obsidian, y más por venir.
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
## SOLICITUDES DE FUNCIONES Y FUTURO
|
||||
|
||||
|
||||
**SurfSense está en desarrollo activo.** Aunque aún no está listo para producción, puedes ayudarnos a acelerar el proceso.
|
||||
**SurfSense está en desarrollo activo.** Aunque todavía no está listo para producción, puedes ayudarnos a acelerar el proceso.
|
||||
|
||||
¡Únete al [Discord de SurfSense](https://discord.gg/ejRNvftDp9) y ayuda a dar forma al futuro de SurfSense!
|
||||
|
||||
## Hoja de Ruta
|
||||
## Hoja de ruta
|
||||
|
||||
¡Mantente al día con nuestro progreso de desarrollo y próximas funcionalidades!
|
||||
Consulta nuestra hoja de ruta pública y contribuye con tus ideas o comentarios:
|
||||
Mantente al día con nuestro progreso de desarrollo y las próximas funciones. Consulta nuestra hoja de ruta pública y aporta tus ideas o comentarios:
|
||||
|
||||
**Discusión de la Hoja de Ruta:** [SurfSense 2026 Roadmap](https://github.com/MODSetter/SurfSense/discussions/565)
|
||||
**Discusión de la hoja de ruta:** [SurfSense 2026 Roadmap](https://github.com/MODSetter/SurfSense/discussions/565)
|
||||
|
||||
**Tablero Kanban:** [SurfSense Project Board](https://github.com/users/MODSetter/projects/3)
|
||||
|
||||
## Contribuye
|
||||
|
||||
## Contribuir
|
||||
|
||||
Todas las contribuciones son bienvenidas, desde estrellas y reportes de bugs hasta mejoras del backend. Consulta [CONTRIBUTING.md](CONTRIBUTING.md) para comenzar.
|
||||
Todas las contribuciones son bienvenidas, desde estrellas y reportes de errores hasta mejoras del backend. Consulta [CONTRIBUTING.md](CONTRIBUTING.md) para comenzar.
|
||||
|
||||
Gracias a todos nuestros Surfers:
|
||||
|
||||
|
|
@ -310,7 +291,7 @@ Gracias a todos nuestros Surfers:
|
|||
<img src="https://contrib.rocks/image?repo=MODSetter/SurfSense" />
|
||||
</a>
|
||||
|
||||
## Historial de Stars
|
||||
## Historial de estrellas
|
||||
|
||||
<a href="https://www.star-history.com/#MODSetter/SurfSense&Date">
|
||||
<picture>
|
||||
|
|
|
|||
385
README.hi.md
385
README.hi.md
|
|
@ -1,4 +1,4 @@
|
|||
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="readme_banner" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
|
||||
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="SurfSense, AI एजेंट्स के लिए ओपन सोर्स कॉम्पिटिटिव इंटेलिजेंस प्लेटफ़ॉर्म" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
|
||||
|
||||
|
||||
|
||||
|
|
@ -20,297 +20,278 @@
|
|||
<a href="https://trendshift.io/repositories/13606" target="_blank"><img src="https://trendshift.io/api/badge/repositories/13606" alt="MODSetter%2FSurfSense | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
|
||||
</div>
|
||||
|
||||
# SurfSense
|
||||
# SurfSense: अपने AI एजेंट्स को दें कॉम्पिटिटिव इंटेलिजेंस
|
||||
|
||||
NotebookLM वहाँ उपलब्ध सबसे अच्छे और सबसे उपयोगी AI प्लेटफ़ॉर्म में से एक है, लेकिन जब आप इसे नियमित रूप से उपयोग करना शुरू करते हैं तो आप इसकी सीमाओं को भी महसूस करते हैं जो कुछ और की चाह छोड़ती हैं।
|
||||
SurfSense **AI एजेंट्स के लिए ओपन सोर्स कॉम्पिटिटिव इंटेलिजेंस प्लेटफ़ॉर्म** है। आपके एजेंट प्रतिस्पर्धियों पर नज़र रखते हैं, रैंकिंग ट्रैक करते हैं, और **Reddit, YouTube, Google Maps, Google Search और ओपन वेब** से लाइव डेटा के साथ आपके बाज़ार की बात सुनते हैं, वह भी एक ही **REST API** या **MCP सर्वर** के ज़रिए। शेड्यूल्ड और इवेंट-ट्रिगर्ड एजेंट अपनी खोजों को ब्रीफ़ और अलर्ट में बदलते हैं, और एक बिल्ट-इन नॉलेज बेस हर खोज को साइटेशन के साथ खोजने योग्य बनाए रखता है।
|
||||
|
||||
1. एक notebook में जोड़े जा सकने वाले स्रोतों की मात्रा पर सीमाएं हैं।
|
||||
2. आपके पास कितने notebooks हो सकते हैं इस पर सीमाएं हैं।
|
||||
3. आपके पास ऐसे स्रोत नहीं हो सकते जो 500,000 शब्दों और 200MB से अधिक हों।
|
||||
4. आप Google सेवाओं (LLMs, उपयोग मॉडल, आदि) में बंद हैं और उन्हें कॉन्फ़िगर करने का कोई विकल्प नहीं है।
|
||||
5. सीमित बाहरी डेटा स्रोत और सेवा एकीकरण।
|
||||
6. NotebookLM एजेंट विशेष रूप से केवल अध्ययन और शोध के लिए अनुकूलित है, लेकिन आप स्रोत डेटा के साथ और भी बहुत कुछ कर सकते हैं।
|
||||
7. मल्टीप्लेयर सपोर्ट की कमी।
|
||||
> [!NOTE]
|
||||
> **📢 हमारे NotebookLM-विकल्प उपयोगकर्ताओं के लिए एक सूचना**
|
||||
>
|
||||
> पिछले कुछ महीनों में हमने SurfSense को आपके अपने ज्ञान के लिए सबसे बेहतरीन जनरल रिसर्च एजेंट के रूप में बनाया, और उस अध्याय ने हमें एक ऐसा समुदाय दिया जिस पर हमें सचमुच गर्व है। Claude, OpenCode, Hermes और OpenClaw जैसे एजेंटिक टूल्स ने अब साबित कर दिया है कि एजेंट ही भविष्य हैं, और जनरल रिसर्च अब हर सक्षम एजेंट की एक बुनियादी क्षमता बनती जा रही है। एजेंट्स के पास अब भी जिस चीज़ की कमी है वह है **लाइव मार्केट डेटा और उसके इर्द-गिर्द के वर्कफ़्लो**, इसलिए हम अपनी पूरी ऊर्जा वहीं लगा रहे हैं: निश्चित रूप से सबसे बेहतरीन ओपन सोर्स कॉम्पिटिटिव इंटेलिजेंस एजेंट प्लेटफ़ॉर्म बनना।
|
||||
>
|
||||
> **आप जिस भी चीज़ पर निर्भर हैं, वह कहीं नहीं जा रही।** आपका नॉलेज बेस, साइटेशन वाली चैट, रिपोर्ट, पॉडकास्ट, प्रेज़ेंटेशन, ऑटोमेशन और सहयोगी चैट, सब पहले की तरह काम करते रहेंगे, और सेल्फ-होस्टिंग मुफ़्त और ओपन सोर्स बनी रहेगी। पूरी घोषणा [हमारे changelog](https://www.surfsense.com/changelog) पर पढ़ें।
|
||||
|
||||
...और भी बहुत कुछ।
|
||||
## विषय-सूची
|
||||
|
||||
**SurfSense विशेष रूप से इन समस्याओं को हल करने के लिए बनाया गया है।** SurfSense आपको सक्षम बनाता है:
|
||||
- [एजेंट्स को SurfSense की ज़रूरत क्यों है](#एजेंट्स-को-surfsense-की-ज़रूरत-क्यों-है)
|
||||
- [SurfSense से आप क्या कर सकते हैं?](#surfsense-से-आप-क्या-कर-सकते-हैं)
|
||||
- [लाइव डेटा कनेक्टर](#लाइव-डेटा-कनेक्टर)
|
||||
- [क्विक स्टार्ट](#क्विक-स्टार्ट)
|
||||
- [बॉक्स में बाकी सब कुछ](#बॉक्स-में-बाकी-सब-कुछ)
|
||||
- [SurfSense बनाम Google NotebookLM](#surfsense-बनाम-google-notebooklm)
|
||||
- [रोडमैप](#रोडमैप)
|
||||
- [योगदान करें](#योगदान-करें)
|
||||
|
||||
- **अपने डेटा प्रवाह को नियंत्रित करें** - अपने डेटा को निजी और सुरक्षित रखें।
|
||||
- **कोई डेटा सीमा नहीं** - असीमित मात्रा में स्रोत और notebooks जोड़ें।
|
||||
- **कोई विक्रेता लॉक-इन नहीं** - किसी भी LLM, इमेज, TTS और STT मॉडल को कॉन्फ़िगर करें।
|
||||
- **25+ बाहरी डेटा स्रोत** - Google Drive, OneDrive, Dropbox, Notion और कई अन्य बाहरी सेवाओं से अपने स्रोत जोड़ें।
|
||||
- **रीयल-टाइम मल्टीप्लेयर सपोर्ट** - एक साझा notebook में अपनी टीम के सदस्यों के साथ आसानी से काम करें।
|
||||
- **AI ऑटोमेशन और एजेंट** - AI एजेंट को शेड्यूल पर चलाएं या जैसे ही कोई दस्तावेज़ किसी फ़ोल्डर में आए उसे ट्रिगर करें, फिर परिणाम वापस Notion, Slack, Linear और Drive में लिखें। चैट में बस वर्णन करके बिना-कोड ऑटोमेशन बनाएं।
|
||||
- **डेस्कटॉप ऐप** - Quick Assist, General Assist, Screenshot Assist और लोकल फ़ोल्डर सिंक के साथ किसी भी एप्लिकेशन में AI सहायता प्राप्त करें।
|
||||
## एजेंट्स को SurfSense की ज़रूरत क्यों है
|
||||
|
||||
...और भी बहुत कुछ आने वाला है।
|
||||
किसी भी सक्षम एजेंट से पूछिए "इस हफ़्ते प्रतिस्पर्धी क्या दाम ले रहे हैं?" या "लॉन्च के बाद से Reddit पर हमारे बारे में क्या कहा जा रहा है?" और उसके पास देखने के लिए कोई भरोसेमंद जगह नहीं होती। आधिकारिक प्लेटफ़ॉर्म API या तो रेट-लिमिटेड हैं, एंटरप्राइज़ के हिसाब से महंगे हैं, या हैं ही नहीं, और स्क्रैपिंग का ढांचा नाज़ुक होता है। SurfSense इसी कमी को पूरा करता है:
|
||||
|
||||
- **प्लेटफ़ॉर्म-नेटिव कनेक्टर**, हर एक टाइप्ड REST एंडपॉइंट है जो स्ट्रक्चर्ड JSON लौटाता है। न रेट-लिमिट का जुआ, न HTML पार्सिंग।
|
||||
- **एक MCP सर्वर** जो हर कनेक्टर को नेटिव टूल के रूप में (`surfsense_reddit_scrape`, `surfsense_google_search` और अन्य) Claude, Cursor या किसी भी एजेंट फ़्रेमवर्क को उपलब्ध कराता है।
|
||||
- **एक एजेंट हार्नेस**, सिर्फ़ कच्चा डेटा नहीं: रीट्राई, स्ट्रक्चर्ड आउटपुट और क्रेडिट मीटरिंग बिल्ट-इन हैं, ताकि एजेंट सवाल से सीधे ब्रीफ़ तक पहुंच सकें और आपको ढांचा खुद न बनाना पड़े।
|
||||
- **ओपन सोर्स और सेल्फ-होस्ट करने योग्य**, ताकि आपकी प्रतिस्पर्धी रिसर्च आपके अपने इन्फ्रास्ट्रक्चर पर ही रहे।
|
||||
|
||||
## SurfSense से आप क्या कर सकते हैं?
|
||||
|
||||
## वीडियो एजेंट नमूना
|
||||
नीचे दिया गया हर यूज़ केस एक वास्तविक कार्य है जिसे SurfSense एजेंट आज शुरू से अंत तक पूरा करता है, एक ही प्रॉम्प्ट में या शेड्यूल पर।
|
||||
|
||||
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
|
||||
### मल्टी-कनेक्टर वर्कफ़्लो
|
||||
|
||||
एक ही एजेंट रन में कई कनेक्टर जोड़ें और बदले में एक ही साइटेशन वाला ब्रीफ़ पाएं।
|
||||
|
||||
- **लॉन्च का असर, हर प्लेटफ़ॉर्म पर** — "हमारे प्रतिस्पर्धी ने कल v2 लॉन्च किया। सर्च, Reddit और YouTube पर प्रतिक्रिया मापो।" एजेंट SERPs, Reddit थ्रेड और YouTube कमेंट स्क्रैप करता है, फिर तीनों सिग्नल को एक ही लॉन्च-इम्पैक्ट ब्रीफ़ में मिला देता है।
|
||||
- **स्थानीय प्रतिस्पर्धियों का विश्लेषण** — Google Maps खिलाड़ियों को ढूंढता है, वेब क्रॉलर उनके प्राइसिंग पेज पढ़ता है, और Google Search दिखाता है कि सर्च में कौन जीत रहा है, सब एक ही रन में।
|
||||
- **कॉम्पिटिटर 360, शेड्यूल पर** — एक ऑटोमेशन हर हफ़्ते चार कनेक्टर जोड़ता है: साइट में बदलाव, रैंक में उतार-चढ़ाव, Reddit सेंटिमेंट और YouTube की प्रतिक्रिया।
|
||||
|
||||
## पॉडकास्ट एजेंट नमूना
|
||||
### प्रतिस्पर्धी मॉनिटरिंग
|
||||
|
||||
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
|
||||
- **प्राइसिंग वॉच** — प्रतिस्पर्धियों के प्राइसिंग पेजों से हर प्लान, कीमत और लिमिट एक ही टेबल में निकालो, फिर रोज़ाना दोबारा जांचो और किसी भी बदलाव पर अलर्ट पाओ।
|
||||
- **प्रोडक्ट और changelog ट्रैकिंग** — हर सोमवार प्रतिद्वंद्वियों के changelog, प्रोडक्ट और करियर पेज क्रॉल करो और जानो कि उन्होंने क्या लॉन्च किया।
|
||||
- **रैंक और विज्ञापन मॉनिटरिंग** — वे Google रैंकिंग, पेड विज्ञापन और AI Overview साइटेशन ट्रैक करो जो आपका बाज़ार वास्तव में देखता है, और दिन-प्रतिदिन के बदलावों को फ़्लैग करो।
|
||||
|
||||
### B2B लीड जनरेशन
|
||||
|
||||
## SurfSense का उपयोग कैसे करें
|
||||
- **स्थानीय बिज़नेस लीड** — किसी श्रेणी और क्षेत्र ("सैन होज़े में बर्गर की दुकानें") को फ़ोन, वेबसाइट, रेटिंग और उनकी साइटों से निकाले गए निर्णयकर्ताओं के संपर्कों वाली लीड सूची में बदलो।
|
||||
- **टीम रोस्टर और संपर्क** — किसी भी कंपनी की साइट को क्रॉल करो और ईमेल, सोशल मीडिया और हर डेटा के स्रोत के साथ पूरी टीम निकालो, CSV में एक्सपोर्ट के साथ।
|
||||
- **पोर्टफ़ोलियो और मार्केट मैपिंग** — किसी निवेशक का पोर्टफ़ोलियो या पूरी श्रेणी मैप करो, फिर हर कंपनी को प्राइसिंग और संपर्कों के साथ समृद्ध करो।
|
||||
|
||||
### Cloud
|
||||
### ब्रांड और मार्केट लिसनिंग
|
||||
|
||||
1. [surfsense.com](https://www.surfsense.com) पर जाएं और लॉगिन करें।
|
||||
- **Reddit ब्रांड मॉनिटरिंग** — उन थ्रेड्स में सुनो जहां ख़रीदार खुलकर बोलते हैं कि आपका बाज़ार आपके, आपके प्रतिस्पर्धियों और आपकी श्रेणी के बारे में क्या कह रहा है।
|
||||
- **YouTube ऑडियंस सेंटिमेंट** — वीडियो, ट्रांसक्रिप्ट और कमेंट बड़े पैमाने पर खींचो, फिर पता लगाओ कि दर्शक किस चीज़ की तारीफ़ करते हैं और किस पर शिकायत।
|
||||
- **विकल्प खोजने वालों और इरादे की माइनिंग** — उन लोगों को ढूंढो जो सक्रिय रूप से किसी प्रतिस्पर्धी का विकल्प खोज रहे हैं, इस आधार पर क्रमबद्ध कि वे बदलने के लिए कितने तैयार हैं।
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/LoginFlowGif.gif" alt="लॉगिन" /></p>
|
||||
### मार्केट रिसर्च
|
||||
|
||||
2. अपने कनेक्टर जोड़ें और सिंक करें। कनेक्टर्स को अपडेट रखने के लिए आवधिक सिंकिंग सक्षम करें।
|
||||
- **लाइव वेब पर गहन रिसर्च** — एजेंट किसी सवाल पर दर्जनों लाइव स्रोत क्रॉल करता है और साइटेशन के साथ जवाब तैयार करता है, कोई पुराना इंडेक्स नहीं।
|
||||
- **AI Overview और GEO ट्रैकिंग** — कैप्चर करो कि Google के AI Overviews आपके बाज़ार की क्वेरी का जवाब कब देते हैं, और वे ठीक-ठीक किन स्रोतों को साइट करते हैं।
|
||||
- **साइटेशन वाले ब्रीफ़ और अलर्ट** — एजेंट जो कुछ भी इकट्ठा करते हैं वह आपके वर्कस्पेस में ब्रीफ़ और अलर्ट के रूप में पहुंचता है, ऐसे स्रोतों के साथ जिन्हें आप जांच सकते हैं।
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="कनेक्टर्स" /></p>
|
||||
### इनमें से कुछ भी ऑटोमेट करें, बिना कोड के
|
||||
|
||||
3. जब तक कनेक्टर्स का डेटा इंडेक्स हो रहा है, दस्तावेज़ अपलोड करें।
|
||||
ऑटोमेशन शेड्यूल पर या इवेंट के जवाब में पूरे एजेंट टर्न चलाते हैं, फिर नतीजे Notion, Slack, Linear और Jira में वापस लिख देते हैं। वर्कफ़्लो को सीधी-सादी भाषा में बताइए और SurfSense उसे बना देगा। ये प्रॉम्प्ट आज़माएं:
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="दस्तावेज़ अपलोड करें" /></p>
|
||||
- "हमारे टॉप 3 प्रतिस्पर्धियों के प्राइसिंग पेज पर नज़र रखो और जब कोई प्लान या कीमत बदले तो मुझे Slack में अलर्ट करो।"
|
||||
- "Reddit और YouTube पर हमारे ब्रांड के हर उल्लेख को ट्रैक करो और मुझे रोज़ाना डाइजेस्ट भेजो।"
|
||||
- "हमारे टॉप 10 कीवर्ड के लिए हमारी Google रैंकिंग मॉनिटर करो और हफ़्ते-दर-हफ़्ते गिरावट को फ़्लैग करो।"
|
||||
- "हर सोमवार हमारी लोकेशनों और हमारे प्रतिस्पर्धियों की नई Google Maps रिव्यू खींचो।"
|
||||
- "हर महीने एक प्रतिस्पर्धी विश्लेषण रिपोर्ट चलाओ और उसे मेरे वर्कस्पेस में सेव करो।"
|
||||
|
||||
4. सब कुछ इंडेक्स हो जाने के बाद, कुछ भी पूछें (उपयोग के मामले):
|
||||
## लाइव डेटा कनेक्टर
|
||||
|
||||
**डेस्कटॉप ऐप** (नीचे दी गई सभी सुविधाओं के अलावा नेटिव एक्स्ट्रा, कोई अलग सेट नहीं)
|
||||
| कनेक्टर | आपके एजेंट्स को क्या मिलता है | और जानें |
|
||||
|---|---|---|
|
||||
| **Reddit** | आधिकारिक API की रेट लिमिट के बिना पोस्ट, कमेंट और सबरेडिट स्ट्रीम | [Reddit Scraper API](https://www.surfsense.com/reddit) |
|
||||
| **YouTube** | ब्रांड और प्रोडक्ट लिसनिंग के लिए वीडियो, ट्रांसक्रिप्ट और कमेंट थ्रेड | [YouTube Scraper API](https://www.surfsense.com/youtube) |
|
||||
| **Google Maps** | स्थानीय प्रतिस्पर्धी और लीड रिसर्च के लिए स्थान, रेटिंग और रिव्यू | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
|
||||
| **Google Search** | रैंक ट्रैकिंग और मार्केट मॉनिटरिंग के लिए लाइव SERP | [Google Search API](https://www.surfsense.com/google-search) |
|
||||
| **Web Crawl** | ओपन वेब का कोई भी पेज साफ़-सुथरे, स्ट्रक्चर्ड कंटेंट के रूप में | [Web Crawling API](https://www.surfsense.com/web-crawl) |
|
||||
| **External MCP Connectors** | कोई भी MCP सर्वर अपने एजेंट्स से जोड़ें, Notion, Slack, Jira और अन्य के लिए वन-क्लिक OAuth के साथ | [External MCP Connectors](https://www.surfsense.com/external-mcp-connectors) |
|
||||
|
||||
- General Assist: किसी भी ऐप्लिकेशन से ग्लोबल शॉर्टकट के ज़रिए SurfSense तुरंत खोलें।
|
||||
बिलिंग पे-एज़-यू-गो है: कनेक्टर सिर्फ़ वास्तव में लौटाए गए हर आइटम पर बिल करते हैं, क्रॉल सफलतापूर्वक फ़ेच किए गए हर पेज पर, और असफल कॉल कभी बिल नहीं होतीं। सेल्फ-होस्टेड इंस्टॉल बिलिंग बंद रखकर चलते हैं। देखें [pricing](https://www.surfsense.com/pricing)।
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/general_assist.gif" alt="General Assist" /></p>
|
||||
## क्विक स्टार्ट
|
||||
|
||||
- Quick Assist: कहीं भी टेक्स्ट चुनें और AI से उसे समझाने, दोबारा लिखने या उस पर कार्रवाई करने को कहें।
|
||||
### कोड से कनेक्टर कॉल करें
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
|
||||
हर कनेक्टर एक REST एंडपॉइंट है जिसे आप अपनी SurfSense API कुंजी के साथ किसी भी भाषा से कॉल कर सकते हैं:
|
||||
|
||||
- Screenshot Assist: अपनी स्क्रीन का कोई भी हिस्सा कैप्चर करें और AI से उसमें मौजूद चीज़ों के बारे में पूछें।
|
||||
```bash
|
||||
curl -X POST "$SURFSENSE_API_URL/workspaces/$WORKSPACE_ID/scrapers/reddit/scrape" \
|
||||
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"search_queries": ["your brand"],
|
||||
"community": "webscraping",
|
||||
"sort": "top",
|
||||
"time_filter": "week"
|
||||
}'
|
||||
```
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/screenshot_assist.gif" alt="Screenshot Assist" /></p>
|
||||
हर [कनेक्टर पेज](https://www.surfsense.com/connectors) पर Python, JavaScript, Go, PHP, Ruby, Java और C# में कॉपी-पेस्ट उदाहरण मौजूद हैं।
|
||||
|
||||
- Watch Local Folder: किसी लोकल फ़ोल्डर को अपने नॉलेज बेस के साथ अपने-आप सिंक करें। Obsidian vaults के लिए बढ़िया।
|
||||
### MCP के ज़रिए ये टूल्स अपने एजेंट्स को दें
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/folder_watch.gif" alt="Watch Local Folder" /></p>
|
||||
SurfSense MCP सर्वर को Claude, Cursor या अपने एजेंट फ़्रेमवर्क में जोड़ें:
|
||||
|
||||
**डिलीवरेबल स्टूडियो**
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"surfsense": {
|
||||
"url": "https://mcp.surfsense.com",
|
||||
"headers": { "Authorization": "Bearer ${SURFSENSE_API_KEY}" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- AI Report Generator: उद्धरण सहित रिसर्च रिपोर्ट बनाएं और PDF, DOCX, HTML, LaTeX, EPUB, ODT या सादे टेक्स्ट में एक्सपोर्ट करें।
|
||||
अब आपका एजेंट हर कनेक्टर को नेटिव टूल के रूप में कॉल कर सकता है। टूल्स की पूरी सूची के लिए [SurfSense MCP सर्वर](https://www.surfsense.com/mcp-server) पेज देखें, या [`surfsense_mcp`](./surfsense_mcp) से सर्वर को लोकली चलाएँ।
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="AI Report Generator" /></p>
|
||||
### क्लाउड इस्तेमाल करें
|
||||
|
||||
- AI Podcast Generator: किसी भी दस्तावेज़ या फ़ोल्डर को 20 सेकंड से भी कम में दो-होस्ट वाले AI पॉडकास्ट में बदलें।
|
||||
[surfsense.com](https://www.surfsense.com) पर जाएं, लॉग इन करें, और एजेंट से सीधी-सादी भाषा में लाइव मार्केट डेटा मांगें। नए अकाउंट $5 के मुफ़्त क्रेडिट के साथ शुरू होते हैं, बिना किसी सब्सक्रिप्शन के।
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/PodcastGenGif.gif" alt="AI Podcast Generator" /></p>
|
||||
|
||||
- AI Presentation & Video Maker: अपने स्रोतों से एडिट करने योग्य स्लाइड डेक और नैरेटेड वीडियो बनाएं।
|
||||
### मुफ़्त में सेल्फ-होस्ट करें
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/video_gen_gif.gif" alt="AI Presentation and Video Maker" /></p>
|
||||
पूरा प्लेटफ़ॉर्म, कनेक्टर, एजेंट, ऑटोमेशन और MCP सर्वर अपने इन्फ्रास्ट्रक्चर पर चलाएं। सेल्फ-होस्टेड इंस्टॉल बिलिंग बंद के साथ आते हैं, इसलिए स्क्रैपिंग, क्रॉलिंग और एजेंट रन की सीमा सिर्फ़ आपके हार्डवेयर और आपके द्वारा लाई गई मॉडल कुंजियों पर निर्भर करती है।
|
||||
|
||||
- AI Image Generator: अपनी चैट और दस्तावेज़ों से सीधे उच्च-गुणवत्ता वाली इमेज बनाएं।
|
||||
**आवश्यकताएं:** [Docker Desktop](https://www.docker.com/products/docker-desktop/) इंस्टॉल होना और चल रहा होना चाहिए।
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ImageGenGif.gif" alt="AI Image Generator" /></p>
|
||||
|
||||
- AI Resume Builder: अपने मौजूदा रिज़्यूमे को किसी भी जॉब डिस्क्रिप्शन के अनुसार ढालें और ATS को पार करें।
|
||||
इस तरह के प्रॉम्प्ट आज़माएं:
|
||||
|
||||
- "मेरे रिज़्यूमे को इस जॉब डिस्क्रिप्शन के अनुसार ढालें ताकि वह ATS पार करे और इंटरव्यू दिलाए।"
|
||||
- "इस जॉब पोस्टिंग के कीवर्ड्स से मिलान करके मेरे रिज़्यूमे को ATS के लिए ऑप्टिमाइज़ करें।"
|
||||
- "इस भूमिका के लिए ज़रूरी स्किल्स को उभारने के लिए मेरे रिज़्यूमे के बुलेट पॉइंट फिर से लिखें।"
|
||||
- "मेरे रिज़्यूमे की तुलना इस जॉब डिस्क्रिप्शन से करें और सुधारने योग्य कमियों की सूची दें।"
|
||||
- "मेरे रिज़्यूमे और इस जॉब डिस्क्रिप्शन से मेल खाता एक कवर लेटर लिखें।"
|
||||
|
||||
**सर्च और चैट**
|
||||
|
||||
- Chat With Your PDFs & Docs: अपनी सभी फ़ाइलों पर सवाल पूछें और इनलाइन उद्धरणों के साथ जवाब पाएं।
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="Chat With Your PDFs and Docs" /></p>
|
||||
|
||||
- AI Search With Citations: अपने पूरे नॉलेज बेस में हाइब्रिड सेमांटिक और कीवर्ड सर्च।
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BSNCGif.gif" alt="AI Search With Citations" /></p>
|
||||
|
||||
- Collaborative AI Chat: अपनी टीम के साथ रियल टाइम में AI बातचीत पर काम करें।
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Collaborative AI Chat" /></p>
|
||||
|
||||
- Comments & Mentions: किसी भी AI संदेश पर टिप्पणी करें और टीम के साथियों को टैग करें।
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Comments and Mentions" /></p>
|
||||
|
||||
**कनेक्टर्स और इंटीग्रेशन**
|
||||
|
||||
- Connect & Sync Your Tools: Notion, Slack, Google Drive, Gmail, GitHub, Linear और 25+ स्रोतों को एक खोजने योग्य कॉर्पस में सिंक करें।
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="Connect and Sync Your Tools" /></p>
|
||||
|
||||
- Chat With Uploaded Files: PDF, Office दस्तावेज़, इमेज और ऑडियो अपलोड करें। तुरंत खोजने योग्य।
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="Chat With Uploaded Files" /></p>
|
||||
|
||||
- Connector Write-Back: एजेंट को परिणाम वापस Notion, Slack, Linear और Drive में पोस्ट करने दें।
|
||||
इस तरह के प्रॉम्प्ट आज़माएं:
|
||||
|
||||
- "इस रिसर्च सारांश को मेरे Notion वर्कस्पेस में पोस्ट करें।"
|
||||
- "इन मीटिंग एक्शन आइटम्स को हमारे टीम Slack चैनल पर भेजें।"
|
||||
- "इस बग रिपोर्ट से एक Jira टिकट बनाएं।"
|
||||
- "इस फ़ीचर अनुरोध से Linear में एक इश्यू खोलें।"
|
||||
- "इस जनरेट की गई रिपोर्ट को Google Drive में एक डॉक के रूप में सेव करें।"
|
||||
|
||||
- Obsidian & Knowledge Base Sync: अपने Obsidian vault और व्यक्तिगत नॉलेज बेस को सिंक रखें।
|
||||
|
||||
**ऑटोमेशन**
|
||||
|
||||
- Scheduled AI Workflows: किसी एजेंट को शेड्यूल पर चलाएं: रोज़ाना ब्रीफ़, साप्ताहिक डाइजेस्ट, आवर्ती रिपोर्ट।
|
||||
इस तरह के प्रॉम्प्ट आज़माएं:
|
||||
|
||||
- "हर सुबह मेरे नॉलेज बेस में जुड़े नए दस्तावेज़ों का रोज़ाना ब्रीफ़ मुझे ईमेल करें।"
|
||||
- "हर शुक्रवार मेरे Slack और Gmail से एक साप्ताहिक स्टेटस रिपोर्ट बनाएं।"
|
||||
- "एक मासिक प्रतिस्पर्धी विश्लेषण रिपोर्ट चलाएं और उसे मेरे वर्कस्पेस में सेव करें।"
|
||||
- "मेरी GitHub और Linear गतिविधि को एक रोज़ाना standup अपडेट में सारांशित करें।"
|
||||
- "मैं जिन विषयों को ट्रैक करता हूं उन पर एक आवर्ती साप्ताहिक रिसर्च रिपोर्ट बनाएं।"
|
||||
|
||||
- Event-Triggered Automations: जैसे ही कोई दस्तावेज़ किसी फ़ोल्डर में आता है, एजेंट को चलाएं और परिणाम अपने टूल में पोस्ट करें।
|
||||
इस तरह के प्रॉम्प्ट आज़माएं:
|
||||
|
||||
- "जब मेरे Research फ़ोल्डर में कोई PDF आए, तो उद्धरण सहित एक AI सारांश बनाएं।"
|
||||
- "जब नई मीटिंग नोट्स जुड़ें, तो उन्हें एक्शन आइटम्स के साथ मीटिंग मिनट्स में बदलें।"
|
||||
- "जब कोई इनवॉइस अपलोड हो, तो विक्रेता, कुल राशि और देय तिथि को एक तालिका में निकालें।"
|
||||
- "जब मेरे Legal फ़ोल्डर में कोई अनुबंध आए, तो मुख्य शर्तों और नवीनीकरण तिथियों को चिह्नित करें।"
|
||||
- "जब Candidates में कोई रिज़्यूमे जुड़े, तो उसे जॉब डिस्क्रिप्शन के विरुद्ध स्क्रीन करें।"
|
||||
|
||||
- Chat-Built Automations: सरल भाषा में किसी ऑटोमेशन का वर्णन करें और SurfSense उसे आपके लिए बना देगा।
|
||||
इस तरह के प्रॉम्प्ट आज़माएं:
|
||||
|
||||
- "एक AI एजेंट बनाएं जो हर सुबह नई Notion पेजों का सारांश मुझे ईमेल करे।"
|
||||
- "एक नो-कोड ऑटोमेशन बनाएं जो हर सप्ताह एक रिसर्च डाइजेस्ट Slack पर पोस्ट करे।"
|
||||
- "एक AI नोट-टेकर सेट करें जो नई मीटिंग नोट्स को मिनट्स में बदल दे।"
|
||||
- "एक वर्कफ़्लो बनाएं जो मीटिंग नोट्स से एक्शन आइटम्स निकाले और ज़िम्मेदार सौंपे।"
|
||||
- "मेरे Gmail और Google Drive से एक रोज़ाना ईमेल ब्रीफ़ को ऑटोमेट करें।"
|
||||
|
||||
|
||||
### सेल्फ-होस्टेड
|
||||
|
||||
पूर्ण डेटा नियंत्रण और गोपनीयता के लिए SurfSense को अपने स्वयं के बुनियादी ढांचे पर चलाएं।
|
||||
|
||||
**आवश्यकताएँ:** [Docker Desktop](https://www.docker.com/products/docker-desktop/) इंस्टॉल और चालू होना चाहिए।
|
||||
|
||||
#### Linux/MacOS उपयोगकर्ताओं के लिए:
|
||||
Linux/macOS के लिए:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash
|
||||
```
|
||||
|
||||
#### Windows उपयोगकर्ताओं के लिए:
|
||||
Windows के लिए:
|
||||
|
||||
```powershell
|
||||
```bash
|
||||
irm https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.ps1 | iex
|
||||
```
|
||||
|
||||
इंस्टॉल स्क्रिप्ट दैनिक ऑटो-अपडेट के लिए स्वचालित रूप से [Watchtower](https://github.com/nicholas-fedor/watchtower) सेटअप करती है। इसे छोड़ने के लिए, `--no-watchtower` फ्लैग जोड़ें।
|
||||
इंस्टॉल स्क्रिप्ट दैनिक ऑटो-अपडेट के लिए [Watchtower](https://github.com/nicholas-fedor/watchtower) अपने आप सेट कर देती है। इसे छोड़ने के लिए `--no-watchtower` फ़्लैग जोड़ें। Docker Compose, मैनुअल इंस्टॉलेशन और अन्य डिप्लॉयमेंट विकल्पों के लिए [docs](https://www.surfsense.com/docs/) देखें।
|
||||
|
||||
Docker Compose, मैनुअल इंस्टॉलेशन और अन्य डिप्लॉयमेंट विकल्पों के लिए, [डॉक्स](https://www.surfsense.com/docs/) देखें।
|
||||
## बॉक्स में बाकी सब कुछ
|
||||
|
||||
### डेस्कटॉप ऐप
|
||||
जिस रिसर्च वर्कस्पेस ने SurfSense को अग्रणी ओपन सोर्स NotebookLM विकल्प बनाया, वह अब भी यहीं है, और आपके एजेंट जो कुछ भी इकट्ठा करते हैं वह सब इसी में पहुंचता है।
|
||||
|
||||
SurfSense एक डेस्कटॉप ऐप भी प्रदान करता है जो आपके कंप्यूटर पर हर एप्लिकेशन में AI सहायता लाता है। इसे [नवीनतम रिलीज़](https://github.com/MODSetter/SurfSense/releases/latest) से डाउनलोड करें।
|
||||
**नॉलेज बेस**
|
||||
|
||||
डेस्कटॉप ऐप में ये शक्तिशाली सुविधाएं शामिल हैं:
|
||||
- PDF, Office दस्तावेज़, इमेज और ऑडियो अपलोड करें, या **Google Drive, OneDrive और Dropbox** सिंक करें। 50+ फ़ाइल फ़ॉर्मैट समर्थित हैं।
|
||||
- हाइब्रिड सिमेंटिक और फ़ुल-टेक्स्ट सर्च, Perplexity-शैली के साइटेड जवाबों के साथ।
|
||||
- AI फ़ाइल सॉर्टिंग दस्तावेज़ों को स्रोत, तारीख़ और विषय के अनुसार अपने आप व्यवस्थित करती है।
|
||||
|
||||
- **General Assist** — एक ग्लोबल शॉर्टकट से किसी भी एप्लिकेशन से तुरंत SurfSense लॉन्च करें।
|
||||
- **Quick Assist** — कहीं भी टेक्स्ट चुनें, फिर AI से समझाने, फिर से लिखने या उस पर कार्रवाई करने को कहें।
|
||||
- **Screenshot Assist** — स्क्रीन पर एक क्षेत्र चुनें और उसे चैट में जोड़ें, ताकि उत्तर आपकी नॉलेज बेस पर आधारित रहें।
|
||||
- **Watch Local Folder** — एक लोकल फ़ोल्डर को वॉच करें और फ़ाइल परिवर्तनों को स्वचालित रूप से अपनी नॉलेज बेस में सिंक करें। **Pro tip:** इसे अपने Obsidian vault पर पॉइंट करें ताकि आपके नोट्स SurfSense में सर्च करने योग्य रहें।
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="अपनी PDF और दस्तावेज़ों से चैट करें" /></p>
|
||||
|
||||
सभी सुविधाएं आपके चुने हुए सर्च स्पेस पर काम करती हैं, ताकि आपके उत्तर हमेशा आपके अपने डेटा पर आधारित हों।
|
||||
**डिलीवरेबल स्टूडियो**
|
||||
|
||||
### रीयल-टाइम सहयोग कैसे करें (बीटा)
|
||||
- AI रिपोर्ट जनरेटर, PDF, DOCX, HTML, LaTeX, EPUB, ODT या सादे टेक्स्ट में एक्सपोर्ट के साथ।
|
||||
- किसी भी दस्तावेज़ या फ़ोल्डर से 20 सेकंड से कम में दो-होस्ट वाले AI पॉडकास्ट।
|
||||
- संपादन योग्य स्लाइड डेक, नैरेटेड वीडियो ओवरव्यू और AI इमेज जनरेशन।
|
||||
|
||||
1. सदस्य प्रबंधन पेज पर जाएं और एक आमंत्रण बनाएं।
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="AI रिपोर्ट जनरेटर" /></p>
|
||||
|
||||
**टीम सहयोग**
|
||||
|
||||
- कमेंट और मेंशन के साथ रीयल-टाइम सहयोगी AI चैट।
|
||||
- Owner, Admin, Editor और Viewer भूमिकाओं के साथ RBAC।
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="सहयोगी AI चैट" /></p>
|
||||
|
||||
**डेस्कटॉप ऐप**
|
||||
|
||||
आपके कंप्यूटर के हर एप्लिकेशन में नेटिव AI सहायता। [latest release](https://github.com/MODSetter/SurfSense/releases/latest) से डाउनलोड करें।
|
||||
|
||||
- **General Assist**: ग्लोबल शॉर्टकट से किसी भी ऐप से SurfSense लॉन्च करें।
|
||||
- **Quick Assist**: कहीं भी टेक्स्ट चुनें, फिर AI से उसे समझाने, फिर से लिखने या उस पर कार्रवाई करने को कहें।
|
||||
- **Screenshot Assist**: अपनी स्क्रीन का कोई भी हिस्सा कैप्चर करें और AI से उसके बारे में पूछें।
|
||||
- **Watch Local Folder**: किसी लोकल फ़ोल्डर को अपने नॉलेज बेस से ऑटो-सिंक करें। इसे अपने Obsidian वॉल्ट पर पॉइंट करें ताकि आपके नोट्स खोजने योग्य बने रहें।
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
|
||||
|
||||
**कोई वेंडर लॉक-इन नहीं**
|
||||
|
||||
- OpenAI स्पेक और LiteLLM के ज़रिए 100+ LLM, जिनमें GPT-5.5, Claude Sonnet 5 और Gemini 3.1 Pro शामिल हैं।
|
||||
- 6,000+ एम्बेडिंग मॉडल और सभी प्रमुख रीरैंकर।
|
||||
- पूर्ण लोकल और प्राइवेट LLM समर्थन (vLLM, Ollama), ताकि आपका डेटा आपका ही रहे।
|
||||
|
||||
## वीडियो एजेंट नमूना
|
||||
|
||||
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
|
||||
|
||||
## पॉडकास्ट एजेंट नमूना
|
||||
|
||||
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
|
||||
|
||||
## रीयल टाइम में सहयोग कैसे करें (बीटा)
|
||||
|
||||
1. Manage Members पेज पर जाएं और एक इनवाइट बनाएं।
|
||||
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/40ed7683-5aa6-48a0-a3df-00575528c392" alt="सदस्यों को आमंत्रित करें" /></p>
|
||||
|
||||
2. टीममेट जुड़ता है और वह SearchSpace साझा हो जाता है।
|
||||
2. कोई साथी जुड़ता है और वह वर्कस्पेस साझा हो जाता है।
|
||||
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/ea4e1057-4d2b-4fd2-9ca0-cd19286a285e" alt="आमंत्रण स्वीकार प्रवाह" /></p>
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/ea4e1057-4d2b-4fd2-9ca0-cd19286a285e" alt="इनवाइट जॉइन फ़्लो" /></p>
|
||||
|
||||
3. चैट को साझा करें।
|
||||
3. किसी चैट को साझा करें और उसमें रीयल टाइम में साथ काम करें, साथियों को टैग करने के लिए कमेंट के साथ।
|
||||
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/17b93904-0888-4c3a-ac12-51a24a8ea26a" alt="चैट साझा करें" /></p>
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="रीयलटाइम कमेंट" /></p>
|
||||
|
||||
4. आपकी टीम अब रीयल-टाइम में चैट कर सकती है।
|
||||
## SurfSense बनाम Google NotebookLM
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="रीयल-टाइम चैट" /></p>
|
||||
अब भी हमें NotebookLM विकल्प के तौर पर तौल रहे हैं? यह रहा ईमानदार तुलनात्मक ब्यौरा।
|
||||
|
||||
5. टीममेट्स को टैग करने के लिए कमेंट जोड़ें।
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="रीयल-टाइम कमेंट्स" /></p>
|
||||
|
||||
## SurfSense vs Google NotebookLM
|
||||
|
||||
| विशेषता | Google NotebookLM | SurfSense |
|
||||
| फ़ीचर | Google NotebookLM | SurfSense |
|
||||
|---------|-------------------|-----------|
|
||||
| **प्रति Notebook स्रोत** | 50 (मुफ़्त) से 600 (Ultra, $249.99/माह) | असीमित |
|
||||
| **Notebooks की संख्या** | 100 (मुफ़्त) से 500 (सशुल्क योजनाएं) | असीमित |
|
||||
| **एजेंट्स के लिए लाइव मार्केट डेटा** | नहीं | REST API और MCP के ज़रिए Reddit, YouTube, Google Maps, Google Search और वेब क्रॉल कनेक्टर |
|
||||
| **MCP सर्वर** | नहीं | हर कनेक्टर नेटिव एजेंट टूल के रूप में उपलब्ध, साथ ही वन-क्लिक OAuth ऐप्स के साथ अपने MCP सर्वर लाने की सुविधा |
|
||||
| **प्रति नोटबुक स्रोत** | 50 (Free) से 600 (Ultra, $249.99/माह) | असीमित |
|
||||
| **नोटबुक की संख्या** | 100 (Free) से 500 (सशुल्क टियर) | असीमित |
|
||||
| **स्रोत आकार सीमा** | 500,000 शब्द / 200MB प्रति स्रोत | कोई सीमा नहीं |
|
||||
| **मूल्य निर्धारण** | मुफ़्त स्तर उपलब्ध; Pro $19.99/माह, Ultra $249.99/माह | मुफ़्त और ओपन सोर्स, अपनी इंफ्रा पर सेल्फ-होस्ट करें |
|
||||
| **LLM सपोर्ट** | केवल Google Gemini | 100+ LLMs OpenAI spec और LiteLLM के माध्यम से |
|
||||
| **एम्बेडिंग मॉडल** | केवल Google | 6,000+ एम्बेडिंग मॉडल, सभी प्रमुख रीरैंकर्स |
|
||||
| **लोकल / प्राइवेट LLMs** | उपलब्ध नहीं | पूर्ण सपोर्ट (vLLM, Ollama) - आपका डेटा आपका रहता है |
|
||||
| **सेल्फ-होस्ट करने योग्य** | नहीं | हाँ - Docker एक कमांड या पूर्ण Docker Compose |
|
||||
| **ओपन सोर्स** | नहीं | हाँ |
|
||||
| **बाहरी कनेक्टर्स** | Google Drive, YouTube, वेबसाइटें | 27+ कनेक्टर्स - सर्च इंजन, Google Drive, OneDrive, Dropbox, Slack, Teams, Jira, Notion, GitHub, Discord और [अधिक](#बाहरी-स्रोत) |
|
||||
| **फ़ाइल फ़ॉर्मेट सपोर्ट** | PDFs, Docs, Slides, Sheets, CSV, Word, EPUB, इमेज, वेब URLs, YouTube | 50+ फ़ॉर्मेट - दस्तावेज़, इमेज, वीडियो LlamaCloud, Unstructured या Docling (लोकल) के माध्यम से |
|
||||
| **सर्च** | सिमैंटिक सर्च | हाइब्रिड सर्च - हायरार्किकल इंडाइसेस और Reciprocal Rank Fusion के साथ सिमैंटिक + फुल टेक्स्ट |
|
||||
| **उद्धृत उत्तर** | हाँ | हाँ - Perplexity शैली के उद्धृत उत्तर |
|
||||
| **एजेंट आर्किटेक्चर** | नहीं | हाँ - [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) द्वारा संचालित, योजना, सब-एजेंट और फ़ाइल सिस्टम एक्सेस |
|
||||
| **रीयल-टाइम मल्टीप्लेयर** | दर्शक/संपादक भूमिकाओं के साथ साझा notebooks (कोई रीयल-टाइम चैट नहीं) | मालिक / एडमिन / संपादक / दर्शक भूमिकाओं के साथ RBAC, रीयल-टाइम चैट और कमेंट थ्रेड |
|
||||
| **वीडियो जनरेशन** | Veo 3 के माध्यम से सिनेमैटिक वीडियो ओवरव्यू (केवल Ultra) | उपलब्ध (NotebookLM यहाँ बेहतर है, सक्रिय रूप से सुधार हो रहा है) |
|
||||
| **प्रेजेंटेशन जनरेशन** | बेहतर दिखने वाली स्लाइड्स लेकिन संपादन योग्य नहीं | संपादन योग्य, स्लाइड आधारित प्रेजेंटेशन बनाएं |
|
||||
| **पॉडकास्ट जनरेशन** | कस्टमाइज़ेबल होस्ट और भाषाओं के साथ ऑडियो ओवरव्यू | कई TTS प्रदाताओं के साथ उपलब्ध (NotebookLM यहाँ बेहतर है, सक्रिय रूप से सुधार हो रहा है) |
|
||||
| **AI ऑटोमेशन और एजेंट** | नहीं | शेड्यूल किए गए AI वर्कफ़्लो, नए दस्तावेज़ों पर इवेंट ट्रिगर, और चैट से बने बिना-कोड ऑटोमेशन, Notion, Slack, Linear और Jira में कनेक्टर राइट-बैक के साथ |
|
||||
| **मूल्य निर्धारण** | Free टियर; Pro $19.99/माह, Ultra $249.99/माह | सेल्फ-होस्ट के लिए मुफ़्त और ओपन सोर्स; क्लाउड पे-एज़-यू-गो है, $5 मुफ़्त क्रेडिट के साथ |
|
||||
| **LLM समर्थन** | केवल Google Gemini | OpenAI स्पेक और LiteLLM के ज़रिए 100+ LLM |
|
||||
| **एम्बेडिंग मॉडल** | केवल Google | 6,000+ एम्बेडिंग मॉडल, सभी प्रमुख रीरैंकर |
|
||||
| **लोकल / प्राइवेट LLM** | उपलब्ध नहीं | पूर्ण समर्थन (vLLM, Ollama), आपका डेटा आपका ही रहता है |
|
||||
| **सेल्फ-होस्ट करने योग्य** | नहीं | हां, Docker वन-लाइनर या पूर्ण Docker Compose |
|
||||
| **ओपन सोर्स** | नहीं | हां |
|
||||
| **नॉलेज बेस स्रोत** | Google Drive, YouTube, वेबसाइट | फ़ाइल अपलोड, Google Drive, OneDrive, Dropbox, लोकल फ़ोल्डर सिंक और क्रॉल किए गए पेज |
|
||||
| **फ़ाइल फ़ॉर्मैट समर्थन** | PDF, Docs, Slides, Sheets, CSV, Word, EPUB, इमेज, वेब URL, YouTube | 50+ फ़ॉर्मैट: LlamaCloud, Unstructured या Docling (लोकल) के ज़रिए दस्तावेज़, इमेज, वीडियो |
|
||||
| **सर्च** | सिमेंटिक सर्च | हाइरार्किकल इंडेक्स और रेसिप्रोकल रैंक फ़्यूज़न के साथ हाइब्रिड सिमेंटिक + फ़ुल-टेक्स्ट |
|
||||
| **साइटेड जवाब** | हां | हां, Perplexity-शैली के साइटेड जवाब |
|
||||
| **एजेंटिक आर्किटेक्चर** | नहीं | हां, [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) द्वारा संचालित, प्लानिंग, सबएजेंट और फ़ाइल सिस्टम एक्सेस के साथ |
|
||||
| **AI ऑटोमेशन और एजेंट** | नहीं | शेड्यूल्ड वर्कफ़्लो, इवेंट ट्रिगर और चैट से बने नो-कोड ऑटोमेशन, Notion, Slack, Linear और Jira में राइट-बैक के साथ |
|
||||
| **रीयल-टाइम मल्टीप्लेयर** | Viewer/Editor भूमिकाओं के साथ साझा नोटबुक (रीयल-टाइम चैट नहीं) | Owner / Admin / Editor / Viewer भूमिकाओं के साथ RBAC, रीयल-टाइम चैट और कमेंट थ्रेड |
|
||||
| **वीडियो जनरेशन** | Veo 3 के ज़रिए सिनेमैटिक वीडियो ओवरव्यू (केवल Ultra) | उपलब्ध (यहां NotebookLM बेहतर है, सक्रिय रूप से सुधार जारी) |
|
||||
| **प्रेज़ेंटेशन जनरेशन** | दिखने में बेहतर स्लाइड, लेकिन संपादन योग्य नहीं | संपादन योग्य, स्लाइड-आधारित प्रेज़ेंटेशन |
|
||||
| **पॉडकास्ट जनरेशन** | कस्टमाइज़ करने योग्य होस्ट और भाषाओं के साथ ऑडियो ओवरव्यू | कई TTS प्रोवाइडर के साथ उपलब्ध (यहां NotebookLM बेहतर है, सक्रिय रूप से सुधार जारी) |
|
||||
| **डेस्कटॉप ऐप** | नहीं | General Assist, Quick Assist, Screenshot Assist और लोकल फ़ोल्डर सिंक के साथ नेटिव ऐप |
|
||||
| **ब्राउज़र एक्सटेंशन** | नहीं | किसी भी वेबपेज को सहेजने के लिए क्रॉस-ब्राउज़र एक्सटेंशन, प्रमाणीकरण सुरक्षित पेज सहित |
|
||||
|
||||
<details>
|
||||
<summary><b>बाहरी स्रोतों की पूरी सूची</b></summary>
|
||||
<a id="बाहरी-स्रोत"></a>
|
||||
|
||||
सर्च इंजन (SearXNG, Tavily, LinkUp, Baidu Search) · Google Drive · OneDrive · Dropbox · Slack · Microsoft Teams · Linear · Jira · ClickUp · Confluence · BookStack · Notion · Gmail · YouTube वीडियो · GitHub · Discord · Airtable · Google Calendar · Luma · Circleback · Elasticsearch · Obsidian, और भी बहुत कुछ आने वाला है।
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
## फ़ीचर अनुरोध और भविष्य
|
||||
|
||||
**SurfSense सक्रिय रूप से विकसित किया जा रहा है।** हालांकि यह अभी प्रोडक्शन-रेडी नहीं है, आप इस प्रक्रिया को तेज़ करने में हमारी मदद कर सकते हैं।
|
||||
|
||||
**SurfSense सक्रिय रूप से विकसित किया जा रहा है।** हालांकि यह अभी प्रोडक्शन-रेडी नहीं है, आप प्रक्रिया को तेज़ करने में हमारी मदद कर सकते हैं।
|
||||
|
||||
[SurfSense Discord](https://discord.gg/ejRNvftDp9) में शामिल हों और SurfSense के भविष्य को आकार देने में मदद करें!
|
||||
[SurfSense Discord](https://discord.gg/ejRNvftDp9) से जुड़ें और SurfSense का भविष्य गढ़ने में मदद करें!
|
||||
|
||||
## रोडमैप
|
||||
|
||||
हमारे विकास की प्रगति और आने वाली सुविधाओं से अपडेट रहें!
|
||||
हमारा सार्वजनिक रोडमैप देखें और अपने विचार या फ़ीडबैक दें:
|
||||
हमारी विकास प्रगति और आने वाले फ़ीचर्स से अपडेट रहें। हमारा सार्वजनिक रोडमैप देखें और अपने विचार या फ़ीडबैक साझा करें:
|
||||
|
||||
**रोडमैप चर्चा:** [SurfSense 2026 Roadmap](https://github.com/MODSetter/SurfSense/discussions/565)
|
||||
|
||||
**कानबन बोर्ड:** [SurfSense Project Board](https://github.com/users/MODSetter/projects/3)
|
||||
|
||||
**कानबान बोर्ड:** [SurfSense Project Board](https://github.com/users/MODSetter/projects/3)
|
||||
|
||||
## योगदान करें
|
||||
|
||||
सभी योगदान स्वागत योग्य हैं, स्टार और बग रिपोर्ट से लेकर बैकएंड सुधार तक। शुरू करने के लिए [CONTRIBUTING.md](CONTRIBUTING.md) देखें।
|
||||
हर तरह का योगदान स्वागत योग्य है, स्टार और बग रिपोर्ट से लेकर बैकएंड सुधार तक। शुरू करने के लिए [CONTRIBUTING.md](CONTRIBUTING.md) देखें।
|
||||
|
||||
हमारे सभी Surfers को धन्यवाद:
|
||||
हमारे सभी Surfers का धन्यवाद:
|
||||
|
||||
<a href="https://github.com/MODSetter/SurfSense/graphs/contributors">
|
||||
<img src="https://contrib.rocks/image?repo=MODSetter/SurfSense" />
|
||||
</a>
|
||||
|
||||
## Star इतिहास
|
||||
## स्टार हिस्ट्री
|
||||
|
||||
<a href="https://www.star-history.com/#MODSetter/SurfSense&Date">
|
||||
<picture>
|
||||
|
|
|
|||
352
README.md
352
README.md
|
|
@ -1,4 +1,4 @@
|
|||
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="readme_banner" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
|
||||
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="SurfSense, the open-source competitive intelligence platform for AI agents" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
|
||||
|
||||
|
||||
|
||||
|
|
@ -20,273 +20,253 @@
|
|||
<a href="https://trendshift.io/repositories/13606" target="_blank"><img src="https://trendshift.io/api/badge/repositories/13606" alt="MODSetter%2FSurfSense | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
|
||||
</div>
|
||||
|
||||
# SurfSense
|
||||
# SurfSense: Give Your AI Agents Competitive Intelligence
|
||||
|
||||
NotebookLM is one of the best and most useful AI platforms out there, but once you start using it regularly you also feel its limitations leaving something to be desired more.
|
||||
SurfSense is the **open-source competitive intelligence platform for AI agents**. Your agents monitor competitors, track rankings, and listen to your market with live data from **Reddit, YouTube, Google Maps, Google Search, and the open web**, through one **REST API** or **MCP server**. Scheduled and event-triggered agents turn what they find into briefs and alerts, and a built-in knowledge base keeps every finding searchable with citations.
|
||||
|
||||
1. There are limits on the amount of sources you can add in a notebook.
|
||||
2. There are limits on the number of notebooks you can have.
|
||||
3. You cannot have sources that exceed 500,000 words and are more than 200MB.
|
||||
4. You are vendor locked in to Google services (LLMs, usage models, etc.) with no option to configure them.
|
||||
5. Limited external data sources and service integrations.
|
||||
6. NotebookLM Agent is specifically optimised for just studying and researching, but you can do so much more with the source data.
|
||||
7. Lack of multiplayer support.
|
||||
> [!NOTE]
|
||||
> **📢 A note for our NotebookLM-alternative users**
|
||||
>
|
||||
> For the past couple of months we built SurfSense as the best general research agent for your own knowledge, and that chapter earned us a community we are genuinely proud of. Agentic tools like Claude, OpenCode, Hermes, and OpenClaw have now proven that agents are the future, and general research is becoming something every capable agent does out of the box. What agents still lack is **live market data and the workflows around it**, so that is where we are pointing all of our energy: becoming the definitive open-source competitive intelligence agent platform.
|
||||
>
|
||||
> **Nothing you rely on is going away.** Your knowledge base, chat with citations, reports, podcasts, presentations, automations, and collaborative chats all keep working, and self-hosting stays free and open source. Read the full announcement on [our changelog](https://www.surfsense.com/changelog).
|
||||
|
||||
...and more.
|
||||
## Table of contents
|
||||
|
||||
**SurfSense is specifically made to solve these problems.** SurfSense empowers you to:
|
||||
- [Why agents need SurfSense](#why-agents-need-surfsense)
|
||||
- [What can you do with SurfSense?](#what-can-you-do-with-surfsense)
|
||||
- [Live data connectors](#live-data-connectors)
|
||||
- [Quick start](#quick-start)
|
||||
- [Everything else in the box](#everything-else-in-the-box)
|
||||
- [SurfSense vs Google NotebookLM](#surfsense-vs-google-notebooklm)
|
||||
- [Roadmap](#roadmap)
|
||||
- [Contribute](#contribute)
|
||||
|
||||
- **Control Your Data Flow** - Keep your data private and secure.
|
||||
- **No Data Limits** - Add an unlimited amount of sources and notebooks.
|
||||
- **No Vendor Lock-in** - Configure any LLM, image, TTS, and STT models to use.
|
||||
- **25+ External Data Sources** - Add your sources from Google Drive, OneDrive, Dropbox, Notion, and many other external services.
|
||||
- **Real-Time Multiplayer Support** - Work easily with your team members in a shared notebook.
|
||||
- **AI File Sorting** - Automatically organize your documents into a smart folder hierarchy using AI-powered categorization by source, date, and topic.
|
||||
- **AI Automations & Agents** - Run AI agents on a schedule or trigger them the moment a document lands in a folder, then write results back to Notion, Slack, Linear, and Drive. Build no-code automations just by describing them in chat.
|
||||
- **Desktop App** - Get AI assistance in any application with Quick Assist, General Assist, Screenshot Assist, and local folder sync.
|
||||
## Why agents need SurfSense
|
||||
|
||||
...and more to come.
|
||||
Ask any capable agent "what are competitors charging this week?" or "what is Reddit saying about us since the launch?" and it has nowhere trustworthy to look. Official platform APIs are rate-limited, priced for enterprises, or missing entirely, and scraping plumbing is brittle. SurfSense closes that gap:
|
||||
|
||||
- **Platform-native connectors**, each a typed REST endpoint returning structured JSON. No rate-limit roulette, no HTML parsing.
|
||||
- **An MCP server** that exposes every connector as a native tool (`surfsense_reddit_scrape`, `surfsense_google_search`, and more) to Claude, Cursor, or any agent framework.
|
||||
- **An agent harness**, not just raw data: retries, structured output, and credit metering are built in, so agents go from a question to a brief without you building the plumbing.
|
||||
- **Open source and self-hostable**, so your competitive research stays on your own infrastructure.
|
||||
|
||||
## What can you do with SurfSense?
|
||||
|
||||
## Video Agent Sample
|
||||
Every use case below is a real task the SurfSense agent runs end-to-end today, in one prompt or on a schedule.
|
||||
|
||||
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
|
||||
### Multi-connector workflows
|
||||
|
||||
Chain several connectors in a single agent run and get one cited brief back.
|
||||
|
||||
- **Launch impact, across every platform** — "Our competitor launched v2 yesterday. Measure the reaction across search, Reddit, and YouTube." The agent scrapes SERPs, Reddit threads, and YouTube comments, then merges all three signals into one launch-impact brief.
|
||||
- **Local competitor teardown** — Google Maps finds the players, the web crawler reads their pricing pages, and Google Search shows who wins the query, in one run.
|
||||
- **Competitor 360, on a schedule** — an automation chains four connectors every week: site changes, rank movements, Reddit sentiment, and YouTube reaction.
|
||||
|
||||
## Podcast Agent Sample
|
||||
### Competitor monitoring
|
||||
|
||||
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
|
||||
- **Pricing watch** — extract every plan, price, and limit from competitors' pricing pages into one table, then re-check daily and alert on any change.
|
||||
- **Product & changelog tracking** — crawl rivals' changelog, product, and careers pages every Monday and get a brief on what they shipped.
|
||||
- **Rank & ad monitoring** — track the Google rankings, paid ads, and AI Overview citations your market actually sees, and flag movements day over day.
|
||||
|
||||
### B2B lead generation
|
||||
|
||||
## How to Use SurfSense
|
||||
- **Local business leads** — turn a category and a territory ("burger places in San Jose") into a lead list with phones, websites, ratings, and decision-maker contacts pulled from their sites.
|
||||
- **Team rosters & contacts** — spider any company site and pull the full team with emails, socials, and source provenance, exported to CSV.
|
||||
- **Portfolio & market mapping** — map an investor's portfolio or a whole category, then enrich every company with pricing and contacts.
|
||||
|
||||
### Cloud
|
||||
### Brand & market listening
|
||||
|
||||
1. Go to [surfsense.com](https://www.surfsense.com) and login.
|
||||
- **Reddit brand monitoring** — hear what your market says about you, your competitors, and your category in the threads where buyers speak candidly.
|
||||
- **YouTube audience sentiment** — pull videos, transcripts, and comments at scale, then cluster what audiences praise and complain about.
|
||||
- **Switcher & intent mining** — find the people actively asking for an alternative to a competitor, ranked by how ready they are to move.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/LoginFlowGif.gif" alt="Login" /></p>
|
||||
### Market research
|
||||
|
||||
2. Connect your connectors and sync. Enable periodic syncing to keep connectors synced.
|
||||
- **Deep research on the live web** — the agent crawls dozens of live sources on a question and synthesizes a cited answer, not a stale index.
|
||||
- **AI Overview & GEO tracking** — capture when Google's AI Overviews answer your market's queries, and exactly which sources they cite.
|
||||
- **Cited briefs & alerts** — everything the agents gather lands in your workspace as briefs and alerts with sources you can check.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="Connectors" /></p>
|
||||
### Automate any of it, no code
|
||||
|
||||
3. Till connectors data index, upload Documents.
|
||||
Automations run full agent turns on a schedule or in response to events, then write results back to Notion, Slack, Linear, and Jira. Describe the workflow in plain English and SurfSense builds it. Try prompts like:
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="Upload Documents" /></p>
|
||||
- "Watch our top 3 competitors' pricing pages and alert me in Slack when a plan or price changes."
|
||||
- "Track every mention of our brand on Reddit and YouTube and send me a daily digest."
|
||||
- "Monitor our Google ranking for our top 10 keywords and flag drops week over week."
|
||||
- "Pull new Google Maps reviews for our locations and our competitors' every Monday."
|
||||
- "Run a monthly competitor analysis report and save it to my workspace."
|
||||
|
||||
4. Once everything is indexed, Ask Away (Use Cases):
|
||||
## Live data connectors
|
||||
|
||||
**Desktop App** (native extras on top of everything below, not a separate feature set)
|
||||
| Connector | What your agents get | Learn more |
|
||||
|---|---|---|
|
||||
| **Reddit** | Posts, comments, and subreddit streams without the official API's rate limits | [Reddit Scraper API](https://www.surfsense.com/reddit) |
|
||||
| **YouTube** | Videos, transcripts, and comment threads for brand and product listening | [YouTube Scraper API](https://www.surfsense.com/youtube) |
|
||||
| **Google Maps** | Places, ratings, and reviews for local competitor and lead research | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
|
||||
| **Google Search** | Live SERPs for rank tracking and market monitoring | [Google Search API](https://www.surfsense.com/google-search) |
|
||||
| **Web Crawl** | Any page on the open web as clean, structured content | [Web Crawling API](https://www.surfsense.com/web-crawl) |
|
||||
| **External MCP Connectors** | Bring any MCP server to your agents, with one-click OAuth for Notion, Slack, Jira, and more | [External MCP Connectors](https://www.surfsense.com/external-mcp-connectors) |
|
||||
|
||||
- General Assist: launch SurfSense instantly from any application with a global shortcut.
|
||||
Billing is pay as you go: connectors bill per item actually returned, crawls per page successfully fetched, and failed calls are never billed. Self-hosted installs run with billing off. See [pricing](https://www.surfsense.com/pricing).
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/general_assist.gif" alt="General Assist" /></p>
|
||||
## Quick start
|
||||
|
||||
- Quick Assist: select text anywhere, then ask AI to explain, rewrite, or act on it.
|
||||
### Call a connector from code
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
|
||||
Every connector is a REST endpoint you can call from any language with your SurfSense API key:
|
||||
|
||||
- Screenshot Assist: capture any region of your screen and ask AI about what's in it.
|
||||
```bash
|
||||
curl -X POST "$SURFSENSE_API_URL/workspaces/$WORKSPACE_ID/scrapers/reddit/scrape" \
|
||||
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"search_queries": ["your brand"],
|
||||
"community": "webscraping",
|
||||
"sort": "top",
|
||||
"time_filter": "week"
|
||||
}'
|
||||
```
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/screenshot_assist.gif" alt="Screenshot Assist" /></p>
|
||||
Each [connector page](https://www.surfsense.com/connectors) has copy-paste examples in Python, JavaScript, Go, PHP, Ruby, Java, and C#.
|
||||
|
||||
- Watch Local Folder: auto-sync a local folder to your knowledge base. Great for Obsidian vaults.
|
||||
### Give the tools to your agents over MCP
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/folder_watch.gif" alt="Watch Local Folder" /></p>
|
||||
Add the SurfSense MCP server to Claude, Cursor, or your own agent framework:
|
||||
|
||||
**Deliverable Studio**
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"surfsense": {
|
||||
"url": "https://mcp.surfsense.com",
|
||||
"headers": { "Authorization": "Bearer ${SURFSENSE_API_KEY}" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- AI Report Generator: generate cited research reports and export to PDF, DOCX, HTML, LaTeX, EPUB, ODT, or plain text.
|
||||
Your agent can now call every connector as a native tool. See the [SurfSense MCP server](https://www.surfsense.com/mcp-server) page for the full tool list, or run the server locally from [`surfsense_mcp`](./surfsense_mcp).
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="AI Report Generator" /></p>
|
||||
### Use the cloud
|
||||
|
||||
- AI Podcast Generator: turn any document or folder into a two-host AI podcast in under 20 seconds.
|
||||
Go to [surfsense.com](https://www.surfsense.com), log in, and ask the agent for live market data in plain English. New accounts start with $5 of free credit and no subscription.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/PodcastGenGif.gif" alt="AI Podcast Generator" /></p>
|
||||
### Self-host for free
|
||||
|
||||
- AI Presentation & Video Maker: create editable slide decks and narrated video overviews from your sources.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/video_gen_gif.gif" alt="AI Presentation and Video Maker" /></p>
|
||||
|
||||
- AI Image Generator: generate high-quality images straight from your chats and documents.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ImageGenGif.gif" alt="AI Image Generator" /></p>
|
||||
|
||||
- AI Resume Builder: tailor your existing resume to any job description and beat the ATS.
|
||||
Try prompts like these:
|
||||
|
||||
- "Tailor my resume to this job description so it gets past ATS and lands an interview."
|
||||
- "Optimize my resume for ATS by matching the keywords in this job posting."
|
||||
- "Rewrite my resume bullet points to highlight the skills this role is asking for."
|
||||
- "Compare my resume against this job description and list the gaps to fix."
|
||||
- "Write a matching cover letter from my resume and this job description."
|
||||
|
||||
**Search & Chat**
|
||||
|
||||
- Chat With Your PDFs & Docs: ask questions across all your files and get answers with inline citations.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="Chat With Your PDFs and Docs" /></p>
|
||||
|
||||
- AI Search With Citations: hybrid semantic and keyword search across your entire knowledge base.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BSNCGif.gif" alt="AI Search With Citations" /></p>
|
||||
|
||||
- Collaborative AI Chat: work on AI conversations with your team in real time.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Collaborative AI Chat" /></p>
|
||||
|
||||
- Comments & Mentions: comment and tag teammates on any AI message.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Comments and Mentions" /></p>
|
||||
|
||||
**Connectors & Integrations**
|
||||
|
||||
- Connect & Sync Your Tools: sync Notion, Slack, Google Drive, Gmail, GitHub, Linear and 25+ sources into one searchable corpus.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="Connect and Sync Your Tools" /></p>
|
||||
|
||||
- Chat With Uploaded Files: drop in PDFs, Office docs, images and audio. Instantly searchable.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="Chat With Uploaded Files" /></p>
|
||||
|
||||
- Connector Write-Back: let the agent post results back to Notion, Slack, Linear and Drive.
|
||||
Try prompts like these:
|
||||
|
||||
- "Post this research summary to my Notion workspace."
|
||||
- "Send these meeting action items to our team Slack channel."
|
||||
- "Create a Jira ticket from this bug report."
|
||||
- "Open a Linear issue from this feature request."
|
||||
- "Save this generated report to Google Drive as a doc."
|
||||
|
||||
- Obsidian & Knowledge Base Sync: keep your Obsidian vault and personal knowledge base in sync.
|
||||
|
||||
**Automations**
|
||||
|
||||
- Scheduled AI Workflows: run an agent on a schedule: daily briefs, weekly digests, recurring reports.
|
||||
Try prompts like these:
|
||||
|
||||
- "Email me a daily brief of new documents in my knowledge base every morning."
|
||||
- "Generate a weekly status report from my Slack and Gmail every Friday."
|
||||
- "Run a monthly competitor analysis report and save it to my workspace."
|
||||
- "Summarize my GitHub and Linear activity into a daily standup update."
|
||||
- "Create a recurring weekly research report on the topics I track."
|
||||
|
||||
- Event-Triggered Automations: fire an agent the moment a document lands in a folder, then post the result to your tools.
|
||||
Try prompts like these:
|
||||
|
||||
- "When a PDF lands in my Research folder, generate a cited AI summary."
|
||||
- "When new meeting notes are added, turn them into meeting minutes with action items."
|
||||
- "When an invoice is uploaded, extract the vendor, total, and due date into a table."
|
||||
- "When a contract enters my Legal folder, flag key terms and renewal dates."
|
||||
- "When a resume is added to Candidates, screen it against the job description."
|
||||
|
||||
- Chat-Built Automations: describe an automation in plain English and SurfSense builds it for you.
|
||||
Try prompts like these:
|
||||
|
||||
- "Build an AI agent that emails me a summary of new Notion pages each morning."
|
||||
- "Create a no-code automation that posts a weekly research digest to Slack."
|
||||
- "Set up an AI note taker that turns new meeting notes into minutes."
|
||||
- "Make a workflow that extracts action items from meeting notes and assigns owners."
|
||||
- "Automate a daily email brief from my Gmail and Google Drive."
|
||||
|
||||
|
||||
### Self Hosted
|
||||
|
||||
Run SurfSense on your own infrastructure for full data control and privacy.
|
||||
Run the entire platform, connectors, agents, automations, and the MCP server on your own infrastructure. Self-hosted installs ship with billing off, so scraping, crawling, and agent runs are limited only by your hardware and the model keys you bring.
|
||||
|
||||
**Prerequisites:** [Docker Desktop](https://www.docker.com/products/docker-desktop/) must be installed and running.
|
||||
|
||||
#### For Linux/MacOS users:
|
||||
For Linux/macOS:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash
|
||||
```
|
||||
|
||||
#### For Windows users:
|
||||
For Windows:
|
||||
|
||||
```bash
|
||||
irm https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.ps1 | iex
|
||||
```
|
||||
|
||||
The install script sets up [Watchtower](https://github.com/nicholas-fedor/watchtower) automatically for daily auto-updates. To skip it, add the `--no-watchtower` flag.
|
||||
The install script sets up [Watchtower](https://github.com/nicholas-fedor/watchtower) automatically for daily auto-updates. To skip it, add the `--no-watchtower` flag. For Docker Compose, manual installation, and other deployment options, see the [docs](https://www.surfsense.com/docs/).
|
||||
|
||||
For Docker Compose, manual installation, and other deployment options, see the [docs](https://www.surfsense.com/docs/).
|
||||
## Everything else in the box
|
||||
|
||||
### Desktop App
|
||||
The research workspace that made SurfSense the leading open-source NotebookLM alternative is still here, and everything your agents gather lands in it.
|
||||
|
||||
SurfSense also ships a desktop app that brings AI assistance to every application on your computer. Download it from the [latest release](https://github.com/MODSetter/SurfSense/releases/latest).
|
||||
**Knowledge base**
|
||||
|
||||
The desktop app includes these powerful features:
|
||||
- Upload PDFs, Office docs, images, and audio, or sync **Google Drive, OneDrive, and Dropbox**. 50+ file formats supported.
|
||||
- Hybrid semantic and full-text search with cited, Perplexity-style answers.
|
||||
- AI file sorting auto-organizes documents by source, date, and topic.
|
||||
|
||||
- **General Assist** — Launch SurfSense instantly from any application with a global shortcut.
|
||||
- **Quick Assist** — Select text anywhere, then ask AI to explain, rewrite, or act on it.
|
||||
- **Screenshot Assist** — Select a region on your screen and attach it to chat so answers stay grounded in your knowledge base.
|
||||
- **Watch Local Folder** — Watch a local folder and automatically sync file changes to your knowledge base. **Pro tip:** Point it at your Obsidian vault to keep your notes searchable in SurfSense.
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="Chat With Your PDFs and Docs" /></p>
|
||||
|
||||
All features operate against your chosen search space, so your answers are always grounded in your own data.
|
||||
**Deliverable studio**
|
||||
|
||||
### How to Realtime Collaborate (Beta)
|
||||
- AI report generator with export to PDF, DOCX, HTML, LaTeX, EPUB, ODT, or plain text.
|
||||
- Two-host AI podcasts from any document or folder in under 20 seconds.
|
||||
- Editable slide decks, narrated video overviews, and AI image generation.
|
||||
|
||||
1. Go to Manage Members page and create an invite.
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="AI Report Generator" /></p>
|
||||
|
||||
**Team collaboration**
|
||||
|
||||
- Real-time collaborative AI chats with comments and mentions.
|
||||
- RBAC with Owner, Admin, Editor, and Viewer roles.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Collaborative AI Chat" /></p>
|
||||
|
||||
**Desktop app**
|
||||
|
||||
Native AI assistance in every application on your computer. Download from the [latest release](https://github.com/MODSetter/SurfSense/releases/latest).
|
||||
|
||||
- **General Assist**: launch SurfSense from any app with a global shortcut.
|
||||
- **Quick Assist**: select text anywhere, then ask AI to explain, rewrite, or act on it.
|
||||
- **Screenshot Assist**: capture any region of your screen and ask AI about it.
|
||||
- **Watch Local Folder**: auto-sync a local folder to your knowledge base. Point it at your Obsidian vault to keep your notes searchable.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
|
||||
|
||||
**No vendor lock-in**
|
||||
|
||||
- 100+ LLMs via the OpenAI spec and LiteLLM, including GPT-5.5, Claude Sonnet 5, and Gemini 3.1 Pro.
|
||||
- 6,000+ embedding models and all major rerankers.
|
||||
- Full local and private LLM support (vLLM, Ollama), so your data stays yours.
|
||||
|
||||
## Video Agent Sample
|
||||
|
||||
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
|
||||
|
||||
## Podcast Agent Sample
|
||||
|
||||
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
|
||||
|
||||
## How to collaborate in real time (Beta)
|
||||
|
||||
1. Go to the Manage Members page and create an invite.
|
||||
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/40ed7683-5aa6-48a0-a3df-00575528c392" alt="Invite Members" /></p>
|
||||
|
||||
2. Teammate joins and that SearchSpace becomes shared.
|
||||
2. A teammate joins and that workspace becomes shared.
|
||||
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/ea4e1057-4d2b-4fd2-9ca0-cd19286a285e" alt="Invite Join Flow" /></p>
|
||||
|
||||
3. Make chat shared.
|
||||
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/17b93904-0888-4c3a-ac12-51a24a8ea26a" alt="Make Chat Shared" /></p>
|
||||
|
||||
4. Your team can now chat in realtime.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Realtime Chat" /></p>
|
||||
|
||||
5. Add comment to tag teammates.
|
||||
3. Make a chat shared and work in it together in real time, with comments to tag teammates.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Realtime Comments" /></p>
|
||||
|
||||
## SurfSense vs Google NotebookLM
|
||||
|
||||
Still comparing us as a NotebookLM alternative? Here is the honest breakdown.
|
||||
|
||||
| Feature | Google NotebookLM | SurfSense |
|
||||
|---------|-------------------|-----------|
|
||||
| **Live market data for agents** | No | Reddit, YouTube, Google Maps, Google Search, and web crawl connectors via REST API and MCP |
|
||||
| **MCP server** | No | Every connector exposed as a native agent tool, plus bring-your-own MCP servers with one-click OAuth apps |
|
||||
| **Sources per Notebook** | 50 (Free) to 600 (Ultra, $249.99/mo) | Unlimited |
|
||||
| **Number of Notebooks** | 100 (Free) to 500 (paid tiers) | Unlimited |
|
||||
| **Source Size Limit** | 500,000 words / 200MB per source | No limit |
|
||||
| **Pricing** | Free tier available; Pro $19.99/mo, Ultra $249.99/mo | Free and open source, self-host on your own infra |
|
||||
| **Pricing** | Free tier; Pro $19.99/mo, Ultra $249.99/mo | Free and open source to self-host; cloud is pay as you go with $5 free credit |
|
||||
| **LLM Support** | Google Gemini only | 100+ LLMs via OpenAI spec & LiteLLM |
|
||||
| **Embedding Models** | Google only | 6,000+ embedding models, all major rerankers |
|
||||
| **Local / Private LLMs** | Not available | Full support (vLLM, Ollama) - your data stays yours |
|
||||
| **Self Hostable** | No | Yes - Docker one-liner or full Docker Compose |
|
||||
| **Local / Private LLMs** | Not available | Full support (vLLM, Ollama), your data stays yours |
|
||||
| **Self Hostable** | No | Yes, Docker one-liner or full Docker Compose |
|
||||
| **Open Source** | No | Yes |
|
||||
| **External Connectors** | Google Drive, YouTube, websites | 27+ connectors - Search Engines, Google Drive, OneDrive, Dropbox, Slack, Teams, Jira, Notion, GitHub, Discord & [more](#external-sources) |
|
||||
| **File Format Support** | PDFs, Docs, Slides, Sheets, CSV, Word, EPUB, images, web URLs, YouTube | 50+ formats - documents, images, videos via LlamaCloud, Unstructured, or Docling (local) |
|
||||
| **Search** | Semantic search | Hybrid Search - Semantic + Full Text with Hierarchical Indices & Reciprocal Rank Fusion |
|
||||
| **Cited Answers** | Yes | Yes - Perplexity-style cited responses |
|
||||
| **Agentic Architecture** | No | Yes - powered by [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) with planning, subagents, and file system access |
|
||||
| **Knowledge Base Sources** | Google Drive, YouTube, websites | File uploads, Google Drive, OneDrive, Dropbox, local folder sync, and crawled pages |
|
||||
| **File Format Support** | PDFs, Docs, Slides, Sheets, CSV, Word, EPUB, images, web URLs, YouTube | 50+ formats: documents, images, videos via LlamaCloud, Unstructured, or Docling (local) |
|
||||
| **Search** | Semantic search | Hybrid semantic + full-text with hierarchical indices & reciprocal rank fusion |
|
||||
| **Cited Answers** | Yes | Yes, Perplexity-style cited responses |
|
||||
| **Agentic Architecture** | No | Yes, powered by [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) with planning, subagents, and file system access |
|
||||
| **AI Automations & Agents** | No | Scheduled workflows, event triggers, and chat-built no-code automations with write-back to Notion, Slack, Linear & Jira |
|
||||
| **Real-Time Multiplayer** | Shared notebooks with Viewer/Editor roles (no real-time chat) | RBAC with Owner / Admin / Editor / Viewer roles, real-time chat & comment threads |
|
||||
| **Video Generation** | Cinematic Video Overviews via Veo 3 (Ultra only) | Available (NotebookLM is better here, actively improving) |
|
||||
| **Presentation Generation** | Better looking slides but not editable | Create editable, slide-based presentations |
|
||||
| **Presentation Generation** | Better looking slides but not editable | Editable, slide-based presentations |
|
||||
| **Podcast Generation** | Audio Overviews with customizable hosts and languages | Available with multiple TTS providers (NotebookLM is better here, actively improving) |
|
||||
| **AI File Sorting** | No | LLM-powered auto-categorization into source, date, category, and subcategory folders |
|
||||
| **AI Automations & Agents** | No | Scheduled AI workflows, event triggers on new documents, and chat-built no-code automations with connector write-back to Notion, Slack, Linear & Jira |
|
||||
| **Desktop App** | No | Native app with General Assist, Quick Assist, Screenshot Assist, and local folder sync |
|
||||
| **Browser Extension** | No | Cross-browser extension to save any webpage, including auth-protected pages |
|
||||
|
||||
<details>
|
||||
<summary><b>Full list of External Sources</b></summary>
|
||||
<a id="external-sources"></a>
|
||||
|
||||
Search Engines (SearXNG, Tavily, LinkUp, Baidu Search) · Google Drive · OneDrive · Dropbox · Slack · Microsoft Teams · Linear · Jira · ClickUp · Confluence · BookStack · Notion · Gmail · YouTube Videos · GitHub · Discord · Airtable · Google Calendar · Luma · Circleback · Elasticsearch · Obsidian, and more to come.
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
## FEATURE REQUESTS AND FUTURE
|
||||
|
||||
## Feature requests and future
|
||||
|
||||
**SurfSense is actively being developed.** While it's not yet production-ready, you can help us speed up the process.
|
||||
|
||||
|
|
@ -294,14 +274,12 @@ Join the [SurfSense Discord](https://discord.gg/ejRNvftDp9) and help shape the f
|
|||
|
||||
## Roadmap
|
||||
|
||||
Stay up to date with our development progress and upcoming features!
|
||||
Check out our public roadmap and contribute your ideas or feedback:
|
||||
Stay up to date with our development progress and upcoming features. Check out our public roadmap and contribute your ideas or feedback:
|
||||
|
||||
**Roadmap Discussion:** [SurfSense 2026 Roadmap](https://github.com/MODSetter/SurfSense/discussions/565)
|
||||
|
||||
**Kanban Board:** [SurfSense Project Board](https://github.com/users/MODSetter/projects/3)
|
||||
|
||||
|
||||
## Contribute
|
||||
|
||||
All contributions welcome, from stars and bug reports to backend improvements. See [CONTRIBUTING.md](CONTRIBUTING.md) to get started.
|
||||
|
|
|
|||
383
README.pt-BR.md
383
README.pt-BR.md
|
|
@ -1,4 +1,4 @@
|
|||
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="readme_banner" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
|
||||
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="SurfSense, a plataforma open source de inteligência competitiva para agentes de IA" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
|
||||
|
||||
|
||||
|
||||
|
|
@ -20,289 +20,270 @@
|
|||
<a href="https://trendshift.io/repositories/13606" target="_blank"><img src="https://trendshift.io/api/badge/repositories/13606" alt="MODSetter%2FSurfSense | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
|
||||
</div>
|
||||
|
||||
# SurfSense
|
||||
# SurfSense: Dê Inteligência Competitiva aos Seus Agentes de IA
|
||||
|
||||
O NotebookLM é uma das melhores e mais úteis plataformas de IA disponíveis, mas quando você começa a usá-lo regularmente também sente suas limitações deixando algo a desejar.
|
||||
O SurfSense é a **plataforma open source de inteligência competitiva para agentes de IA**. Seus agentes monitoram concorrentes, acompanham rankings e escutam o seu mercado com dados ao vivo do **Reddit, YouTube, Google Maps, Google Search e da web aberta**, por meio de uma única **API REST** ou de um **servidor MCP**. Agentes agendados ou acionados por eventos transformam o que encontram em relatórios e alertas, e uma base de conhecimento integrada mantém cada descoberta pesquisável, com citações.
|
||||
|
||||
1. Há limites na quantidade de fontes que você pode adicionar em um notebook.
|
||||
2. Há limites no número de notebooks que você pode ter.
|
||||
3. Você não pode ter fontes que excedam 500.000 palavras e mais de 200MB.
|
||||
4. Você fica preso aos serviços do Google (LLMs, modelos de uso, etc.) sem opção de configurá-los.
|
||||
5. Fontes de dados externas e integrações de serviços limitadas.
|
||||
6. O agente do NotebookLM é especificamente otimizado apenas para estudar e pesquisar, mas você pode fazer muito mais com os dados de origem.
|
||||
7. Falta de suporte multiplayer.
|
||||
> [!NOTE]
|
||||
> **📢 Um recado para nossos usuários que buscavam uma alternativa ao NotebookLM**
|
||||
>
|
||||
> Nos últimos meses, construímos o SurfSense como o melhor agente de pesquisa geral para o seu próprio conhecimento, e esse capítulo nos rendeu uma comunidade da qual temos muito orgulho. Ferramentas agênticas como Claude, OpenCode, Hermes e OpenClaw já provaram que os agentes são o futuro, e a pesquisa geral está se tornando algo que todo agente capaz faz nativamente. O que ainda falta aos agentes são **dados de mercado ao vivo e os fluxos de trabalho em torno deles**, então é para lá que estamos direcionando toda a nossa energia: nos tornar a plataforma open source definitiva de agentes de inteligência competitiva.
|
||||
>
|
||||
> **Nada do que você usa vai deixar de existir.** Sua base de conhecimento, o chat com citações, os relatórios, os podcasts, as apresentações, as automações e os chats colaborativos continuam funcionando, e a auto-hospedagem segue gratuita e open source. Leia o anúncio completo no [nosso changelog](https://www.surfsense.com/changelog).
|
||||
|
||||
...e mais.
|
||||
## Sumário
|
||||
|
||||
**O SurfSense foi feito especificamente para resolver esses problemas.** O SurfSense permite que você:
|
||||
- [Por que os agentes precisam do SurfSense](#por-que-os-agentes-precisam-do-surfsense)
|
||||
- [O que você pode fazer com o SurfSense?](#o-que-você-pode-fazer-com-o-surfsense)
|
||||
- [Conectores de dados ao vivo](#conectores-de-dados-ao-vivo)
|
||||
- [Início rápido](#início-rápido)
|
||||
- [Tudo o mais que vem na caixa](#tudo-o-mais-que-vem-na-caixa)
|
||||
- [SurfSense vs Google NotebookLM](#surfsense-vs-google-notebooklm)
|
||||
- [Roadmap](#roadmap)
|
||||
- [Contribua](#contribua)
|
||||
|
||||
- **Controle Seu Fluxo de Dados** - Mantenha seus dados privados e seguros.
|
||||
- **Sem Limites de Dados** - Adicione uma quantidade ilimitada de fontes e notebooks.
|
||||
- **Sem Dependência de Fornecedor** - Configure qualquer modelo LLM, de imagem, TTS e STT.
|
||||
- **25+ Fontes de Dados Externas** - Adicione suas fontes do Google Drive, OneDrive, Dropbox, Notion e muitos outros serviços externos.
|
||||
- **Suporte Multiplayer em Tempo Real** - Trabalhe facilmente com os membros da sua equipe em um notebook compartilhado.
|
||||
- **Automações e Agentes de IA** - Execute agentes de IA em uma programação ou dispare-os no momento em que um documento chega a uma pasta, e escreva os resultados de volta no Notion, Slack, Linear e Drive. Crie automações sem código apenas descrevendo-as no chat.
|
||||
- **Aplicativo Desktop** - Obtenha assistência de IA em qualquer aplicativo com Quick Assist, General Assist, Screenshot Assist e sincronização de pastas locais.
|
||||
## Por que os agentes precisam do SurfSense
|
||||
|
||||
...e mais por vir.
|
||||
Pergunte a qualquer agente capaz "quanto os concorrentes estão cobrando esta semana?" ou "o que o Reddit está dizendo sobre nós desde o lançamento?" e ele não terá nenhum lugar confiável para procurar. As APIs oficiais das plataformas têm limites de requisições, preços voltados para empresas ou simplesmente não existem, e a infraestrutura de scraping é frágil. O SurfSense fecha essa lacuna:
|
||||
|
||||
- **Conectores nativos de cada plataforma**, cada um sendo um endpoint REST tipado que retorna JSON estruturado. Sem roleta de limites de requisição, sem parsing de HTML.
|
||||
- **Um servidor MCP** que expõe cada conector como uma ferramenta nativa (`surfsense_reddit_scrape`, `surfsense_google_search` e outras) para o Claude, o Cursor ou qualquer framework de agentes.
|
||||
- **Um harness de agentes**, não apenas dados brutos: novas tentativas, saída estruturada e medição de créditos já vêm prontos, então os agentes vão de uma pergunta a um relatório sem que você precise construir a infraestrutura.
|
||||
- **Open source e auto-hospedável**, para que sua pesquisa competitiva permaneça na sua própria infraestrutura.
|
||||
|
||||
## O que você pode fazer com o SurfSense?
|
||||
|
||||
## Exemplo de Agente de Vídeo
|
||||
Cada caso de uso abaixo é uma tarefa real que o agente do SurfSense executa de ponta a ponta hoje, em um único prompt ou de forma agendada.
|
||||
|
||||
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
|
||||
### Fluxos de trabalho multi-conector
|
||||
|
||||
Encadeie vários conectores em uma única execução do agente e receba um único relatório com citações.
|
||||
|
||||
- **Impacto de um lançamento, em todas as plataformas** — "Nosso concorrente lançou a v2 ontem. Meça a reação nas buscas, no Reddit e no YouTube." O agente raspa SERPs, threads do Reddit e comentários do YouTube, e depois funde os três sinais em um único relatório de impacto do lançamento.
|
||||
- **Análise de concorrentes locais** — o Google Maps encontra os players, o rastreador web lê suas páginas de preços e o Google Search mostra quem vence a busca, tudo em uma única execução.
|
||||
- **Concorrente 360, de forma agendada** — uma automação encadeia quatro conectores toda semana: mudanças no site, movimentos de ranking, sentimento no Reddit e reação no YouTube.
|
||||
|
||||
## Exemplo de Agente de Podcast
|
||||
### Monitoramento de concorrentes
|
||||
|
||||
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
|
||||
- **Vigilância de preços** — extraia cada plano, preço e limite das páginas de preços dos concorrentes em uma única tabela, depois verifique novamente todo dia e receba alertas sobre qualquer mudança.
|
||||
- **Acompanhamento de produto e changelog** — rastreie as páginas de changelog, produto e vagas dos rivais toda segunda-feira e receba um relatório do que eles lançaram.
|
||||
- **Monitoramento de rankings e anúncios** — acompanhe os rankings do Google, os anúncios pagos e as citações em AI Overviews que o seu mercado realmente vê, e sinalize os movimentos dia a dia.
|
||||
|
||||
### Geração de leads B2B
|
||||
|
||||
## Como Usar o SurfSense
|
||||
- **Leads de negócios locais** — transforme uma categoria e um território ("hamburguerias em San Jose") em uma lista de leads com telefones, sites, avaliações e contatos de decisores extraídos dos sites deles.
|
||||
- **Equipes e contatos** — rastreie o site de qualquer empresa e extraia a equipe completa com e-mails, redes sociais e a origem de cada dado, exportado para CSV.
|
||||
- **Mapeamento de portfólios e mercados** — mapeie o portfólio de um investidor ou uma categoria inteira, e depois enriqueça cada empresa com preços e contatos.
|
||||
|
||||
### Cloud
|
||||
### Escuta de marca e mercado
|
||||
|
||||
1. Acesse [surfsense.com](https://www.surfsense.com) e faça login.
|
||||
- **Monitoramento de marca no Reddit** — ouça o que o seu mercado diz sobre você, seus concorrentes e sua categoria nas threads onde os compradores falam com franqueza.
|
||||
- **Sentimento da audiência no YouTube** — colete vídeos, transcrições e comentários em escala, e depois agrupe o que as audiências elogiam e criticam.
|
||||
- **Mineração de intenção e de quem quer trocar** — encontre as pessoas que estão ativamente buscando uma alternativa a um concorrente, ordenadas por quão prontas estão para migrar.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/LoginFlowGif.gif" alt="Login" /></p>
|
||||
### Pesquisa de mercado
|
||||
|
||||
2. Conecte seus conectores e sincronize. Ative a sincronização periódica para manter os conectores atualizados.
|
||||
- **Pesquisa profunda na web ao vivo** — o agente rastreia dezenas de fontes ao vivo sobre uma pergunta e sintetiza uma resposta com citações, não um índice desatualizado.
|
||||
- **Acompanhamento de AI Overviews e GEO** — capture quando os AI Overviews do Google respondem às buscas do seu mercado, e exatamente quais fontes eles citam.
|
||||
- **Relatórios e alertas com citações** — tudo o que os agentes coletam chega ao seu workspace como relatórios e alertas com fontes que você pode verificar.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="Conectores" /></p>
|
||||
### Automatize qualquer uma dessas tarefas, sem código
|
||||
|
||||
3. Enquanto os dados dos conectores são indexados, faça upload de documentos.
|
||||
As automações executam turnos completos de agente de forma agendada ou em resposta a eventos, e depois gravam os resultados no Notion, Slack, Linear e Jira. Descreva o fluxo de trabalho em linguagem natural e o SurfSense o constrói. Experimente prompts como:
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="Upload de Documentos" /></p>
|
||||
- "Monitore as páginas de preços dos nossos 3 principais concorrentes e me avise no Slack quando um plano ou preço mudar."
|
||||
- "Acompanhe cada menção à nossa marca no Reddit e no YouTube e me envie um resumo diário."
|
||||
- "Monitore nosso ranking no Google para nossas 10 principais palavras-chave e sinalize quedas semana a semana."
|
||||
- "Colete os novos reviews do Google Maps das nossas unidades e das dos concorrentes toda segunda-feira."
|
||||
- "Execute um relatório mensal de análise de concorrentes e salve no meu workspace."
|
||||
|
||||
4. Quando tudo estiver indexado, pergunte o que quiser (Casos de uso):
|
||||
## Conectores de dados ao vivo
|
||||
|
||||
**Aplicativo Desktop** (extras nativos, além de tudo o que está abaixo, não um conjunto separado)
|
||||
| Conector | O que seus agentes recebem | Saiba mais |
|
||||
|---|---|---|
|
||||
| **Reddit** | Posts, comentários e fluxos de subreddits sem os limites de requisição da API oficial | [Reddit Scraper API](https://www.surfsense.com/reddit) |
|
||||
| **YouTube** | Vídeos, transcrições e threads de comentários para monitoramento de marca e produto | [YouTube Scraper API](https://www.surfsense.com/youtube) |
|
||||
| **Google Maps** | Estabelecimentos, avaliações e reviews para pesquisa local de concorrentes e leads | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
|
||||
| **Google Search** | SERPs ao vivo para acompanhamento de rankings e monitoramento de mercado | [Google Search API](https://www.surfsense.com/google-search) |
|
||||
| **Web Crawl** (rastreamento web) | Qualquer página da web aberta como conteúdo limpo e estruturado | [Web Crawling API](https://www.surfsense.com/web-crawl) |
|
||||
| **Conectores MCP externos** | Traga qualquer servidor MCP para seus agentes, com OAuth em um clique para Notion, Slack, Jira e outros | [External MCP Connectors](https://www.surfsense.com/external-mcp-connectors) |
|
||||
|
||||
- General Assist: abra o SurfSense instantaneamente de qualquer aplicativo com um atalho global.
|
||||
A cobrança é por uso: os conectores cobram por item efetivamente retornado, os rastreamentos por página obtida com sucesso, e chamadas com falha nunca são cobradas. Instalações auto-hospedadas rodam com a cobrança desativada. Veja os [preços](https://www.surfsense.com/pricing).
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/general_assist.gif" alt="General Assist" /></p>
|
||||
## Início rápido
|
||||
|
||||
- Quick Assist: selecione um texto em qualquer lugar e peça à IA para explicar, reescrever ou agir sobre ele.
|
||||
### Chame um conector a partir do código
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
|
||||
Cada conector é um endpoint REST que você pode chamar de qualquer linguagem com a sua chave de API do SurfSense:
|
||||
|
||||
- Screenshot Assist: capture qualquer região da tela e pergunte à IA sobre o que está nela.
|
||||
```bash
|
||||
curl -X POST "$SURFSENSE_API_URL/workspaces/$WORKSPACE_ID/scrapers/reddit/scrape" \
|
||||
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"search_queries": ["your brand"],
|
||||
"community": "webscraping",
|
||||
"sort": "top",
|
||||
"time_filter": "week"
|
||||
}'
|
||||
```
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/screenshot_assist.gif" alt="Screenshot Assist" /></p>
|
||||
Cada [página de conector](https://www.surfsense.com/connectors) tem exemplos prontos para copiar e colar em Python, JavaScript, Go, PHP, Ruby, Java e C#.
|
||||
|
||||
- Watch Local Folder: sincronize automaticamente uma pasta local com sua base de conhecimento. Ótimo para cofres do Obsidian.
|
||||
### Entregue as ferramentas aos seus agentes via MCP
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/folder_watch.gif" alt="Watch Local Folder" /></p>
|
||||
Adicione o servidor MCP do SurfSense ao Claude, ao Cursor ou ao seu próprio framework de agentes:
|
||||
|
||||
**Estúdio de Entregáveis**
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"surfsense": {
|
||||
"url": "https://mcp.surfsense.com",
|
||||
"headers": { "Authorization": "Bearer ${SURFSENSE_API_KEY}" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- AI Report Generator: gere relatórios de pesquisa com citações e exporte para PDF, DOCX, HTML, LaTeX, EPUB, ODT ou texto simples.
|
||||
Seu agente agora pode chamar cada conector como uma ferramenta nativa. Veja a página do [servidor MCP do SurfSense](https://www.surfsense.com/mcp-server) para a lista completa de ferramentas, ou execute o servidor localmente a partir de [`surfsense_mcp`](./surfsense_mcp).
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="AI Report Generator" /></p>
|
||||
### Use a nuvem
|
||||
|
||||
- AI Podcast Generator: transforme qualquer documento ou pasta em um podcast de IA com dois apresentadores em menos de 20 segundos.
|
||||
Acesse [surfsense.com](https://www.surfsense.com), faça login e peça ao agente dados de mercado ao vivo em linguagem natural. Contas novas começam com US$ 5 de crédito gratuito e sem assinatura.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/PodcastGenGif.gif" alt="AI Podcast Generator" /></p>
|
||||
|
||||
- AI Presentation & Video Maker: crie apresentações editáveis e vídeos narrados a partir das suas fontes.
|
||||
### Auto-hospede gratuitamente
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/video_gen_gif.gif" alt="AI Presentation and Video Maker" /></p>
|
||||
Rode a plataforma inteira, conectores, agentes, automações e o servidor MCP na sua própria infraestrutura. Instalações auto-hospedadas vêm com a cobrança desativada, então scraping, rastreamento e execuções de agentes são limitados apenas pelo seu hardware e pelas chaves de modelo que você trouxer.
|
||||
|
||||
- AI Image Generator: gere imagens de alta qualidade diretamente das suas conversas e documentos.
|
||||
**Pré-requisitos:** o [Docker Desktop](https://www.docker.com/products/docker-desktop/) precisa estar instalado e em execução.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ImageGenGif.gif" alt="AI Image Generator" /></p>
|
||||
|
||||
- AI Resume Builder: adapte seu currículo atual a qualquer descrição de vaga e supere o ATS.
|
||||
Experimente prompts como estes:
|
||||
|
||||
- "Adapte meu currículo a esta descrição de vaga para passar pelo ATS e conseguir uma entrevista."
|
||||
- "Otimize meu currículo para o ATS combinando as palavras-chave desta vaga."
|
||||
- "Reescreva os tópicos do meu currículo para destacar as habilidades que esta vaga exige."
|
||||
- "Compare meu currículo com esta descrição de vaga e liste as lacunas a corrigir."
|
||||
- "Escreva uma carta de apresentação combinando com meu currículo e esta descrição de vaga."
|
||||
|
||||
**Busca e Chat**
|
||||
|
||||
- Chat With Your PDFs & Docs: faça perguntas sobre todos os seus arquivos e receba respostas com citações inline.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="Chat With Your PDFs and Docs" /></p>
|
||||
|
||||
- AI Search With Citations: busca híbrida semântica e por palavra-chave em toda a sua base de conhecimento.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BSNCGif.gif" alt="AI Search With Citations" /></p>
|
||||
|
||||
- Collaborative AI Chat: trabalhe em conversas de IA com sua equipe em tempo real.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Collaborative AI Chat" /></p>
|
||||
|
||||
- Comments & Mentions: comente e marque colegas em qualquer mensagem de IA.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Comments and Mentions" /></p>
|
||||
|
||||
**Conectores e Integrações**
|
||||
|
||||
- Connect & Sync Your Tools: sincronize Notion, Slack, Google Drive, Gmail, GitHub, Linear e mais de 25 fontes em um único acervo pesquisável.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="Connect and Sync Your Tools" /></p>
|
||||
|
||||
- Chat With Uploaded Files: envie PDFs, documentos do Office, imagens e áudio. Pesquisáveis instantaneamente.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="Chat With Uploaded Files" /></p>
|
||||
|
||||
- Connector Write-Back: deixe o agente publicar os resultados de volta no Notion, Slack, Linear e Drive.
|
||||
Experimente prompts como estes:
|
||||
|
||||
- "Publique este resumo de pesquisa no meu espaço do Notion."
|
||||
- "Envie estes itens de ação da reunião para o nosso canal do Slack."
|
||||
- "Crie um ticket no Jira a partir deste relatório de bug."
|
||||
- "Abra uma issue no Linear a partir desta solicitação de funcionalidade."
|
||||
- "Salve este relatório gerado no Google Drive como um documento."
|
||||
|
||||
- Obsidian & Knowledge Base Sync: mantenha seu cofre do Obsidian e sua base de conhecimento pessoal sincronizados.
|
||||
|
||||
**Automações**
|
||||
|
||||
- Scheduled AI Workflows: execute um agente em uma programação: resumos diários, boletins semanais, relatórios recorrentes.
|
||||
Experimente prompts como estes:
|
||||
|
||||
- "Envie-me todas as manhãs um resumo diário dos novos documentos na minha base de conhecimento."
|
||||
- "Gere um relatório de status semanal a partir do meu Slack e Gmail toda sexta-feira."
|
||||
- "Execute um relatório mensal de análise da concorrência e salve-o no meu espaço de trabalho."
|
||||
- "Resuma minha atividade no GitHub e Linear em uma atualização diária de standup."
|
||||
- "Crie um relatório de pesquisa semanal recorrente sobre os temas que acompanho."
|
||||
|
||||
- Event-Triggered Automations: dispare um agente no momento em que um documento chega a uma pasta e publique o resultado nas suas ferramentas.
|
||||
Experimente prompts como estes:
|
||||
|
||||
- "Quando um PDF chegar à minha pasta de Pesquisa, gere um resumo com IA e citações."
|
||||
- "Quando novas notas de reunião forem adicionadas, transforme-as em atas com itens de ação."
|
||||
- "Quando uma fatura for enviada, extraia o fornecedor, o total e a data de vencimento em uma tabela."
|
||||
- "Quando um contrato entrar na minha pasta Jurídica, sinalize os termos-chave e as datas de renovação."
|
||||
- "Quando um currículo for adicionado a Candidatos, avalie-o em relação à descrição da vaga."
|
||||
|
||||
- Chat-Built Automations: descreva uma automação em linguagem simples e o SurfSense a cria para você.
|
||||
Experimente prompts como estes:
|
||||
|
||||
- "Crie um agente de IA que me envie todas as manhãs um resumo das novas páginas do Notion."
|
||||
- "Crie uma automação sem código que publique um resumo de pesquisa semanal no Slack."
|
||||
- "Configure um anotador com IA que transforme as novas notas de reunião em atas."
|
||||
- "Crie um fluxo que extraia os itens de ação das notas de reunião e atribua responsáveis."
|
||||
- "Automatize um resumo diário por e-mail a partir do meu Gmail e Google Drive."
|
||||
|
||||
|
||||
### Auto-Hospedado
|
||||
|
||||
Execute o SurfSense na sua própria infraestrutura para controle total de dados e privacidade.
|
||||
|
||||
**Pré-requisitos:** [Docker Desktop](https://www.docker.com/products/docker-desktop/) deve estar instalado e em execução.
|
||||
|
||||
#### Para usuários de Linux/MacOS:
|
||||
Para Linux/macOS:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash
|
||||
```
|
||||
|
||||
#### Para usuários do Windows:
|
||||
Para Windows:
|
||||
|
||||
```powershell
|
||||
```bash
|
||||
irm https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.ps1 | iex
|
||||
```
|
||||
|
||||
O script de instalação configura o [Watchtower](https://github.com/nicholas-fedor/watchtower) automaticamente para atualizações diárias. Para pular, adicione a flag `--no-watchtower`.
|
||||
O script de instalação configura o [Watchtower](https://github.com/nicholas-fedor/watchtower) automaticamente para atualizações automáticas diárias. Para pular essa etapa, adicione a flag `--no-watchtower`. Para Docker Compose, instalação manual e outras opções de implantação, consulte a [documentação](https://www.surfsense.com/docs/).
|
||||
|
||||
Para Docker Compose, instalação manual e outras opções de implantação, consulte a [documentação](https://www.surfsense.com/docs/).
|
||||
## Tudo o mais que vem na caixa
|
||||
|
||||
### Aplicativo Desktop
|
||||
O workspace de pesquisa que fez do SurfSense a principal alternativa open source ao NotebookLM continua aqui, e tudo o que seus agentes coletam chega até ele.
|
||||
|
||||
O SurfSense também oferece um aplicativo desktop que traz assistência de IA para cada aplicativo no seu computador. Baixe-o na [última versão](https://github.com/MODSetter/SurfSense/releases/latest).
|
||||
**Base de conhecimento**
|
||||
|
||||
O aplicativo desktop inclui estes recursos poderosos:
|
||||
- Envie PDFs, documentos do Office, imagens e áudio, ou sincronize **Google Drive, OneDrive e Dropbox**. Mais de 50 formatos de arquivo suportados.
|
||||
- Busca híbrida semântica e de texto completo, com respostas citadas no estilo Perplexity.
|
||||
- Organização de arquivos por IA que classifica automaticamente os documentos por origem, data e tópico.
|
||||
|
||||
- **General Assist** — Abra o SurfSense instantaneamente de qualquer aplicativo com um atalho global.
|
||||
- **Quick Assist** — Selecione texto em qualquer lugar, depois peça à IA para explicar, reescrever ou agir sobre ele.
|
||||
- **Screenshot Assist** — Selecione uma região da tela e anexe ao chat para respostas fundamentadas na sua base de conhecimento.
|
||||
- **Watch Local Folder** — Monitore uma pasta local e sincronize automaticamente as alterações de arquivos com sua base de conhecimento. **Pro tip:** Aponte para seu cofre do Obsidian para manter suas notas pesquisáveis no SurfSense.
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="Converse com seus PDFs e documentos" /></p>
|
||||
|
||||
Todos os recursos operam no espaço de busca escolhido, para que suas respostas sejam sempre baseadas nos seus próprios dados.
|
||||
**Estúdio de entregáveis**
|
||||
|
||||
### Como Colaborar em Tempo Real (Beta)
|
||||
- Gerador de relatórios com IA, com exportação para PDF, DOCX, HTML, LaTeX, EPUB, ODT ou texto simples.
|
||||
- Podcasts de IA com dois apresentadores a partir de qualquer documento ou pasta em menos de 20 segundos.
|
||||
- Apresentações de slides editáveis, resumos em vídeo narrados e geração de imagens por IA.
|
||||
|
||||
1. Acesse a página de Gerenciar Membros e crie um convite.
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="Gerador de Relatórios com IA" /></p>
|
||||
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/40ed7683-5aa6-48a0-a3df-00575528c392" alt="Convidar Membros" /></p>
|
||||
**Colaboração em equipe**
|
||||
|
||||
2. O colega aceita e aquele SearchSpace se torna compartilhado.
|
||||
- Chats de IA colaborativos em tempo real, com comentários e menções.
|
||||
- RBAC com os papéis de Owner, Admin, Editor e Viewer.
|
||||
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/ea4e1057-4d2b-4fd2-9ca0-cd19286a285e" alt="Fluxo de Entrada por Convite" /></p>
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Chat de IA colaborativo" /></p>
|
||||
|
||||
3. Torne o chat compartilhado.
|
||||
**Aplicativo desktop**
|
||||
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/17b93904-0888-4c3a-ac12-51a24a8ea26a" alt="Tornar Chat Compartilhado" /></p>
|
||||
Assistência de IA nativa em todos os aplicativos do seu computador. Baixe na [versão mais recente](https://github.com/MODSetter/SurfSense/releases/latest).
|
||||
|
||||
4. Sua equipe agora pode conversar em tempo real.
|
||||
- **General Assist**: abra o SurfSense a partir de qualquer aplicativo com um atalho global.
|
||||
- **Quick Assist**: selecione texto em qualquer lugar e peça à IA para explicar, reescrever ou agir sobre ele.
|
||||
- **Screenshot Assist**: capture qualquer região da tela e faça perguntas à IA sobre ela.
|
||||
- **Watch Local Folder** (monitorar pasta local): sincronize automaticamente uma pasta local com a sua base de conhecimento. Aponte para o seu cofre do Obsidian para manter suas notas pesquisáveis.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Chat em Tempo Real" /></p>
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
|
||||
|
||||
5. Adicione comentários para marcar colegas de equipe.
|
||||
**Sem dependência de fornecedor**
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Comentários em Tempo Real" /></p>
|
||||
- Mais de 100 LLMs via especificação da OpenAI e LiteLLM, incluindo GPT-5.5, Claude Sonnet 5 e Gemini 3.1 Pro.
|
||||
- Mais de 6.000 modelos de embedding e todos os principais rerankers.
|
||||
- Suporte completo a LLMs locais e privados (vLLM, Ollama), para que seus dados continuem sendo seus.
|
||||
|
||||
## Exemplo do Agente de Vídeo
|
||||
|
||||
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
|
||||
|
||||
## Exemplo do Agente de Podcast
|
||||
|
||||
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
|
||||
|
||||
## Como colaborar em tempo real (Beta)
|
||||
|
||||
1. Vá até a página Manage Members e crie um convite.
|
||||
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/40ed7683-5aa6-48a0-a3df-00575528c392" alt="Convidar membros" /></p>
|
||||
|
||||
2. Um colega entra e aquele workspace passa a ser compartilhado.
|
||||
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/ea4e1057-4d2b-4fd2-9ca0-cd19286a285e" alt="Fluxo de entrada por convite" /></p>
|
||||
|
||||
3. Torne um chat compartilhado e trabalhem nele juntos em tempo real, com comentários para marcar colegas.
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Comentários em tempo real" /></p>
|
||||
|
||||
## SurfSense vs Google NotebookLM
|
||||
|
||||
Ainda nos comparando como alternativa ao NotebookLM? Aqui está o comparativo honesto.
|
||||
|
||||
| Recurso | Google NotebookLM | SurfSense |
|
||||
|---------|-------------------|-----------|
|
||||
| **Fontes por Notebook** | 50 (Grátis) a 600 (Ultra, $249.99/mês) | Ilimitadas |
|
||||
| **Número de Notebooks** | 100 (Grátis) a 500 (planos pagos) | Ilimitados |
|
||||
| **Limite de Tamanho da Fonte** | 500.000 palavras / 200MB por fonte | Sem limite |
|
||||
| **Preços** | Nível gratuito disponível; Pro $19.99/mês, Ultra $249.99/mês | Gratuito e de código aberto, auto-hospedável na sua própria infra |
|
||||
| **Suporte a LLM** | Apenas Google Gemini | 100+ LLMs via OpenAI spec e LiteLLM |
|
||||
| **Modelos de Embeddings** | Apenas Google | 6.000+ modelos de embeddings, todos os principais rerankers |
|
||||
| **LLMs Locais / Privados** | Não disponível | Suporte completo (vLLM, Ollama) - seus dados ficam com você |
|
||||
| **Auto-Hospedável** | Não | Sim - Docker em um único comando ou Docker Compose completo |
|
||||
| **Código Aberto** | Não | Sim |
|
||||
| **Conectores Externos** | Google Drive, YouTube, sites | 27+ conectores - Mecanismos de busca, Google Drive, OneDrive, Dropbox, Slack, Teams, Jira, Notion, GitHub, Discord e [mais](#fontes-externas) |
|
||||
| **Suporte a Formatos de Arquivo** | PDFs, Docs, Slides, Sheets, CSV, Word, EPUB, imagens, URLs web, YouTube | 50+ formatos - documentos, imagens, vídeos via LlamaCloud, Unstructured ou Docling (local) |
|
||||
| **Busca** | Busca semântica | Busca Híbrida - Semântica + Texto completo com Índices Hierárquicos e Reciprocal Rank Fusion |
|
||||
| **Respostas com Citações** | Sim | Sim - Respostas citadas no estilo Perplexity |
|
||||
| **Arquitetura de Agentes** | Não | Sim - alimentado por [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) com planejamento, subagentes e acesso ao sistema de arquivos |
|
||||
| **Multiplayer em Tempo Real** | Notebooks compartilhados com papéis de Visualizador/Editor (sem chat em tempo real) | RBAC com papéis de Proprietário / Admin / Editor / Visualizador, chat em tempo real e threads de comentários |
|
||||
| **Geração de Vídeos** | Visões gerais cinemáticas via Veo 3 (apenas Ultra) | Disponível (NotebookLM é melhor aqui, melhorando ativamente) |
|
||||
| **Geração de Apresentações** | Slides mais bonitos mas não editáveis | Cria apresentações editáveis baseadas em slides |
|
||||
| **Geração de Podcasts** | Visões gerais em áudio com hosts e idiomas personalizáveis | Disponível com múltiplos provedores TTS (NotebookLM é melhor aqui, melhorando ativamente) |
|
||||
| **Automações e Agentes de IA** | Não | Fluxos de trabalho de IA agendados, gatilhos por eventos em novos documentos e automações sem código criadas por chat com escrita de volta no Notion, Slack, Linear e Jira |
|
||||
| **Aplicativo Desktop** | Não | Aplicativo nativo com General Assist, Quick Assist, Screenshot Assist e sincronização de pastas locais |
|
||||
| **Extensão de Navegador** | Não | Extensão multi-navegador para salvar qualquer página web, incluindo páginas protegidas por autenticação |
|
||||
|
||||
<details>
|
||||
<summary><b>Lista completa de Fontes Externas</b></summary>
|
||||
<a id="fontes-externas"></a>
|
||||
|
||||
Mecanismos de Busca (SearXNG, Tavily, LinkUp, Baidu Search) · Google Drive · OneDrive · Dropbox · Slack · Microsoft Teams · Linear · Jira · ClickUp · Confluence · BookStack · Notion · Gmail · Vídeos do YouTube · GitHub · Discord · Airtable · Google Calendar · Luma · Circleback · Elasticsearch · Obsidian, e mais por vir.
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
## SOLICITAÇÕES DE FUNCIONALIDADES E FUTURO
|
||||
| **Dados de mercado ao vivo para agentes** | Não | Conectores de Reddit, YouTube, Google Maps, Google Search e rastreamento web via API REST e MCP |
|
||||
| **Servidor MCP** | Não | Cada conector exposto como ferramenta nativa de agente, além de servidores MCP próprios com apps OAuth em um clique |
|
||||
| **Fontes por Notebook** | 50 (gratuito) a 600 (Ultra, US$ 249,99/mês) | Ilimitadas |
|
||||
| **Número de Notebooks** | 100 (gratuito) a 500 (planos pagos) | Ilimitado |
|
||||
| **Limite de tamanho por fonte** | 500.000 palavras / 200 MB por fonte | Sem limite |
|
||||
| **Preços** | Plano gratuito; Pro US$ 19,99/mês, Ultra US$ 249,99/mês | Gratuito e open source para auto-hospedar; a nuvem é paga por uso, com US$ 5 de crédito gratuito |
|
||||
| **Suporte a LLMs** | Apenas Google Gemini | Mais de 100 LLMs via especificação da OpenAI e LiteLLM |
|
||||
| **Modelos de embedding** | Apenas Google | Mais de 6.000 modelos de embedding, todos os principais rerankers |
|
||||
| **LLMs locais / privados** | Não disponível | Suporte completo (vLLM, Ollama), seus dados continuam sendo seus |
|
||||
| **Auto-hospedável** | Não | Sim, com uma linha de Docker ou Docker Compose completo |
|
||||
| **Open source** | Não | Sim |
|
||||
| **Fontes da base de conhecimento** | Google Drive, YouTube, sites | Upload de arquivos, Google Drive, OneDrive, Dropbox, sincronização de pasta local e páginas rastreadas |
|
||||
| **Formatos de arquivo suportados** | PDFs, Docs, Slides, Sheets, CSV, Word, EPUB, imagens, URLs, YouTube | Mais de 50 formatos: documentos, imagens, vídeos via LlamaCloud, Unstructured ou Docling (local) |
|
||||
| **Busca** | Busca semântica | Híbrida semântica + texto completo, com índices hierárquicos e fusão de rankings recíprocos |
|
||||
| **Respostas com citações** | Sim | Sim, respostas citadas no estilo Perplexity |
|
||||
| **Arquitetura agêntica** | Não | Sim, com [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview), incluindo planejamento, subagentes e acesso ao sistema de arquivos |
|
||||
| **Automações e agentes de IA** | Não | Fluxos agendados, gatilhos por eventos e automações sem código criadas via chat, com gravação no Notion, Slack, Linear e Jira |
|
||||
| **Multiplayer em tempo real** | Notebooks compartilhados com papéis Viewer/Editor (sem chat em tempo real) | RBAC com papéis Owner / Admin / Editor / Viewer, chat em tempo real e threads de comentários |
|
||||
| **Geração de vídeo** | Video Overviews cinematográficos via Veo 3 (apenas Ultra) | Disponível (o NotebookLM é melhor aqui, em melhoria contínua) |
|
||||
| **Geração de apresentações** | Slides mais bonitos, porém não editáveis | Apresentações editáveis baseadas em slides |
|
||||
| **Geração de podcasts** | Audio Overviews com apresentadores e idiomas personalizáveis | Disponível com vários provedores de TTS (o NotebookLM é melhor aqui, em melhoria contínua) |
|
||||
| **Aplicativo desktop** | Não | App nativo com General Assist, Quick Assist, Screenshot Assist e sincronização de pasta local |
|
||||
|
||||
## Pedidos de recursos e futuro
|
||||
|
||||
**O SurfSense está em desenvolvimento ativo.** Embora ainda não esteja pronto para produção, você pode nos ajudar a acelerar o processo.
|
||||
|
||||
Junte-se ao [Discord do SurfSense](https://discord.gg/ejRNvftDp9) e ajude a moldar o futuro do SurfSense!
|
||||
Entre no [Discord do SurfSense](https://discord.gg/ejRNvftDp9) e ajude a moldar o futuro do SurfSense!
|
||||
|
||||
## Roadmap
|
||||
|
||||
Fique atualizado com nosso progresso de desenvolvimento e próximas funcionalidades!
|
||||
Confira nosso roadmap público e contribua com suas ideias ou feedback:
|
||||
Fique por dentro do nosso progresso de desenvolvimento e dos próximos recursos. Confira nosso roadmap público e contribua com suas ideias ou feedback:
|
||||
|
||||
**Discussão do Roadmap:** [SurfSense 2026 Roadmap](https://github.com/MODSetter/SurfSense/discussions/565)
|
||||
**Discussão do roadmap:** [SurfSense 2026 Roadmap](https://github.com/MODSetter/SurfSense/discussions/565)
|
||||
|
||||
**Quadro Kanban:** [SurfSense Project Board](https://github.com/users/MODSetter/projects/3)
|
||||
|
||||
## Contribua
|
||||
|
||||
## Contribuir
|
||||
|
||||
Todas as contribuições são bem-vindas, desde estrelas e relatórios de bugs até melhorias no backend. Consulte [CONTRIBUTING.md](CONTRIBUTING.md) para começar.
|
||||
Todas as contribuições são bem-vindas, de estrelas e relatos de bugs a melhorias no backend. Veja o [CONTRIBUTING.md](CONTRIBUTING.md) para começar.
|
||||
|
||||
Obrigado a todos os nossos Surfers:
|
||||
|
||||
|
|
@ -310,13 +291,13 @@ Obrigado a todos os nossos Surfers:
|
|||
<img src="https://contrib.rocks/image?repo=MODSetter/SurfSense" />
|
||||
</a>
|
||||
|
||||
## Histórico de Stars
|
||||
## Histórico de estrelas
|
||||
|
||||
<a href="https://www.star-history.com/#MODSetter/SurfSense&Date">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=MODSetter/SurfSense&type=Date&theme=dark" />
|
||||
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=MODSetter/SurfSense&type=Date" />
|
||||
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=MODSetter/SurfSense&type=Date" />
|
||||
<img alt="Gráfico do histórico de estrelas" src="https://api.star-history.com/svg?repos=MODSetter/SurfSense&type=Date" />
|
||||
</picture>
|
||||
</a>
|
||||
|
||||
|
|
|
|||
385
README.zh-CN.md
385
README.zh-CN.md
|
|
@ -1,4 +1,4 @@
|
|||
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="readme_banner" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
|
||||
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="SurfSense，面向 AI 智能体的开源竞争情报平台" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
|
||||
|
||||
|
||||
|
||||
|
|
@ -20,291 +20,272 @@
|
|||
<a href="https://trendshift.io/repositories/13606" target="_blank"><img src="https://trendshift.io/api/badge/repositories/13606" alt="MODSetter%2FSurfSense | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
|
||||
</div>
|
||||
|
||||
# SurfSense
|
||||
# SurfSense：为你的 AI 智能体注入竞争情报能力
|
||||
|
||||
NotebookLM 是目前最好、最实用的 AI 平台之一，但当你开始经常使用它时，你也会感受到它的局限性，总觉得还有不足之处。
|
||||
SurfSense 是**面向 AI 智能体的开源竞争情报平台**。你的智能体可以通过一个 **REST API** 或 **MCP 服务器**，利用来自 **Reddit、YouTube、Google Maps、Google Search 和开放网络**的实时数据，监控竞争对手、追踪排名、倾听市场动态。定时和事件触发的智能体会把发现的内容转化为简报和预警，内置的知识库则让每一条发现都可搜索、可引用。
|
||||
|
||||
1. 一个笔记本中可以添加的来源数量有限制。
|
||||
2. 可以拥有的笔记本数量有限制。
|
||||
3. 来源不能超过 500,000 个单词和 200MB。
|
||||
4. 你被锁定在 Google 服务中（LLM、使用模型等），没有配置选项。
|
||||
5. 有限的外部数据源和服务集成。
|
||||
6. NotebookLM 代理专门针对学习和研究进行了优化，但你可以用源数据做更多事情。
|
||||
7. 缺乏多人协作支持。
|
||||
> [!NOTE]
|
||||
> **📢 致我们的 NotebookLM 替代品用户**
|
||||
>
|
||||
> 在过去几个月里，我们把 SurfSense 打造成了针对个人知识的最佳通用研究智能体，这段旅程为我们赢得了一个令我们由衷自豪的社区。如今，Claude、OpenCode、Hermes、OpenClaw 等智能体工具已经证明智能体就是未来，通用研究正在成为每个有能力的智能体开箱即用的功能。而智能体仍然缺少的是**实时市场数据以及围绕它的工作流**，所以这正是我们全力投入的方向：成为标杆级的开源竞争情报智能体平台。
|
||||
>
|
||||
> **你所依赖的一切功能都不会消失。**你的知识库、带引用的对话、报告、播客、演示文稿、自动化以及协作聊天都会继续可用，自托管也依然免费且开源。完整公告请阅读[我们的更新日志](https://www.surfsense.com/changelog)。
|
||||
|
||||
...还有更多。
|
||||
## 目录
|
||||
|
||||
**SurfSense 正是为了解决这些问题而生。** SurfSense 赋予你：
|
||||
- [为什么智能体需要 SurfSense](#为什么智能体需要-surfsense)
|
||||
- [你可以用 SurfSense 做什么？](#你可以用-surfsense-做什么)
|
||||
- [实时数据连接器](#实时数据连接器)
|
||||
- [快速开始](#快速开始)
|
||||
- [开箱即用的其他能力](#开箱即用的其他能力)
|
||||
- [SurfSense 对比 Google NotebookLM](#surfsense-对比-google-notebooklm)
|
||||
- [路线图](#路线图)
|
||||
- [参与贡献](#参与贡献)
|
||||
|
||||
- **控制你的数据流** - 保持数据私密和安全。
|
||||
- **无数据限制** - 添加无限数量的来源和笔记本。
|
||||
- **无供应商锁定** - 配置任何 LLM、图像、TTS 和 STT 模型。
|
||||
- **25+ 外部数据源** - 从 Google Drive、OneDrive、Dropbox、Notion 和许多其他外部服务添加你的来源。
|
||||
- **实时多人协作支持** - 在共享笔记本中轻松与团队成员协作。
|
||||
- **AI 自动化与智能体** - 按计划运行 AI 智能体，或在文档进入文件夹的那一刻触发它们，然后将结果回写到 Notion、Slack、Linear 和 Drive。只需在聊天中描述即可创建无代码自动化。
|
||||
- **桌面应用** - 通过 Quick Assist、General Assist、Screenshot Assist 和本地文件夹同步在任何应用程序中获得 AI 助手。
|
||||
## 为什么智能体需要 SurfSense
|
||||
|
||||
...更多功能即将推出。
|
||||
问任何一个有能力的智能体"竞争对手这周的定价是多少？"或者"自发布以来 Reddit 上对我们的评价如何？"，它都找不到可信赖的数据来源。官方平台 API 要么有速率限制，要么按企业级定价，要么根本不存在，而自建抓取管线又非常脆弱。SurfSense 正是为填补这一空白而生：
|
||||
|
||||
- **平台原生连接器**，每个都是返回结构化 JSON 的强类型 REST 端点。不用赌速率限制，也不用解析 HTML。
|
||||
- **一个 MCP 服务器**，把每个连接器都作为原生工具（`surfsense_reddit_scrape`、`surfsense_google_search` 等）暴露给 Claude、Cursor 或任何智能体框架。
|
||||
- **一套智能体运行框架**，而不只是原始数据：重试、结构化输出和额度计量都已内置，智能体可以从一个问题直达一份简报，无需你自己搭建管线。
|
||||
- **开源且可自托管**，你的竞争研究数据始终留在你自己的基础设施上。
|
||||
|
||||
## 你可以用 SurfSense 做什么？
|
||||
|
||||
## 视频代理示例
|
||||
下面的每个用例都是 SurfSense 智能体如今能够端到端完成的真实任务，一条提示语或一个定时计划即可触发。
|
||||
|
||||
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
|
||||
### 多连接器工作流
|
||||
|
||||
在一次智能体运行中串联多个连接器，最终得到一份带引用的简报。
|
||||
|
||||
- **发布影响力分析，覆盖所有平台** — "我们的竞争对手昨天发布了 v2。评估搜索、Reddit 和 YouTube 上的反应。"智能体会抓取搜索结果页、Reddit 帖子和 YouTube 评论，然后把三路信号汇总成一份发布影响力简报。
|
||||
- **本地竞争对手拆解** — Google Maps 找到本地玩家，网页爬虫读取他们的定价页面，Google Search 展示谁在搜索中胜出，全部在一次运行中完成。
|
||||
- **定时执行的竞争对手 360 全景** — 一条自动化每周串联四个连接器：网站变更、排名变动、Reddit 舆情和 YouTube 反应。
|
||||
|
||||
## 播客代理示例
|
||||
### 竞争对手监控
|
||||
|
||||
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
|
||||
- **定价监控** — 从竞争对手的定价页面提取每个套餐、价格和限制，汇总到一张表格，然后每天复查，任何变动即时预警。
|
||||
- **产品与更新日志追踪** — 每周一爬取对手的更新日志、产品页和招聘页面，收到一份他们发布了什么的简报。
|
||||
- **排名与广告监控** — 追踪你的市场真正看到的 Google 排名、付费广告和 AI Overview 引用，并逐日标记变动。
|
||||
|
||||
### B2B 潜在客户开发
|
||||
|
||||
## 如何使用 SurfSense
|
||||
- **本地商户线索** — 把一个品类加一个地区（"圣何塞的汉堡店"）变成一份线索清单，包含电话、网站、评分，以及从他们网站上提取的决策人联系方式。
|
||||
- **团队名录与联系方式** — 爬取任意公司网站，提取完整团队名单，附带邮箱、社交账号和数据来源，导出为 CSV。
|
||||
- **投资组合与市场地图** — 绘制一家投资机构的投资组合或整个品类的全景图，再为每家公司补充定价和联系方式。
|
||||
|
||||
### Cloud
|
||||
### 品牌与市场舆情监听
|
||||
|
||||
1. 访问 [surfsense.com](https://www.surfsense.com) 并登录。
|
||||
- **Reddit 品牌监控** — 在买家坦诚发言的帖子里，倾听你的市场对你、你的竞争对手和你所在品类的真实评价。
|
||||
- **YouTube 观众情绪分析** — 大规模拉取视频、字幕转录和评论，然后聚类分析观众在称赞什么、抱怨什么。
|
||||
- **换用意向挖掘** — 找到正在积极寻找某竞争对手替代品的人群，并按他们的换用意愿排序。
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/LoginFlowGif.gif" alt="登录" /></p>
|
||||
### 市场调研
|
||||
|
||||
2. 连接您的连接器并同步。启用定期同步以保持连接器数据更新。
|
||||
- **实时网络深度研究** — 智能体针对一个问题爬取数十个实时来源，并综合出一份带引用的答案，而不是过时的索引。
|
||||
- **AI Overview 与 GEO 追踪** — 捕捉 Google 的 AI Overviews 何时回答了你市场的搜索，以及它们究竟引用了哪些来源。
|
||||
- **带引用的简报与预警** — 智能体收集到的一切都会以简报和预警的形式汇入你的工作区，每条结论都附带可核查的来源。
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="连接器" /></p>
|
||||
### 把以上任何任务自动化，无需代码
|
||||
|
||||
3. 在连接器数据索引期间，上传文档。
|
||||
自动化功能可以按计划或响应事件运行完整的智能体回合，然后把结果写回 Notion、Slack、Linear 和 Jira。用自然语言描述工作流，SurfSense 就会替你构建。可以试试这些提示语：
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="上传文档" /></p>
|
||||
- "盯住我们前 3 名竞争对手的定价页面，一旦套餐或价格变动就在 Slack 上提醒我。"
|
||||
- "追踪 Reddit 和 YouTube 上对我们品牌的每一次提及，每天给我发一份摘要。"
|
||||
- "监控我们前 10 个关键词的 Google 排名，按周标记下滑情况。"
|
||||
- "每周一拉取我们门店以及竞争对手门店的最新 Google Maps 评论。"
|
||||
- "每月生成一份竞争对手分析报告并保存到我的工作区。"
|
||||
|
||||
4. 一切索引完成后，尽管提问（使用场景）：
|
||||
## 实时数据连接器
|
||||
|
||||
**桌面应用**（在以下所有功能之外的原生附加功能，并非独立的功能集）
|
||||
| 连接器 | 你的智能体能获得什么 | 了解更多 |
|
||||
|---|---|---|
|
||||
| **Reddit** | 帖子、评论和子版块信息流，不受官方 API 速率限制 | [Reddit Scraper API](https://www.surfsense.com/reddit) |
|
||||
| **YouTube** | 视频、字幕转录和评论串，用于品牌和产品舆情监听 | [YouTube Scraper API](https://www.surfsense.com/youtube) |
|
||||
| **Google Maps** | 地点、评分和评论，用于本地竞争对手和潜在客户调研 | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
|
||||
| **Google Search** | 实时搜索结果页，用于排名追踪和市场监控 | [Google Search API](https://www.surfsense.com/google-search) |
|
||||
| **Web Crawl** | 把开放网络上的任意页面转为干净、结构化的内容 | [Web Crawling API](https://www.surfsense.com/web-crawl) |
|
||||
| **外部 MCP 连接器** | 将任意 MCP 服务器接入你的智能体，Notion、Slack、Jira 等支持一键 OAuth | [External MCP Connectors](https://www.surfsense.com/external-mcp-connectors) |
|
||||
|
||||
- General Assist：通过全局快捷键，从任意应用中即刻打开 SurfSense。
|
||||
计费采用按量付费：连接器按实际返回的条目计费，爬取按成功抓取的页面计费，失败的调用永不计费。自托管部署默认关闭计费。详见[定价](https://www.surfsense.com/pricing)。
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/general_assist.gif" alt="General Assist" /></p>
|
||||
## 快速开始
|
||||
|
||||
- Quick Assist：在任意位置选中文本，让 AI 解释、改写或对其执行操作。
|
||||
### 在代码中调用连接器
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
|
||||
每个连接器都是一个 REST 端点，你可以用任何语言、凭借你的 SurfSense API 密钥来调用：
|
||||
|
||||
- Screenshot Assist：截取屏幕上任意区域，并就其中内容向 AI 提问。
|
||||
```bash
|
||||
curl -X POST "$SURFSENSE_API_URL/workspaces/$WORKSPACE_ID/scrapers/reddit/scrape" \
|
||||
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"search_queries": ["your brand"],
|
||||
"community": "webscraping",
|
||||
"sort": "top",
|
||||
"time_filter": "week"
|
||||
}'
|
||||
```
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/screenshot_assist.gif" alt="Screenshot Assist" /></p>
|
||||
每个[连接器页面](https://www.surfsense.com/connectors)都提供 Python、JavaScript、Go、PHP、Ruby、Java 和 C# 的可直接复制粘贴的示例。
|
||||
|
||||
- Watch Local Folder：将本地文件夹自动同步到你的知识库。非常适合 Obsidian 库。
|
||||
### 通过 MCP 把工具交给你的智能体
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/folder_watch.gif" alt="Watch Local Folder" /></p>
|
||||
把 SurfSense MCP 服务器添加到 Claude、Cursor 或你自己的智能体框架：
|
||||
|
||||
**成果工作室**
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"surfsense": {
|
||||
"url": "https://mcp.surfsense.com",
|
||||
"headers": { "Authorization": "Bearer ${SURFSENSE_API_KEY}" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- AI Report Generator：生成带引用的研究报告，并导出为 PDF、DOCX、HTML、LaTeX、EPUB、ODT 或纯文本。
|
||||
现在，你的智能体就可以把每个连接器当作原生工具来调用。完整工具列表请查看 [SurfSense MCP 服务器](https://www.surfsense.com/mcp-server) 页面，也可以通过 [`surfsense_mcp`](./surfsense_mcp) 在本地运行该服务器。
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="AI Report Generator" /></p>
|
||||
### 使用云端服务
|
||||
|
||||
- AI Podcast Generator：在 20 秒内将任意文档或文件夹转换为双主持人 AI 播客。
|
||||
访问 [surfsense.com](https://www.surfsense.com)，登录后用自然语言向智能体索取实时市场数据。新账户自带 5 美元免费额度，无需订阅。
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/PodcastGenGif.gif" alt="AI Podcast Generator" /></p>
|
||||
|
||||
- AI Presentation & Video Maker：根据你的资料创建可编辑的幻灯片和带旁白的视频概览。
|
||||
### 免费自托管
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/video_gen_gif.gif" alt="AI Presentation and Video Maker" /></p>
|
||||
在你自己的基础设施上运行整个平台，包括连接器、智能体、自动化和 MCP 服务器。自托管部署默认关闭计费，抓取、爬取和智能体运行只受你的硬件和你自带的模型密钥限制。
|
||||
|
||||
- AI Image Generator：直接从你的聊天和文档生成高质量图像。
|
||||
**前置条件：**必须已安装并运行 [Docker Desktop](https://www.docker.com/products/docker-desktop/)。
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ImageGenGif.gif" alt="AI Image Generator" /></p>
|
||||
|
||||
- AI Resume Builder：根据任意职位描述定制你现有的简历，顺利通过 ATS。
|
||||
可以试试这样的提示：
|
||||
|
||||
- “根据这份职位描述定制我的简历，让它通过 ATS 并赢得面试。”
|
||||
- “匹配这份招聘启事中的关键词，为 ATS 优化我的简历。”
|
||||
- “重写我的简历要点，突出这个岗位所需要的技能。”
|
||||
- “将我的简历与这份职位描述对比，列出需要改进的差距。”
|
||||
- “根据我的简历和这份职位描述，写一封相匹配的求职信。”
|
||||
|
||||
**搜索与聊天**
|
||||
|
||||
- Chat With Your PDFs & Docs：跨所有文件提问，并获得带内联引用的答案。
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="Chat With Your PDFs and Docs" /></p>
|
||||
|
||||
- AI Search With Citations：在整个知识库中进行语义与关键词的混合搜索。
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BSNCGif.gif" alt="AI Search With Citations" /></p>
|
||||
|
||||
- Collaborative AI Chat：与团队实时协作处理 AI 对话。
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Collaborative AI Chat" /></p>
|
||||
|
||||
- Comments & Mentions：在任意 AI 消息上评论并 @ 你的队友。
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Comments and Mentions" /></p>
|
||||
|
||||
**连接器与集成**
|
||||
|
||||
- Connect & Sync Your Tools：将 Notion、Slack、Google Drive、Gmail、GitHub、Linear 等 25+ 数据源同步为一个可搜索的语料库。
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="Connect and Sync Your Tools" /></p>
|
||||
|
||||
- Chat With Uploaded Files：上传 PDF、Office 文档、图像和音频。即刻可搜索。
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="Chat With Uploaded Files" /></p>
|
||||
|
||||
- Connector Write-Back：让智能体将结果回写到 Notion、Slack、Linear 和 Drive。
|
||||
可以试试这样的提示：
|
||||
|
||||
- “把这份研究摘要发布到我的 Notion 工作区。”
|
||||
- “把这些会议行动项发送到我们的团队 Slack 频道。”
|
||||
- “根据这份缺陷报告创建一个 Jira 工单。”
|
||||
- “根据这个功能需求在 Linear 中创建一个 issue。”
|
||||
- “把这份生成的报告作为文档保存到 Google Drive。”
|
||||
|
||||
- Obsidian & Knowledge Base Sync：让你的 Obsidian 库与个人知识库保持同步。
|
||||
|
||||
**自动化**
|
||||
|
||||
- Scheduled AI Workflows：按计划运行智能体：每日简报、每周摘要、周期性报告。
|
||||
可以试试这样的提示：
|
||||
|
||||
- “每天早上把我知识库中新增文档的每日简报发邮件给我。”
|
||||
- “每周五根据我的 Slack 和 Gmail 生成一份每周状态报告。”
|
||||
- “每月运行一次竞争对手分析报告并保存到我的工作区。”
|
||||
- “把我的 GitHub 和 Linear 活动汇总成一份每日站会更新。”
|
||||
- “针对我关注的主题创建一份周期性的每周研究报告。”
|
||||
|
||||
- Event-Triggered Automations：在文档进入文件夹的那一刻触发智能体，并将结果发布到你的工具中。
|
||||
可以试试这样的提示：
|
||||
|
||||
- “当一个 PDF 进入我的 Research 文件夹时，生成一份带引用的 AI 摘要。”
|
||||
- “当新增会议记录时，把它整理成带行动项的会议纪要。”
|
||||
- “当上传发票时，把供应商、总额和到期日提取到一张表格中。”
|
||||
- “当一份合同进入我的 Legal 文件夹时，标记关键条款和续约日期。”
|
||||
- “当一份简历加入 Candidates 时，根据职位描述对其进行筛选。”
|
||||
|
||||
- Chat-Built Automations：用通俗的语言描述一个自动化，SurfSense 就会为你构建它。
|
||||
可以试试这样的提示：
|
||||
|
||||
- “创建一个 AI 智能体，每天早上把新增 Notion 页面的摘要发邮件给我。”
|
||||
- “创建一个无代码自动化，每周把研究摘要发布到 Slack。”
|
||||
- “设置一个 AI 笔记助手，把新增会议记录整理成纪要。”
|
||||
- “创建一个工作流，从会议记录中提取行动项并指派负责人。”
|
||||
- “自动化一份来自我的 Gmail 和 Google Drive 的每日邮件简报。”
|
||||
|
||||
|
||||
### 自托管
|
||||
|
||||
在您自己的基础设施上运行 SurfSense，实现完全的数据控制和隐私保护。
|
||||
|
||||
**前置条件：** 需要安装并运行 [Docker Desktop](https://www.docker.com/products/docker-desktop/)。
|
||||
|
||||
#### Linux/MacOS 用户：
|
||||
Linux/macOS：
|
||||
|
||||
```bash
|
||||
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash
|
||||
```
|
||||
|
||||
#### Windows 用户：
|
||||
Windows：
|
||||
|
||||
```powershell
|
||||
```bash
|
||||
irm https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.ps1 | iex
|
||||
```
|
||||
|
||||
安装脚本会自动配置 [Watchtower](https://github.com/nicholas-fedor/watchtower) 以实现每日自动更新。如需跳过，请添加 `--no-watchtower` 参数。
|
||||
安装脚本会自动配置 [Watchtower](https://github.com/nicholas-fedor/watchtower) 以实现每日自动更新。如需跳过，请加上 `--no-watchtower` 参数。关于 Docker Compose、手动安装及其他部署方式，请参阅[文档](https://www.surfsense.com/docs/)。
|
||||
|
||||
如需 Docker Compose、手动安装及其他部署方式，请查看[文档](https://www.surfsense.com/docs/)。
|
||||
## 开箱即用的其他能力
|
||||
|
||||
### 桌面应用
|
||||
让 SurfSense 成为领先的开源 NotebookLM 替代品的那套研究工作区依然都在，而且你的智能体收集到的一切都会汇入其中。
|
||||
|
||||
SurfSense 还提供桌面应用，将 AI 助手带到您计算机上的每个应用程序中。从[最新版本](https://github.com/MODSetter/SurfSense/releases/latest)下载。
|
||||
**知识库**
|
||||
|
||||
桌面应用包含以下强大功能：
|
||||
- 上传 PDF、Office 文档、图片和音频，或同步 **Google Drive、OneDrive 和 Dropbox**。支持 50 多种文件格式。
|
||||
- 混合语义与全文搜索，提供 Perplexity 风格的带引用回答。
|
||||
- AI 文件整理功能按来源、日期和主题自动归类文档。
|
||||
|
||||
- **General Assist** — 通过全局快捷键从任何应用程序即时启动 SurfSense。
|
||||
- **Quick Assist** — 在任何位置选中文本，然后让 AI 解释、改写或对其执行操作。
|
||||
- **Screenshot Assist** — 在屏幕上框选区域并附加到聊天，让回复基于您的知识库。
|
||||
- **Watch Local Folder** — 监视本地文件夹，自动将文件更改同步到您的知识库。**Pro tip：** 将其指向您的 Obsidian vault，让笔记在 SurfSense 中随时可搜索。
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="与你的 PDF 和文档对话" /></p>
|
||||
|
||||
所有功能均基于您选择的搜索空间运行，确保回答始终以您自己的数据为依据。
|
||||
**成果工作室**
|
||||
|
||||
### 如何实时协作（Beta）
|
||||
- AI 报告生成器，可导出为 PDF、DOCX、HTML、LaTeX、EPUB、ODT 或纯文本。
|
||||
- 20 秒内基于任意文档或文件夹生成双主持人 AI 播客。
|
||||
- 可编辑的幻灯片、带旁白的视频概览以及 AI 图像生成。
|
||||
|
||||
1. 前往成员管理页面并创建邀请。
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="AI 报告生成器" /></p>
|
||||
|
||||
**团队协作**
|
||||
|
||||
- 支持评论和提及的实时协作 AI 聊天。
|
||||
- 基于角色的访问控制（RBAC），提供所有者、管理员、编辑者和查看者角色。
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="协作 AI 聊天" /></p>
|
||||
|
||||
**桌面应用**
|
||||
|
||||
在电脑上的每个应用中获得原生 AI 辅助。从[最新版本](https://github.com/MODSetter/SurfSense/releases/latest)下载。
|
||||
|
||||
- **General Assist**：通过全局快捷键在任意应用中唤起 SurfSense。
|
||||
- **Quick Assist**：在任意位置选中文本，让 AI 解释、改写或据此执行操作。
|
||||
- **Screenshot Assist**：截取屏幕上任意区域，向 AI 提问。
|
||||
- **Watch Local Folder**：把本地文件夹自动同步到知识库。将它指向你的 Obsidian 仓库，让笔记随时可搜索。
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
|
||||
|
||||
**无供应商锁定**
|
||||
|
||||
- 通过 OpenAI 规范和 LiteLLM 支持 100 多种 LLM，包括 GPT-5.5、Claude Sonnet 5 和 Gemini 3.1 Pro。
|
||||
- 支持 6,000 多种嵌入模型和所有主流重排序器。
|
||||
- 完整支持本地和私有 LLM（vLLM、Ollama），你的数据始终属于你。
|
||||
|
||||
## 视频智能体示例
|
||||
|
||||
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
|
||||
|
||||
## 播客智能体示例
|
||||
|
||||
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
|
||||
|
||||
## 如何进行实时协作（Beta）
|
||||
|
||||
1. 进入成员管理页面并创建邀请。
|
||||
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/40ed7683-5aa6-48a0-a3df-00575528c392" alt="邀请成员" /></p>
|
||||
|
||||
2. 队友加入后，该 SearchSpace 变为共享。
|
||||
2. 队友加入后，该工作区即变为共享。
|
||||
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/ea4e1057-4d2b-4fd2-9ca0-cd19286a285e" alt="邀请加入流程" /></p>
|
||||
|
||||
3. 将聊天设为共享。
|
||||
|
||||
<p align="center"><img src="https://github.com/user-attachments/assets/17b93904-0888-4c3a-ac12-51a24a8ea26a" alt="设为共享聊天" /></p>
|
||||
|
||||
4. 您的团队现在可以实时聊天。
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="实时聊天" /></p>
|
||||
|
||||
5. 添加评论以标记队友。
|
||||
3. 将聊天设为共享，即可与团队实时协作，并通过评论标记队友。
|
||||
|
||||
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="实时评论" /></p>
|
||||
|
||||
## SurfSense vs Google NotebookLM
|
||||
## SurfSense 对比 Google NotebookLM
|
||||
|
||||
还在把我们当作 NotebookLM 替代品来比较？这里是坦诚的对比。
|
||||
|
||||
| 功能 | Google NotebookLM | SurfSense |
|
||||
|---------|-------------------|-----------|
|
||||
| **每个笔记本的来源数** | 50（免费）到 600（Ultra，$249.99/月） | 无限制 |
|
||||
| **笔记本数量** | 100（免费）到 500（付费方案） | 无限制 |
|
||||
| **来源大小限制** | 500,000 词 / 200MB 每个来源 | 无限制 |
|
||||
| **定价** | 免费版可用；Pro $19.99/月，Ultra $249.99/月 | 免费开源，在自己的基础设施上自托管 |
|
||||
| **LLM 支持** | 仅 Google Gemini | 100+ LLM，通过 OpenAI spec 和 LiteLLM |
|
||||
| **嵌入模型** | 仅 Google | 6,000+ 嵌入模型，所有主流重排序器 |
|
||||
| **本地 / 私有 LLM** | 不可用 | 完整支持（vLLM、Ollama）- 您的数据由您掌控 |
|
||||
| **可自托管** | 否 | 是 - Docker 一行命令或完整 Docker Compose |
|
||||
| **面向智能体的实时市场数据** | 无 | 通过 REST API 和 MCP 提供 Reddit、YouTube、Google Maps、Google Search 和网页爬取连接器 |
|
||||
| **MCP 服务器** | 无 | 每个连接器都作为原生智能体工具暴露，还可自带 MCP 服务器并使用一键 OAuth 应用 |
|
||||
| **每个笔记本的来源数** | 50 个（免费版）至 600 个（Ultra 版，249.99 美元/月） | 无限制 |
|
||||
| **笔记本数量** | 100 个（免费版）至 500 个（付费档位） | 无限制 |
|
||||
| **来源大小限制** | 每个来源 50 万字 / 200MB | 无限制 |
|
||||
| **定价** | 免费档；Pro 19.99 美元/月，Ultra 249.99 美元/月 | 自托管免费且开源；云端按量付费，附赠 5 美元免费额度 |
|
||||
| **LLM 支持** | 仅 Google Gemini | 通过 OpenAI 规范和 LiteLLM 支持 100 多种 LLM |
|
||||
| **嵌入模型** | 仅 Google | 6,000 多种嵌入模型，所有主流重排序器 |
|
||||
| **本地 / 私有 LLM** | 不支持 | 完整支持（vLLM、Ollama），你的数据始终属于你 |
|
||||
| **可自托管** | 否 | 是，Docker 一行命令或完整 Docker Compose |
|
||||
| **开源** | 否 | 是 |
|
||||
| **外部连接器** | Google Drive、YouTube、网站 | 27+ 连接器 - 搜索引擎、Google Drive、OneDrive、Dropbox、Slack、Teams、Jira、Notion、GitHub、Discord 等[更多](#外部数据源) |
|
||||
| **文件格式支持** | PDF、Docs、Slides、Sheets、CSV、Word、EPUB、图像、网页 URL、YouTube | 50+ 格式 - 文档、图像、视频，通过 LlamaCloud、Unstructured 或 Docling（本地） |
|
||||
| **搜索** | 语义搜索 | 混合搜索 - 语义 + 全文搜索，结合层次化索引和倒数排名融合 |
|
||||
| **引用回答** | 是 | 是 - Perplexity 风格的引用回答 |
|
||||
| **代理架构** | 否 | 是 - 基于 [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) 构建，支持规划、子代理和文件系统访问 |
|
||||
| **实时多人协作** | 共享笔记本，支持查看者/编辑者角色（无实时聊天） | RBAC 角色控制（所有者/管理员/编辑者/查看者），实时聊天和评论线程 |
|
||||
| **视频生成** | 通过 Veo 3 的电影级视频概览（仅 Ultra） | 可用（NotebookLM 在此方面更好，正在积极改进） |
|
||||
| **演示文稿生成** | 更美观的幻灯片但不可编辑 | 创建可编辑的幻灯片式演示文稿 |
|
||||
| **播客生成** | 可自定义主持人和语言的音频概览 | 可用，支持多种 TTS 提供商（NotebookLM 在此方面更好，正在积极改进） |
|
||||
| **AI 自动化与智能体** | 否 | 定时 AI 工作流、新文档的事件触发，以及通过聊天构建的无代码自动化，支持回写到 Notion、Slack、Linear 和 Jira |
|
||||
| **桌面应用** | 否 | 原生应用，包含 General Assist、Quick Assist、Screenshot Assist 和本地文件夹同步 |
|
||||
| **浏览器扩展** | 否 | 跨浏览器扩展，保存任何网页，包括需要身份验证的页面 |
|
||||
|
||||
<details>
|
||||
<summary><b>外部数据源完整列表</b></summary>
|
||||
<a id="外部数据源"></a>
|
||||
|
||||
搜索引擎（SearXNG、Tavily、LinkUp、Baidu Search）· Google Drive · OneDrive · Dropbox · Slack · Microsoft Teams · Linear · Jira · ClickUp · Confluence · BookStack · Notion · Gmail · YouTube 视频 · GitHub · Discord · Airtable · Google Calendar · Luma · Circleback · Elasticsearch · Obsidian，更多即将推出。
|
||||
|
||||
</details>
|
||||
|
||||
| **知识库来源** | Google Drive、YouTube、网站 | 文件上传、Google Drive、OneDrive、Dropbox、本地文件夹同步以及爬取的网页 |
|
||||
| **文件格式支持** | PDF、Docs、Slides、Sheets、CSV、Word、EPUB、图片、网页 URL、YouTube | 50 多种格式：文档、图片、视频，通过 LlamaCloud、Unstructured 或 Docling（本地）解析 |
|
||||
| **搜索** | 语义搜索 | 混合语义 + 全文搜索，带分层索引和倒数排名融合 |
|
||||
| **带引用的回答** | 有 | 有，Perplexity 风格的引用回答 |
|
||||
| **智能体架构** | 无 | 有，由 [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) 驱动，具备规划、子智能体和文件系统访问能力 |
|
||||
| **AI 自动化与智能体** | 无 | 定时工作流、事件触发以及通过聊天构建的无代码自动化，可写回 Notion、Slack、Linear 和 Jira |
|
||||
| **实时多人协作** | 共享笔记本，仅有查看者/编辑者角色（无实时聊天） | RBAC 提供所有者 / 管理员 / 编辑者 / 查看者角色，支持实时聊天和评论串 |
|
||||
| **视频生成** | 通过 Veo 3 生成电影级视频概览（仅 Ultra 版） | 已提供（此项 NotebookLM 更强，我们正在积极改进） |
|
||||
| **演示文稿生成** | 幻灯片更美观但不可编辑 | 可编辑的幻灯片式演示文稿 |
|
||||
| **播客生成** | 音频概览，支持自定义主持人和语言 | 已提供，支持多种 TTS 服务商（此项 NotebookLM 更强，我们正在积极改进） |
|
||||
| **桌面应用** | 无 | 原生应用，包含 General Assist、Quick Assist、Screenshot Assist 和本地文件夹同步 |
|
||||
|
||||
## 功能请求与未来规划
|
||||
|
||||
**SurfSense 正在积极开发中。**虽然它尚未达到生产就绪状态，但你可以帮助我们加快进度。
|
||||
|
||||
**SurfSense 正在积极开发中。** 虽然它还未达到生产就绪状态，但您可以帮助我们加快进度。
|
||||
|
||||
加入 [SurfSense Discord](https://discord.gg/ejRNvftDp9) 一起塑造 SurfSense 的未来！
|
||||
加入 [SurfSense Discord](https://discord.gg/ejRNvftDp9)，一起塑造 SurfSense 的未来！
|
||||
|
||||
## 路线图
|
||||
|
||||
随时了解我们的开发进度和即将推出的功能！
|
||||
查看我们的公开路线图并贡献您的想法或反馈：
|
||||
随时了解我们的开发进度和即将推出的功能。查看我们的公开路线图，贡献你的想法或反馈：
|
||||
|
||||
**路线图讨论：** [SurfSense 2026 Roadmap](https://github.com/MODSetter/SurfSense/discussions/565)
|
||||
**路线图讨论：**[SurfSense 2026 Roadmap](https://github.com/MODSetter/SurfSense/discussions/565)
|
||||
|
||||
**看板：** [SurfSense Project Board](https://github.com/users/MODSetter/projects/3)
|
||||
**看板：**[SurfSense Project Board](https://github.com/users/MODSetter/projects/3)
|
||||
|
||||
## 参与贡献
|
||||
|
||||
## 贡献
|
||||
欢迎一切形式的贡献，从点星标、报告缺陷到后端改进。请参阅 [CONTRIBUTING.md](CONTRIBUTING.md) 开始参与。
|
||||
|
||||
欢迎所有贡献，从 Star 和 Bug 报告到后端改进。请参阅 [CONTRIBUTING.md](CONTRIBUTING.md) 开始贡献。
|
||||
|
||||
感谢所有 Surfers:
|
||||
感谢所有 Surfer：
|
||||
|
||||
<a href="https://github.com/MODSetter/SurfSense/graphs/contributors">
|
||||
<img src="https://contrib.rocks/image?repo=MODSetter/SurfSense" />
|
||||
|
|
|
|||
2
VERSION
2
VERSION
|
|
@ -1 +1 @@
|
|||
0.0.30
|
||||
0.0.31
|
||||
|
|
|
|||
|
|
@ -333,16 +333,6 @@ STT_SERVICE=local/base
|
|||
# GATEWAY_DISCORD_ENABLED=FALSE
|
||||
# GATEWAY_DISCORD_REDIRECT_URI=http://localhost:3929/api/v1/gateway/discord/callback
|
||||
|
||||
# ------------------------------------------------------------------------------
|
||||
# SearXNG (bundled web search, works out of the box with no config needed)
|
||||
# ------------------------------------------------------------------------------
|
||||
# SearXNG provides web search to all search spaces automatically.
|
||||
# To access the SearXNG UI directly in dev/deps-only compose: http://localhost:8888
|
||||
# To disable the service entirely: docker compose up --scale searxng=0
|
||||
# To point at your own SearXNG instance instead of the bundled one:
|
||||
# SEARXNG_DEFAULT_HOST=http://your-searxng:8080
|
||||
# SEARXNG_SECRET=surfsense-searxng-secret
|
||||
|
||||
# ------------------------------------------------------------------------------
|
||||
# Daytona Sandbox (optional cloud code execution for the deep agent)
|
||||
# ------------------------------------------------------------------------------
|
||||
|
|
@ -435,6 +425,31 @@ SURFSENSE_ENABLE_DOOM_LOOP=true
|
|||
# ETL_CREDIT_BILLING_ENABLED=FALSE
|
||||
# MICROS_PER_PAGE=1000
|
||||
|
||||
# Debit the credit wallet per *successful* web crawl. Default FALSE keeps
|
||||
# crawling effectively free for self-hosted installs. Price is config-driven:
|
||||
# WEB_CRAWL_MICROS_PER_SUCCESS = round(USD_per_1000_crawls * 1_000)
|
||||
# 2000 == $2/1000 (default) | 1000 == $1/1000. Captcha solves bill as a
|
||||
# separate per-attempt unit (independent flag).
|
||||
# WEB_CRAWL_CREDIT_BILLING_ENABLED=FALSE
|
||||
# WEB_CRAWL_MICROS_PER_SUCCESS=2000
|
||||
# WEB_CRAWL_CAPTCHA_BILLING_ENABLED=FALSE
|
||||
# WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE=3000
|
||||
|
||||
# Debit the credit wallet per *item returned* by the platform-native scrapers
|
||||
# (Reddit, Google Search, Google Maps, YouTube). Default FALSE keeps scraping
|
||||
# effectively free for self-hosted installs. Each rate is micro-USD per item,
|
||||
# config-driven: <KEY> = round(USD_per_1000_items * 1_000). Defaults sit
|
||||
# at/above Apify's first-party actor rates (we charge no subscription tiers,
|
||||
# start fees, or separate proxy/compute billing). google_maps.scrape is
|
||||
# dual-metered (places + attached reviews).
|
||||
# PLATFORM_SCRAPE_BILLING_ENABLED=FALSE
|
||||
# REDDIT_SCRAPE_MICROS_PER_ITEM=3500
|
||||
# GOOGLE_SEARCH_MICROS_PER_SERP=5500
|
||||
# GOOGLE_MAPS_MICROS_PER_PLACE=3500
|
||||
# GOOGLE_MAPS_MICROS_PER_REVIEW=1500
|
||||
# YOUTUBE_MICROS_PER_VIDEO=2500
|
||||
# YOUTUBE_MICROS_PER_COMMENT=1500
|
||||
|
||||
# Safety ceiling on per-call premium reservation, in micro-USD ($1.00 default).
|
||||
# QUOTA_MAX_RESERVE_MICROS=1000000
|
||||
|
||||
|
|
@ -467,12 +482,37 @@ NOLOGIN_MODE_ENABLED=FALSE
|
|||
# Connector indexing lock TTL in seconds (default: 28800 = 8 hours)
|
||||
# CONNECTOR_INDEXING_LOCK_TTL_SECONDS=28800
|
||||
|
||||
# Residential proxy for web crawling
|
||||
# RESIDENTIAL_PROXY_USERNAME=
|
||||
# RESIDENTIAL_PROXY_PASSWORD=
|
||||
# RESIDENTIAL_PROXY_HOSTNAME=
|
||||
# RESIDENTIAL_PROXY_LOCATION=
|
||||
# RESIDENTIAL_PROXY_TYPE=1
|
||||
# Proxy provider selection: "custom" (default) or "dataimpulse".
|
||||
# PROXY_PROVIDER=custom
|
||||
|
||||
# Proxy endpoint(s), shared across providers. PROXY_URL is a single full URL used
|
||||
# by every provider (for "dataimpulse", country is a "__cr.<country>" username
|
||||
# suffix the provider parses for geoip). PROXY_URLS is a comma-separated pool the
|
||||
# "custom" provider rotates client-side. Leave unset to disable proxying.
|
||||
# PROXY_URL=http://user:pass@host:port
|
||||
# PROXY_URLS=http://user:pass@host1:port,http://user:pass@host2:port
|
||||
|
||||
# Captcha solving — last-resort bypass tier via captchatools. Only fires on the
|
||||
# stealth browser tier when a sitekey is detected AND the flag is TRUE.
|
||||
# Cloudflare Turnstile is already solved free in-framework. Off by default.
|
||||
# NOTE: automated solving may violate a target site's ToS — opt-in, public
|
||||
# data only. See surfsense_backend/.env.example for the full option docs.
|
||||
# CAPTCHA_SOLVING_ENABLED=FALSE
|
||||
# CAPTCHA_SOLVER_PROVIDER=capsolver
|
||||
# CAPTCHA_SOLVER_API_KEY=
|
||||
# CAPTCHA_MAX_ATTEMPTS_PER_URL=1
|
||||
# CAPTCHA_SOLVE_TIMEOUT_S=120
|
||||
# CAPTCHA_TYPE_DEFAULT=v2
|
||||
# CAPTCHA_V3_MIN_SCORE=0.7
|
||||
# CAPTCHA_V3_ACTION=verify
|
||||
|
||||
# Stealth hardening levers on the stealth browser tier. Defaults preserve
|
||||
# current behavior; see surfsense_backend/.env.example for per-flag docs.
|
||||
# CRAWL_GEOIP_MATCH_ENABLED=FALSE
|
||||
# CRAWL_BLOCK_WEBRTC=TRUE
|
||||
# CRAWL_HIDE_CANVAS=FALSE
|
||||
# CRAWL_GOOGLE_SEARCH_REFERER=TRUE
|
||||
# CRAWL_DNS_OVER_HTTPS=FALSE
|
||||
|
||||
# ==============================================================================
|
||||
# DEV / DEPS-ONLY COMPOSE OVERRIDES
|
||||
|
|
@ -489,9 +529,6 @@ NOLOGIN_MODE_ENABLED=FALSE
|
|||
# -- Redis exposed port (dev/deps-only only; Redis is internal-only in prod) --
|
||||
# REDIS_PORT=6379
|
||||
|
||||
# -- SearXNG exposed port (dev/deps-only only; internal-only in prod) --
|
||||
# SEARXNG_PORT=8888
|
||||
|
||||
# -- WhatsApp bridge exposed port (dev/hybrid only; prod keeps it Docker-internal) --
|
||||
# WHATSAPP_BRIDGE_PORT=9929
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
# =============================================================================
|
||||
# SurfSense — Dependencies only (no backend / frontend / Celery images)
|
||||
# =============================================================================
|
||||
# Postgres, Redis, SearXNG, pgAdmin, Zero — run API + Next + Celery on the host.
|
||||
# Postgres, Redis, pgAdmin, Zero — run API + Next + Celery on the host.
|
||||
# Celery is not Dockerized here: use `uv run` from surfsense_backend/ (no extra
|
||||
# backend image build just for workers).
|
||||
#
|
||||
|
|
@ -9,7 +9,7 @@
|
|||
# docker compose -f docker/docker-compose.deps-only.yml up -d
|
||||
#
|
||||
# Compose variable substitution uses `docker/.env` (copy from .env.example).
|
||||
# Bind mounts use ./postgresql.conf and ./searxng in this directory.
|
||||
# Bind mounts use ./postgresql.conf in this directory.
|
||||
#
|
||||
# Local Celery (from surfsense_backend/, after Redis is up):
|
||||
# uv run celery -A celery_worker.celery_app worker --loglevel=info --concurrency=1 --pool=solo --queues=surfsense,surfsense.connectors
|
||||
|
|
@ -17,7 +17,6 @@
|
|||
#
|
||||
# Host setup:
|
||||
# - Backend .env: DATABASE_URL=postgresql+asyncpg://postgres:postgres@localhost:5432/surfsense
|
||||
# - Backend .env: SEARXNG_DEFAULT_HOST=http://localhost:${SEARXNG_PORT:-8888}
|
||||
# - Backend .env: CELERY_BROKER_URL / REDIS_APP_URL → redis://localhost:6379/0
|
||||
# - Web .env: NEXT_PUBLIC_ZERO_CACHE_URL=http://localhost:${ZERO_CACHE_PORT:-4848}
|
||||
#
|
||||
|
|
@ -80,20 +79,6 @@ services:
|
|||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
searxng:
|
||||
image: searxng/searxng:2026.3.13-3c1f68c59
|
||||
ports:
|
||||
- "${SEARXNG_PORT:-8888}:8080"
|
||||
volumes:
|
||||
- ./searxng:/etc/searxng
|
||||
environment:
|
||||
- SEARXNG_SECRET=${SEARXNG_SECRET:-surfsense-searxng-secret}
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "--spider", "-q", "http://localhost:8080/healthz"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
# NOTE: zero-cache requires the `zero_publication` Postgres publication to
|
||||
# exist before it starts. In this deps-only stack there is no backend
|
||||
# container to run migrations, so you must run `uv run alembic upgrade head`
|
||||
|
|
|
|||
|
|
@ -85,20 +85,6 @@ services:
|
|||
- "${OTEL_TEMPO_PORT:-3200}:3200"
|
||||
restart: unless-stopped
|
||||
|
||||
searxng:
|
||||
image: searxng/searxng:2026.3.13-3c1f68c59
|
||||
ports:
|
||||
- "${SEARXNG_PORT:-8888}:8080"
|
||||
volumes:
|
||||
- ./searxng:/etc/searxng
|
||||
environment:
|
||||
- SEARXNG_SECRET=${SEARXNG_SECRET:-surfsense-searxng-secret}
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "--spider", "-q", "http://localhost:8080/healthz"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
backend:
|
||||
build: *backend-build
|
||||
ports:
|
||||
|
|
@ -125,7 +111,6 @@ services:
|
|||
- LANGSMITH_TRACING=false
|
||||
- AUTH_TYPE=${AUTH_TYPE:-LOCAL}
|
||||
- NEXT_FRONTEND_URL=${NEXT_FRONTEND_URL:-http://localhost:3000}
|
||||
- SEARXNG_DEFAULT_HOST=${SEARXNG_DEFAULT_HOST:-http://searxng:8080}
|
||||
- WHATSAPP_BRIDGE_URL=${WHATSAPP_BRIDGE_URL:-http://whatsapp-bridge:9929}
|
||||
# Daytona Sandbox – uncomment and set credentials to enable cloud code execution
|
||||
# - DAYTONA_SANDBOX_ENABLED=TRUE
|
||||
|
|
@ -138,8 +123,6 @@ services:
|
|||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
searxng:
|
||||
condition: service_healthy
|
||||
migrations:
|
||||
condition: service_completed_successfully
|
||||
healthcheck:
|
||||
|
|
@ -186,7 +169,6 @@ services:
|
|||
- CELERY_TASK_DEFAULT_QUEUE=surfsense
|
||||
- PYTHONPATH=/app
|
||||
- FILE_STORAGE_LOCAL_PATH=/app/.local_object_store
|
||||
- SEARXNG_DEFAULT_HOST=${SEARXNG_DEFAULT_HOST:-http://searxng:8080}
|
||||
- SERVICE_ROLE=worker
|
||||
depends_on:
|
||||
db:
|
||||
|
|
|
|||
52
docker/docker-compose.watch-e2e.yml
Normal file
52
docker/docker-compose.watch-e2e.yml
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
# =============================================================================
|
||||
# SurfSense — Isolated deps for the watch (Phase 6) manual E2E
|
||||
# =============================================================================
|
||||
# Fresh Postgres + Redis ONLY, with their own project name, volumes, and host
|
||||
# ports so they never collide with the day-to-day `surfsense-deps` stack.
|
||||
#
|
||||
# Backend + Celery run on the HOST (tests/e2e/run_backend.py / run_celery.py)
|
||||
# pointed at these ports via env:
|
||||
# DATABASE_URL=postgresql+asyncpg://postgres:postgres@localhost:5442/surfsense
|
||||
# CELERY_BROKER_URL / CELERY_RESULT_BACKEND / REDIS_APP_URL=redis://localhost:6389/0
|
||||
#
|
||||
# Up: docker compose -p surfsense-watch -f docker/docker-compose.watch-e2e.yml up -d
|
||||
# Down: docker compose -p surfsense-watch -f docker/docker-compose.watch-e2e.yml down -v
|
||||
# =============================================================================
|
||||
|
||||
name: surfsense-watch
|
||||
|
||||
services:
|
||||
db:
|
||||
image: pgvector/pgvector:pg17
|
||||
ports:
|
||||
- "5442:5432"
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
environment:
|
||||
- POSTGRES_USER=postgres
|
||||
- POSTGRES_PASSWORD=postgres
|
||||
- POSTGRES_DB=surfsense
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U postgres -d surfsense"]
|
||||
interval: 3s
|
||||
timeout: 5s
|
||||
retries: 20
|
||||
|
||||
redis:
|
||||
image: redis:8-alpine
|
||||
ports:
|
||||
- "6389:6379"
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
command: redis-server --appendonly yes
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 3s
|
||||
timeout: 5s
|
||||
retries: 20
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
name: surfsense-watch-postgres
|
||||
redis_data:
|
||||
name: surfsense-watch-redis
|
||||
|
|
@ -81,19 +81,6 @@ services:
|
|||
# timeout: 5s
|
||||
# retries: 3
|
||||
|
||||
searxng:
|
||||
image: searxng/searxng:2026.3.13-3c1f68c59
|
||||
volumes:
|
||||
- ./searxng:/etc/searxng
|
||||
environment:
|
||||
SEARXNG_SECRET: ${SEARXNG_SECRET:-surfsense-searxng-secret}
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "--spider", "-q", "http://localhost:8080/healthz"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
# Single public entry point for the Docker stack. Comment this service out
|
||||
# only if you front SurfSense with your own reverse proxy.
|
||||
proxy:
|
||||
|
|
@ -146,7 +133,6 @@ services:
|
|||
FILE_STORAGE_LOCAL_PATH: /app/.local_object_store
|
||||
NEXT_FRONTEND_URL: ${NEXT_FRONTEND_URL:-${SURFSENSE_PUBLIC_URL:-http://localhost:${LISTEN_HTTP_PORT:-3929}}}
|
||||
BACKEND_URL: ${BACKEND_URL:-${SURFSENSE_PUBLIC_URL:-http://localhost:${LISTEN_HTTP_PORT:-3929}}}
|
||||
SEARXNG_DEFAULT_HOST: ${SEARXNG_DEFAULT_HOST:-http://searxng:8080}
|
||||
WHATSAPP_BRIDGE_URL: ${WHATSAPP_BRIDGE_URL:-http://whatsapp-bridge:9929}
|
||||
# Daytona Sandbox – uncomment and set credentials to enable cloud code execution
|
||||
# DAYTONA_SANDBOX_ENABLED: "TRUE"
|
||||
|
|
@ -161,8 +147,6 @@ services:
|
|||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
searxng:
|
||||
condition: service_healthy
|
||||
migrations:
|
||||
condition: service_completed_successfully
|
||||
restart: unless-stopped
|
||||
|
|
@ -210,7 +194,6 @@ services:
|
|||
CELERY_TASK_DEFAULT_QUEUE: surfsense
|
||||
PYTHONPATH: /app
|
||||
FILE_STORAGE_LOCAL_PATH: /app/.local_object_store
|
||||
SEARXNG_DEFAULT_HOST: ${SEARXNG_DEFAULT_HOST:-http://searxng:8080}
|
||||
SERVICE_ROLE: worker
|
||||
depends_on:
|
||||
db:
|
||||
|
|
|
|||
|
|
@ -329,7 +329,6 @@ Write-Step "Downloading SurfSense files"
|
|||
Write-Info "Installation directory: $InstallDir"
|
||||
|
||||
New-Item -ItemType Directory -Path "$InstallDir\scripts" -Force | Out-Null
|
||||
New-Item -ItemType Directory -Path "$InstallDir\searxng" -Force | Out-Null
|
||||
New-Item -ItemType Directory -Path "$InstallDir\proxy" -Force | Out-Null
|
||||
|
||||
$Files = @(
|
||||
|
|
@ -339,8 +338,6 @@ $Files = @(
|
|||
@{ Src = "docker/proxy/Caddyfile"; Dest = "proxy/Caddyfile" }
|
||||
@{ Src = "docker/postgresql.conf"; Dest = "postgresql.conf" }
|
||||
@{ Src = "docker/scripts/migrate-database.ps1"; Dest = "scripts/migrate-database.ps1" }
|
||||
@{ Src = "docker/searxng/settings.yml"; Dest = "searxng/settings.yml" }
|
||||
@{ Src = "docker/searxng/limiter.toml"; Dest = "searxng/limiter.toml" }
|
||||
)
|
||||
|
||||
foreach ($f in $Files) {
|
||||
|
|
|
|||
|
|
@ -332,7 +332,6 @@ SELECTED_VARIANT=$(resolve_variant)
|
|||
step "Downloading SurfSense files"
|
||||
info "Installation directory: ${INSTALL_DIR}"
|
||||
mkdir -p "${INSTALL_DIR}/scripts"
|
||||
mkdir -p "${INSTALL_DIR}/searxng"
|
||||
mkdir -p "${INSTALL_DIR}/proxy"
|
||||
|
||||
FILES=(
|
||||
|
|
@ -342,8 +341,6 @@ FILES=(
|
|||
"docker/proxy/Caddyfile:proxy/Caddyfile"
|
||||
"docker/postgresql.conf:postgresql.conf"
|
||||
"docker/scripts/migrate-database.sh:scripts/migrate-database.sh"
|
||||
"docker/searxng/settings.yml:searxng/settings.yml"
|
||||
"docker/searxng/limiter.toml:searxng/limiter.toml"
|
||||
)
|
||||
|
||||
for entry in "${FILES[@]}"; do
|
||||
|
|
|
|||
|
|
@ -1,5 +0,0 @@
|
|||
[botdetection.ip_limit]
|
||||
link_token = false
|
||||
|
||||
[botdetection.ip_lists]
|
||||
pass_ip = ["0.0.0.0/0"]
|
||||
|
|
@ -1,90 +0,0 @@
|
|||
use_default_settings:
|
||||
engines:
|
||||
remove:
|
||||
- ahmia
|
||||
- torch
|
||||
- qwant
|
||||
- qwant news
|
||||
- qwant images
|
||||
- qwant videos
|
||||
- mojeek
|
||||
- mojeek images
|
||||
- mojeek news
|
||||
|
||||
server:
|
||||
secret_key: "override-me-via-env"
|
||||
limiter: false
|
||||
image_proxy: false
|
||||
method: "GET"
|
||||
default_http_headers:
|
||||
X-Robots-Tag: "noindex, nofollow"
|
||||
|
||||
search:
|
||||
formats:
|
||||
- html
|
||||
- json
|
||||
default_lang: "auto"
|
||||
autocomplete: ""
|
||||
safe_search: 0
|
||||
ban_time_on_fail: 5
|
||||
max_ban_time_on_fail: 120
|
||||
suspended_times:
|
||||
SearxEngineAccessDenied: 3600
|
||||
SearxEngineCaptcha: 3600
|
||||
SearxEngineTooManyRequests: 600
|
||||
cf_SearxEngineCaptcha: 7200
|
||||
cf_SearxEngineAccessDenied: 3600
|
||||
recaptcha_SearxEngineCaptcha: 7200
|
||||
|
||||
ui:
|
||||
static_use_hash: true
|
||||
|
||||
outgoing:
|
||||
request_timeout: 12.0
|
||||
max_request_timeout: 20.0
|
||||
pool_connections: 100
|
||||
pool_maxsize: 20
|
||||
enable_http2: true
|
||||
extra_proxy_timeout: 10
|
||||
retries: 1
|
||||
# Uncomment and set your residential proxy URL to route search engine requests through it.
|
||||
# Format: http://<username>:<base64_password>@<hostname>:<port>/
|
||||
#
|
||||
# proxies:
|
||||
# all://:
|
||||
# - http://user:pass@proxy-host:port/
|
||||
|
||||
engines:
|
||||
- name: google
|
||||
disabled: false
|
||||
weight: 1.2
|
||||
retry_on_http_error: [429, 503]
|
||||
- name: duckduckgo
|
||||
disabled: false
|
||||
weight: 1.1
|
||||
retry_on_http_error: [429, 503]
|
||||
- name: brave
|
||||
disabled: false
|
||||
weight: 1.0
|
||||
retry_on_http_error: [429, 503]
|
||||
- name: bing
|
||||
disabled: false
|
||||
weight: 0.9
|
||||
retry_on_http_error: [429, 503]
|
||||
- name: wikipedia
|
||||
disabled: false
|
||||
weight: 0.8
|
||||
- name: stackoverflow
|
||||
disabled: false
|
||||
weight: 0.7
|
||||
- name: yahoo
|
||||
disabled: false
|
||||
weight: 0.7
|
||||
retry_on_http_error: [429, 503]
|
||||
- name: wikidata
|
||||
disabled: false
|
||||
weight: 0.6
|
||||
- name: currency
|
||||
disabled: false
|
||||
- name: ddg definitions
|
||||
disabled: false
|
||||
|
|
@ -6,6 +6,12 @@
|
|||
"sourceType": "github",
|
||||
"computedHash": "f96aa6f7d8dd982a866ac090c1770e39cd0a01c63c9bc315895d3c210f80b393"
|
||||
},
|
||||
"animation-vocabulary": {
|
||||
"source": "emilkowalski/skills",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/animation-vocabulary/SKILL.md",
|
||||
"computedHash": "280c6688ef2568ad92f40c4dc638f0cc5cde0f49bdd6debd097410aa9945f44f"
|
||||
},
|
||||
"backlink-analyzer": {
|
||||
"source": "aaron-he-zhu/seo-geo-claude-skills",
|
||||
"sourceType": "github",
|
||||
|
|
@ -36,6 +42,12 @@
|
|||
"sourceType": "github",
|
||||
"computedHash": "e97547ac36ea8e59a65bdfdda25b8d130cb7f5e2cf13dac56fd01f24e74718a5"
|
||||
},
|
||||
"emil-design-eng": {
|
||||
"source": "emilkowalski/skills",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/emil-design-eng/SKILL.md",
|
||||
"computedHash": "ac3fc5b4e206488ffc408a2806f829bb2e5eaf518fded09be34790685c2b3a0d"
|
||||
},
|
||||
"entity-optimizer": {
|
||||
"source": "aaron-he-zhu/seo-geo-claude-skills",
|
||||
"sourceType": "github",
|
||||
|
|
@ -87,6 +99,12 @@
|
|||
"sourceType": "github",
|
||||
"computedHash": "2d4c9a44056713d946481031cc00b3113b6055af6f4bea0cfd8dcb8a9d5abbf9"
|
||||
},
|
||||
"review-animations": {
|
||||
"source": "emilkowalski/skills",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/review-animations/SKILL.md",
|
||||
"computedHash": "57c681344448379fbec8c11b45e355b70799998427240662318d017f292571bd"
|
||||
},
|
||||
"schema-markup-generator": {
|
||||
"source": "aaron-he-zhu/seo-geo-claude-skills",
|
||||
"sourceType": "github",
|
||||
|
|
|
|||
|
|
@ -55,11 +55,6 @@ WHATSAPP_WEBHOOK_VERIFY_TOKEN=
|
|||
WHATSAPP_WEBHOOK_APP_SECRET=
|
||||
WHATSAPP_BRIDGE_URL=http://whatsapp-bridge:9929
|
||||
|
||||
# Platform Web Search (SearXNG)
|
||||
# Set this to enable built-in web search. Docker Compose sets it automatically.
|
||||
# Only uncomment if running the backend outside Docker (e.g. uvicorn on host).
|
||||
# SEARXNG_DEFAULT_HOST=http://localhost:8888
|
||||
|
||||
# Periodic task interval
|
||||
# # Run every minute (default)
|
||||
# SCHEDULE_CHECKER_INTERVAL=1m
|
||||
|
|
@ -257,6 +252,41 @@ DEFAULT_CREDIT_MICROS_BALANCE=5000000
|
|||
ETL_CREDIT_BILLING_ENABLED=FALSE
|
||||
MICROS_PER_PAGE=1000
|
||||
|
||||
# Debit the credit wallet per *successful* web crawl. Default FALSE keeps
|
||||
# crawling effectively free for self-hosted/OSS installs; hosted sets TRUE.
|
||||
# Price is fully config-driven (the only source of truth, no hardcoded rate):
|
||||
# WEB_CRAWL_MICROS_PER_SUCCESS = round(USD_per_1000_crawls * 1_000)
|
||||
# 2000 == $2 / 1000 crawls (default) | 1000 == $1/1000 | 500 == $0.50/1000
|
||||
# Retune anytime with just an env change + restart (no code/migration).
|
||||
# WEB_CRAWL_CREDIT_BILLING_ENABLED=FALSE
|
||||
# WEB_CRAWL_MICROS_PER_SUCCESS=2000
|
||||
|
||||
# Phase 3d: bill captcha solves as a SEPARATE per-attempt unit (the solver
|
||||
# charges per attempt regardless of crawl success). Independent of the crawl
|
||||
# flag above. Price config-driven (no hardcoded rate):
|
||||
# WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE = round(USD_per_1000_solves * 1_000)
|
||||
# 3000 == $3 / 1000 solves (default) | 5000 == $5/1000
|
||||
# Set with margin over your solver vendor's per-attempt price.
|
||||
# WEB_CRAWL_CAPTCHA_BILLING_ENABLED=FALSE
|
||||
# WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE=3000
|
||||
|
||||
# Debit the credit wallet per *item returned* by the platform-native scrapers
|
||||
# (Reddit, Google Search, Google Maps, YouTube). Default FALSE keeps scraping
|
||||
# effectively free for self-hosted/OSS installs; hosted deployments set TRUE.
|
||||
# Each rate is micro-USD per item, fully config-driven (no hardcoded rate):
|
||||
# <KEY> = round(USD_per_1000_items * 1_000)
|
||||
# 3500 == $3.50/1000 | 5000 == $5/1000 | 2000 == $2/1000
|
||||
# Defaults sit at/above Apify's first-party actor rates (Jul 2026); justified
|
||||
# because we charge no subscription tiers, no per-run start fees, and no
|
||||
# separate proxy/compute/storage billing. Retune anytime via env + restart.
|
||||
# PLATFORM_SCRAPE_BILLING_ENABLED=FALSE
|
||||
# REDDIT_SCRAPE_MICROS_PER_ITEM=3500
|
||||
# GOOGLE_SEARCH_MICROS_PER_SERP=5500
|
||||
# GOOGLE_MAPS_MICROS_PER_PLACE=3500
|
||||
# GOOGLE_MAPS_MICROS_PER_REVIEW=1500
|
||||
# YOUTUBE_MICROS_PER_VIDEO=2500
|
||||
# YOUTUBE_MICROS_PER_COMMENT=1500
|
||||
|
||||
# Low-balance warning threshold (micro-USD), surfaced to the UI. Default $0.50.
|
||||
CREDIT_LOW_BALANCE_WARNING_MICROS=500000
|
||||
|
||||
|
|
@ -313,17 +343,57 @@ TURNSTILE_SECRET_KEY=
|
|||
|
||||
|
||||
# Proxy provider selection. Selects a ProxyProvider implementation registered in
|
||||
# app/utils/proxy/registry.py. Default: "anonymous_proxies". Add new vendors there.
|
||||
# PROXY_PROVIDER=anonymous_proxies
|
||||
# app/utils/proxy/registry.py. Default: "custom". Options: "custom", "dataimpulse".
|
||||
# PROXY_PROVIDER=custom
|
||||
|
||||
# Residential Proxy Configuration (anonymous-proxies.net)
|
||||
# Used for web crawling, link previews, and YouTube transcript fetching to avoid IP bans.
|
||||
# Consumed by the "anonymous_proxies" provider. Leave commented out to disable proxying.
|
||||
# RESIDENTIAL_PROXY_USERNAME=your_proxy_username
|
||||
# RESIDENTIAL_PROXY_PASSWORD=your_proxy_password
|
||||
# RESIDENTIAL_PROXY_HOSTNAME=rotating.dnsproxifier.com:31230
|
||||
# RESIDENTIAL_PROXY_LOCATION=
|
||||
# RESIDENTIAL_PROXY_TYPE=1
|
||||
# Proxy endpoint(s), shared across providers — PROXY_PROVIDER picks the behavior,
|
||||
# not a different env name. Used for web crawling, link previews, and YouTube
|
||||
# transcript fetching to avoid IP bans. Leave unset to disable proxying.
|
||||
#
|
||||
# PROXY_URL: a single full http://user:pass@host:port endpoint (used by every
|
||||
# provider). For "dataimpulse", country is a "__cr.<country>" username suffix the
|
||||
# provider parses for geoip-match, e.g.:
|
||||
# PROXY_URL=http://your_token__cr.us:your_password@gw.dataimpulse.com:823
|
||||
# PROXY_URLS: a comma-separated pool the "custom" provider rotates client-side
|
||||
# (cyclic); server-side-rotating gateways ignore it.
|
||||
# PROXY_URL=http://user:pass@host:port
|
||||
# PROXY_URLS=http://user:pass@host1:port,http://user:pass@host2:port
|
||||
|
||||
# =====================================================================
|
||||
# Captcha solving (Phase 3d) — LAST-resort bypass tier via captchatools.
|
||||
# Only fires on the stealth browser tier when a sitekey is detected AND
|
||||
# CAPTCHA_SOLVING_ENABLED=TRUE. Cloudflare Turnstile is already solved free
|
||||
# in-framework (no config needed). Off by default => zero attempts, zero cost.
|
||||
# NOTE: automated solving may violate a target site's ToS — opt-in, public
|
||||
# data only (no logged-in bypass). captchatools is itself the vendor registry.
|
||||
# CAPTCHA_SOLVING_ENABLED=FALSE
|
||||
# solving_site: capmonster | 2captcha | anticaptcha | capsolver | captchaai
|
||||
# CAPTCHA_SOLVER_PROVIDER=capsolver
|
||||
# CAPTCHA_SOLVER_API_KEY=
|
||||
# Per-URL solve cap (bounds solver spend on a hostile page).
|
||||
# CAPTCHA_MAX_ATTEMPTS_PER_URL=1
|
||||
# CAPTCHA_SOLVE_TIMEOUT_S=120
|
||||
# Default type when detection is ambiguous: v2 | v3 | hcaptcha
|
||||
# CAPTCHA_TYPE_DEFAULT=v2
|
||||
# reCAPTCHA v3 tuning (only used for v3 challenges).
|
||||
# CAPTCHA_V3_MIN_SCORE=0.7
|
||||
# CAPTCHA_V3_ACTION=verify
|
||||
|
||||
# =====================================================================
|
||||
# Stealth hardening (Phase 3e, Slice A) — runtime/config-level levers on the
|
||||
# stealth browser tier. Consumed by app/proprietary/web_crawler/stealth.py.
|
||||
# Defaults add no crawl-speed regression and preserve today's behavior.
|
||||
# Match browser locale/timezone to the proxy provider's exit region (no exit-IP lookup).
|
||||
# CRAWL_GEOIP_MATCH_ENABLED=FALSE
|
||||
# WebRTC respects the proxy (anti local-IP leak). Cheap + safe.
|
||||
# CRAWL_BLOCK_WEBRTC=TRUE
|
||||
# Random canvas noise (an unstable canvas hash is itself a tell) — opt-in only.
|
||||
# CRAWL_HIDE_CANVAS=FALSE
|
||||
# Send a Google referer so the first hit looks like organic arrival.
|
||||
# CRAWL_GOOGLE_SEARCH_REFERER=TRUE
|
||||
# Route DNS via Cloudflare DoH (anti DNS-leak). Adds a DNS round-trip => off by
|
||||
# default to avoid any speed regression; enable for leak-safety-first setups.
|
||||
# CRAWL_DNS_OVER_HTTPS=FALSE
|
||||
|
||||
# File Parser Service
|
||||
ETL_SERVICE=UNSTRUCTURED or LLAMACLOUD or DOCLING
|
||||
|
|
@ -472,7 +542,7 @@ LANGSMITH_PROJECT=surfsense
|
|||
# -----------------------------------------------------------------------------
|
||||
# Connector discovery TTL cache (Phase 1.4 perf optimization)
|
||||
# -----------------------------------------------------------------------------
|
||||
# Caches the per-search-space "available connectors" + "available document
|
||||
# Caches the per-workspace "available connectors" + "available document
|
||||
# types" lookups that ``create_surfsense_deep_agent`` hits on every turn.
|
||||
# ORM event listeners auto-invalidate on connector / document inserts,
|
||||
# updates and deletes — the TTL only bounds staleness for bulk-import
|
||||
|
|
@ -512,8 +582,8 @@ LANGSMITH_PROJECT=surfsense
|
|||
# Per-workspace spawn-paused kill switch — set via Redis at runtime, not
|
||||
# this env var. The env var below only disables the check itself (useful
|
||||
# for local dev without Redis). To pause a workspace in production:
|
||||
# redis-cli SET surfsense:spawn_paused:<search_space_id> 1 EX 600
|
||||
# redis-cli DEL surfsense:spawn_paused:<search_space_id>
|
||||
# redis-cli SET surfsense:spawn_paused:<workspace_id> 1 EX 600
|
||||
# redis-cli DEL surfsense:spawn_paused:<workspace_id>
|
||||
# The check is fail-open: a Redis blip never blocks ``task(...)``.
|
||||
# SURFSENSE_TASK_SPAWN_PAUSED_DISABLED=false
|
||||
|
||||
|
|
|
|||
|
|
@ -30,6 +30,20 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
|||
libxrender1 \
|
||||
dos2unix \
|
||||
git \
|
||||
# ── Phase 3e stealth hardening ──────────────────────────────────────────
|
||||
# Xvfb: virtual framebuffer so the stealth browser can run headful
|
||||
# (headless=False) without a real display — many WAFs flag headless Chromium.
|
||||
# Gated at runtime by CRAWL_HEADED_XVFB_ENABLED (Slice B); installed now so
|
||||
# the image is ready. Real font packages make canvas/emoji/font-enumeration
|
||||
# fingerprints resemble a real desktop (set proven against Kasada/Akamai).
|
||||
xvfb \
|
||||
fonts-noto-color-emoji \
|
||||
fonts-unifont \
|
||||
fonts-ipafont-gothic \
|
||||
fonts-wqy-zenhei \
|
||||
fonts-tlwg-loma-otf \
|
||||
fonts-dejavu \
|
||||
fonts-liberation \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN which ffmpeg && ffmpeg -version
|
||||
|
|
@ -109,7 +123,7 @@ RUN --mount=type=secret,id=HF_TOKEN \
|
|||
HF_TOKEN="$(cat /run/secrets/HF_TOKEN 2>/dev/null || true)" \
|
||||
python -c "from chonkie import AutoEmbeddings; AutoEmbeddings.get_embeddings('${EMBEDDING_MODEL}')"
|
||||
|
||||
# Install Scrapling's browser engines (patchright Chromium + Camoufox).
|
||||
# Install Scrapling's browser engines (patchright Chromium).
|
||||
# Scrapling pulls playwright/patchright via the `fetchers` extra; `scrapling install`
|
||||
# downloads the matching browser binaries used by DynamicFetcher/StealthyFetcher.
|
||||
RUN scrapling install
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from logging.config import fileConfig
|
||||
|
|
@ -9,6 +10,7 @@ from sqlalchemy.engine import Connection
|
|||
from sqlalchemy.ext.asyncio import async_engine_from_config
|
||||
|
||||
from alembic import context
|
||||
from alembic.script import ScriptDirectory
|
||||
|
||||
# Ensure the app directory is in the Python path
|
||||
# This allows Alembic to find your models
|
||||
|
|
@ -40,6 +42,153 @@ target_metadata = Base.metadata
|
|||
MIGRATION_ADVISORY_LOCK_NAMESPACE = "surfsense"
|
||||
MIGRATION_ADVISORY_LOCK_NAME = "alembic_migrations"
|
||||
|
||||
# Migration 170 renamed searchspaces -> workspaces, so a ``workspaces`` table
|
||||
# can only exist once the schema is at revision >= 170. If it exists while the
|
||||
# recorded revision is missing or still pre-170, the schema did not come from
|
||||
# this migration history at all -- it was created by the startup bootstrap
|
||||
# (``Base.metadata.create_all`` in ``app.db.create_db_and_tables``), which
|
||||
# always builds the *current* model shape. Replaying history against such a
|
||||
# schema fails (e.g. migration 5's ``ALTER COLUMN ... TYPE`` is rejected
|
||||
# because the column already sits in zero_publication's column list), so the
|
||||
# schema is adopted by stamping head instead.
|
||||
BOOTSTRAP_MARKER_TABLE = "workspaces"
|
||||
RENAME_REVISION = "170"
|
||||
|
||||
|
||||
def _stamp_head(connection: Connection, script: ScriptDirectory) -> None:
|
||||
context.get_context().stamp(script, script.get_current_head())
|
||||
if connection.in_transaction():
|
||||
# The outer begin_transaction() is a no-op under
|
||||
# transaction_per_migration, so commit explicitly.
|
||||
connection.commit()
|
||||
|
||||
|
||||
def _fast_forward_fresh_db(connection: Connection) -> bool:
|
||||
"""Build a fresh (empty) DB at head via create_all instead of replaying.
|
||||
|
||||
Historical migrations were written against the pre-workspace-rename
|
||||
schema (``searchspaces``, ``search_space_id``), while migration 0's
|
||||
``create_all`` builds the *current* models -- so replaying the chain on a
|
||||
fresh DB crashes as soon as a migration touches a renamed object (first
|
||||
at migration 18). A fresh DB needs no history: create the head-shape
|
||||
schema directly, mirror migration 0's indexes, create the Zero
|
||||
publication, and stamp head. Replay remains only for legacy DBs that
|
||||
genuinely contain the old objects.
|
||||
|
||||
ponytail: seed-data migrations (114/128 default prompts) are skipped on
|
||||
this path, same as always for create_all-bootstrapped DBs; the app copes
|
||||
with missing seeds. If seeds ever become mandatory, add a runtime seeding
|
||||
step rather than resurrecting the replay.
|
||||
"""
|
||||
for table in ("documents", "searchspaces", BOOTSTRAP_MARKER_TABLE):
|
||||
if connection.execute(sa.text("SELECT to_regclass(:t)"), {"t": table}).scalar():
|
||||
return False
|
||||
if connection.execute(sa.text("SELECT to_regclass('alembic_version')")).scalar():
|
||||
current = connection.execute(
|
||||
sa.text("SELECT version_num FROM alembic_version")
|
||||
).scalar()
|
||||
if current:
|
||||
return False
|
||||
|
||||
logging.getLogger("alembic.env").info(
|
||||
"Fresh database detected: creating head-shape schema via create_all "
|
||||
"and stamping head instead of replaying migration history."
|
||||
)
|
||||
connection.execute(sa.text("CREATE EXTENSION IF NOT EXISTS vector"))
|
||||
connection.execute(sa.text("CREATE EXTENSION IF NOT EXISTS pg_trgm"))
|
||||
Base.metadata.create_all(bind=connection)
|
||||
# Same core indexes migration 0 created (runtime setup_indexes() adds the
|
||||
# rest concurrently on app boot).
|
||||
connection.execute(
|
||||
sa.text(
|
||||
"CREATE INDEX IF NOT EXISTS document_vector_index ON documents "
|
||||
"USING hnsw (embedding public.vector_cosine_ops)"
|
||||
)
|
||||
)
|
||||
connection.execute(
|
||||
sa.text(
|
||||
"CREATE INDEX IF NOT EXISTS document_search_index ON documents "
|
||||
"USING gin (to_tsvector('english', content))"
|
||||
)
|
||||
)
|
||||
connection.execute(
|
||||
sa.text(
|
||||
"CREATE INDEX IF NOT EXISTS chucks_vector_index ON chunks "
|
||||
"USING hnsw (embedding public.vector_cosine_ops)"
|
||||
)
|
||||
)
|
||||
connection.execute(
|
||||
sa.text(
|
||||
"CREATE INDEX IF NOT EXISTS chucks_search_index ON chunks "
|
||||
"USING gin (to_tsvector('english', content))"
|
||||
)
|
||||
)
|
||||
|
||||
from app.zero_publication import ensure_publication
|
||||
|
||||
ensure_publication(connection)
|
||||
|
||||
_stamp_head(connection, ScriptDirectory.from_config(config))
|
||||
return True
|
||||
|
||||
|
||||
def _adopt_bootstrapped_schema(connection: Connection) -> bool:
|
||||
"""Stamp head instead of replaying history on a create_all-created DB.
|
||||
|
||||
Returns True when the schema was adopted (migrations must then be
|
||||
skipped for this run).
|
||||
|
||||
ponytail: assumes the bootstrapped schema matches the checked-out models
|
||||
(true whenever the backend booted on this checkout, since create_all runs
|
||||
on every startup). If the checkout moved ahead without a backend boot,
|
||||
column-level drift from the skipped migrations is possible; the upgrade
|
||||
path is re-bootstrapping (boot the backend once) before stamping.
|
||||
"""
|
||||
marker = connection.execute(
|
||||
sa.text("SELECT to_regclass(:t)"), {"t": BOOTSTRAP_MARKER_TABLE}
|
||||
).scalar()
|
||||
if marker is None:
|
||||
return False
|
||||
|
||||
# Guard against a legacy-shape DB that merely had missing tables filled in
|
||||
# by a later create_all: adoption requires the core tables to be in the
|
||||
# current (post-rename) shape too, not just the marker table to exist.
|
||||
documents_renamed = connection.execute(
|
||||
sa.text(
|
||||
"SELECT 1 FROM information_schema.columns "
|
||||
"WHERE table_schema = current_schema() "
|
||||
"AND table_name = 'documents' AND column_name = 'workspace_id'"
|
||||
)
|
||||
).scalar()
|
||||
if not documents_renamed:
|
||||
return False
|
||||
|
||||
current = None
|
||||
if connection.execute(sa.text("SELECT to_regclass('alembic_version')")).scalar():
|
||||
current = connection.execute(
|
||||
sa.text("SELECT version_num FROM alembic_version")
|
||||
).scalar()
|
||||
|
||||
script = ScriptDirectory.from_config(config)
|
||||
pre_rename_revisions = {
|
||||
rev.revision for rev in script.iterate_revisions(RENAME_REVISION, "base")
|
||||
} - {RENAME_REVISION}
|
||||
if current is not None and current not in pre_rename_revisions:
|
||||
# Genuinely migration-managed at >= 170; run migrations normally.
|
||||
return False
|
||||
|
||||
logging.getLogger("alembic.env").info(
|
||||
"Adopting bootstrap-created schema (%r exists, recorded revision %r "
|
||||
"predates the workspace rename): stamping %s instead of replaying "
|
||||
"migration history.",
|
||||
BOOTSTRAP_MARKER_TABLE,
|
||||
current,
|
||||
script.get_current_head(),
|
||||
)
|
||||
_stamp_head(connection, script)
|
||||
return True
|
||||
|
||||
|
||||
# other values from the config, defined by the needs of env.py,
|
||||
# can be acquired:
|
||||
# my_important_option = config.get_main_option("my_important_option")
|
||||
|
|
@ -86,8 +235,11 @@ def do_run_migrations(connection: Connection) -> None:
|
|||
lock_params,
|
||||
)
|
||||
try:
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
if not _fast_forward_fresh_db(connection) and not _adopt_bootstrapped_schema(
|
||||
connection
|
||||
):
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
finally:
|
||||
connection.execute(
|
||||
sa.text("SELECT pg_advisory_unlock(hashtext(:namespace), hashtext(:name))"),
|
||||
|
|
|
|||
|
|
@ -50,9 +50,7 @@ def upgrade() -> None:
|
|||
"""
|
||||
)
|
||||
|
||||
op.execute(
|
||||
"ALTER TABLE refresh_tokens ALTER COLUMN token_hash TYPE VARCHAR(64)"
|
||||
)
|
||||
op.execute("ALTER TABLE refresh_tokens ALTER COLUMN token_hash TYPE VARCHAR(64)")
|
||||
op.execute("ALTER TABLE refresh_tokens DROP COLUMN IF EXISTS is_revoked")
|
||||
|
||||
|
||||
|
|
@ -63,14 +61,19 @@ def downgrade() -> None:
|
|||
)
|
||||
op.execute(
|
||||
"""
|
||||
UPDATE refresh_tokens
|
||||
SET is_revoked = TRUE
|
||||
WHERE revoked_at IS NOT NULL
|
||||
DO $$
|
||||
BEGIN
|
||||
IF EXISTS (
|
||||
SELECT 1 FROM information_schema.columns
|
||||
WHERE table_name = 'refresh_tokens'
|
||||
AND column_name = 'revoked_at'
|
||||
) THEN
|
||||
UPDATE refresh_tokens SET is_revoked = TRUE WHERE revoked_at IS NOT NULL;
|
||||
END IF;
|
||||
END $$;
|
||||
"""
|
||||
)
|
||||
op.execute("ALTER TABLE refresh_tokens ALTER COLUMN is_revoked DROP DEFAULT")
|
||||
op.execute(
|
||||
"ALTER TABLE refresh_tokens ALTER COLUMN token_hash TYPE VARCHAR(256)"
|
||||
)
|
||||
op.execute("ALTER TABLE refresh_tokens ALTER COLUMN token_hash TYPE VARCHAR(256)")
|
||||
op.execute("ALTER TABLE refresh_tokens DROP COLUMN IF EXISTS absolute_expiry")
|
||||
op.execute("ALTER TABLE refresh_tokens DROP COLUMN IF EXISTS revoked_at")
|
||||
|
|
|
|||
|
|
@ -0,0 +1,512 @@
|
|||
"""rename searchspace schema to workspace
|
||||
|
||||
Renames the SearchSpace tables/columns/constraints/indexes/sequences to the
|
||||
workspace_* names the ORM already declares, and reconciles the Zero publication
|
||||
to match.
|
||||
|
||||
Revision ID: 170
|
||||
Revises: 169
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
from app.zero_publication import apply_publication
|
||||
|
||||
revision: str = "170"
|
||||
down_revision: str | None = "169"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
PUBLICATION_NAME = "zero_publication"
|
||||
|
||||
# Tables whose published column list references the renamed column. Their
|
||||
# column-list dependency must be removed before RENAME COLUMN is permitted,
|
||||
# then re-added by apply_publication with the new column name.
|
||||
PUBLICATION_COLUMN_LIST_TABLES = [
|
||||
"documents",
|
||||
"automations",
|
||||
"new_chat_threads",
|
||||
"podcasts",
|
||||
]
|
||||
|
||||
# (old_table, new_table)
|
||||
TABLE_RENAMES: list[tuple[str, str]] = [
|
||||
("searchspaces", "workspaces"),
|
||||
("search_space_roles", "workspace_roles"),
|
||||
("search_space_memberships", "workspace_memberships"),
|
||||
("search_space_invites", "workspace_invites"),
|
||||
]
|
||||
|
||||
# (table, old_column, new_column) -- table names are POST-table-rename.
|
||||
COLUMN_RENAMES: list[tuple[str, str, str]] = [
|
||||
("agent_action_log", "search_space_id", "workspace_id"),
|
||||
("agent_permission_rules", "search_space_id", "workspace_id"),
|
||||
("automations", "search_space_id", "workspace_id"),
|
||||
("connections", "search_space_id", "workspace_id"),
|
||||
("document_files", "search_space_id", "workspace_id"),
|
||||
("document_revisions", "search_space_id", "workspace_id"),
|
||||
("documents", "search_space_id", "workspace_id"),
|
||||
("external_chat_bindings", "search_space_id", "workspace_id"),
|
||||
("folder_revisions", "search_space_id", "workspace_id"),
|
||||
("folders", "search_space_id", "workspace_id"),
|
||||
("image_generations", "search_space_id", "workspace_id"),
|
||||
("logs", "search_space_id", "workspace_id"),
|
||||
("new_chat_threads", "search_space_id", "workspace_id"),
|
||||
("notifications", "search_space_id", "workspace_id"),
|
||||
("podcasts", "search_space_id", "workspace_id"),
|
||||
("prompts", "search_space_id", "workspace_id"),
|
||||
("reports", "search_space_id", "workspace_id"),
|
||||
("search_source_connectors", "search_space_id", "workspace_id"),
|
||||
("token_usage", "search_space_id", "workspace_id"),
|
||||
("video_presentations", "search_space_id", "workspace_id"),
|
||||
("external_chat_accounts", "owner_search_space_id", "owner_workspace_id"),
|
||||
("workspace_roles", "search_space_id", "workspace_id"),
|
||||
("workspace_memberships", "search_space_id", "workspace_id"),
|
||||
("workspace_invites", "search_space_id", "workspace_id"),
|
||||
]
|
||||
|
||||
# (table, old_constraint, new_constraint) -- table names are POST-table-rename.
|
||||
CONSTRAINT_RENAMES: list[tuple[str, str, str]] = [
|
||||
(
|
||||
"agent_action_log",
|
||||
"agent_action_log_search_space_id_fkey",
|
||||
"agent_action_log_workspace_id_fkey",
|
||||
),
|
||||
(
|
||||
"agent_permission_rules",
|
||||
"agent_permission_rules_search_space_id_fkey",
|
||||
"agent_permission_rules_workspace_id_fkey",
|
||||
),
|
||||
(
|
||||
"automations",
|
||||
"automations_search_space_id_fkey",
|
||||
"automations_workspace_id_fkey",
|
||||
),
|
||||
(
|
||||
"connections",
|
||||
"connections_search_space_id_fkey",
|
||||
"connections_workspace_id_fkey",
|
||||
),
|
||||
(
|
||||
"document_files",
|
||||
"document_files_search_space_id_fkey",
|
||||
"document_files_workspace_id_fkey",
|
||||
),
|
||||
(
|
||||
"document_revisions",
|
||||
"document_revisions_search_space_id_fkey",
|
||||
"document_revisions_workspace_id_fkey",
|
||||
),
|
||||
("documents", "documents_search_space_id_fkey", "documents_workspace_id_fkey"),
|
||||
(
|
||||
"external_chat_accounts",
|
||||
"external_chat_accounts_owner_search_space_id_fkey",
|
||||
"external_chat_accounts_owner_workspace_id_fkey",
|
||||
),
|
||||
(
|
||||
"external_chat_bindings",
|
||||
"external_chat_bindings_search_space_id_fkey",
|
||||
"external_chat_bindings_workspace_id_fkey",
|
||||
),
|
||||
(
|
||||
"folder_revisions",
|
||||
"folder_revisions_search_space_id_fkey",
|
||||
"folder_revisions_workspace_id_fkey",
|
||||
),
|
||||
("folders", "folders_search_space_id_fkey", "folders_workspace_id_fkey"),
|
||||
(
|
||||
"image_generations",
|
||||
"image_generations_search_space_id_fkey",
|
||||
"image_generations_workspace_id_fkey",
|
||||
),
|
||||
("logs", "logs_search_space_id_fkey", "logs_workspace_id_fkey"),
|
||||
(
|
||||
"new_chat_threads",
|
||||
"new_chat_threads_search_space_id_fkey",
|
||||
"new_chat_threads_workspace_id_fkey",
|
||||
),
|
||||
(
|
||||
"notifications",
|
||||
"notifications_search_space_id_fkey",
|
||||
"notifications_workspace_id_fkey",
|
||||
),
|
||||
("podcasts", "podcasts_search_space_id_fkey", "podcasts_workspace_id_fkey"),
|
||||
("prompts", "prompts_search_space_id_fkey", "prompts_workspace_id_fkey"),
|
||||
("reports", "reports_search_space_id_fkey", "reports_workspace_id_fkey"),
|
||||
(
|
||||
"search_source_connectors",
|
||||
"search_source_connectors_search_space_id_fkey",
|
||||
"search_source_connectors_workspace_id_fkey",
|
||||
),
|
||||
(
|
||||
"search_source_connectors",
|
||||
"uq_searchspace_user_connector_type_name",
|
||||
"uq_workspace_user_connector_type_name",
|
||||
),
|
||||
(
|
||||
"token_usage",
|
||||
"token_usage_search_space_id_fkey",
|
||||
"token_usage_workspace_id_fkey",
|
||||
),
|
||||
(
|
||||
"video_presentations",
|
||||
"video_presentations_search_space_id_fkey",
|
||||
"video_presentations_workspace_id_fkey",
|
||||
),
|
||||
(
|
||||
"workspace_invites",
|
||||
"search_space_invites_created_by_id_fkey",
|
||||
"workspace_invites_created_by_id_fkey",
|
||||
),
|
||||
(
|
||||
"workspace_invites",
|
||||
"search_space_invites_pkey",
|
||||
"workspace_invites_pkey",
|
||||
),
|
||||
(
|
||||
"workspace_invites",
|
||||
"search_space_invites_role_id_fkey",
|
||||
"workspace_invites_role_id_fkey",
|
||||
),
|
||||
(
|
||||
"workspace_invites",
|
||||
"search_space_invites_search_space_id_fkey",
|
||||
"workspace_invites_workspace_id_fkey",
|
||||
),
|
||||
(
|
||||
"workspace_memberships",
|
||||
"search_space_memberships_invited_by_invite_id_fkey",
|
||||
"workspace_memberships_invited_by_invite_id_fkey",
|
||||
),
|
||||
(
|
||||
"workspace_memberships",
|
||||
"search_space_memberships_pkey",
|
||||
"workspace_memberships_pkey",
|
||||
),
|
||||
(
|
||||
"workspace_memberships",
|
||||
"search_space_memberships_role_id_fkey",
|
||||
"workspace_memberships_role_id_fkey",
|
||||
),
|
||||
(
|
||||
"workspace_memberships",
|
||||
"search_space_memberships_search_space_id_fkey",
|
||||
"workspace_memberships_workspace_id_fkey",
|
||||
),
|
||||
(
|
||||
"workspace_memberships",
|
||||
"search_space_memberships_user_id_fkey",
|
||||
"workspace_memberships_user_id_fkey",
|
||||
),
|
||||
(
|
||||
"workspace_memberships",
|
||||
"uq_user_searchspace_membership",
|
||||
"uq_user_workspace_membership",
|
||||
),
|
||||
(
|
||||
"workspace_roles",
|
||||
"search_space_roles_pkey",
|
||||
"workspace_roles_pkey",
|
||||
),
|
||||
(
|
||||
"workspace_roles",
|
||||
"search_space_roles_search_space_id_fkey",
|
||||
"workspace_roles_workspace_id_fkey",
|
||||
),
|
||||
(
|
||||
"workspace_roles",
|
||||
"uq_searchspace_role_name",
|
||||
"uq_workspace_role_name",
|
||||
),
|
||||
("workspaces", "searchspaces_pkey", "workspaces_pkey"),
|
||||
("workspaces", "searchspaces_user_id_fkey", "workspaces_user_id_fkey"),
|
||||
]
|
||||
|
||||
# (old_index, new_index) -- plain indexes only; PK/unique-backed indexes follow
|
||||
# their RENAME CONSTRAINT above. ALTER INDEX IF EXISTS covers both create_all
|
||||
# indexes and the runtime ``setup_indexes()`` ones (idx_documents_*).
|
||||
INDEX_RENAMES: list[tuple[str, str]] = [
|
||||
("ix_agent_action_log_search_space_id", "ix_agent_action_log_workspace_id"),
|
||||
(
|
||||
"ix_agent_permission_rules_search_space_id",
|
||||
"ix_agent_permission_rules_workspace_id",
|
||||
),
|
||||
("ix_automations_search_space_id", "ix_automations_workspace_id"),
|
||||
("ix_document_files_search_space_id", "ix_document_files_workspace_id"),
|
||||
("ix_document_revisions_search_space_id", "ix_document_revisions_workspace_id"),
|
||||
(
|
||||
"ix_external_chat_bindings_search_space_state",
|
||||
"ix_external_chat_bindings_workspace_state",
|
||||
),
|
||||
("ix_folder_revisions_search_space_id", "ix_folder_revisions_workspace_id"),
|
||||
("ix_folders_search_space_id", "ix_folders_workspace_id"),
|
||||
("ix_notifications_search_space_id", "ix_notifications_workspace_id"),
|
||||
("ix_notifications_user_space_created", "ix_notifications_user_workspace_created"),
|
||||
("ix_prompts_search_space_id", "ix_prompts_workspace_id"),
|
||||
("ix_token_usage_search_space_id", "ix_token_usage_workspace_id"),
|
||||
("ix_search_space_invites_created_at", "ix_workspace_invites_created_at"),
|
||||
("ix_search_space_invites_id", "ix_workspace_invites_id"),
|
||||
("ix_search_space_invites_invite_code", "ix_workspace_invites_invite_code"),
|
||||
("ix_search_space_memberships_created_at", "ix_workspace_memberships_created_at"),
|
||||
("ix_search_space_memberships_id", "ix_workspace_memberships_id"),
|
||||
("ix_search_space_roles_created_at", "ix_workspace_roles_created_at"),
|
||||
("ix_search_space_roles_id", "ix_workspace_roles_id"),
|
||||
("ix_search_space_roles_name", "ix_workspace_roles_name"),
|
||||
("ix_searchspaces_created_at", "ix_workspaces_created_at"),
|
||||
("ix_searchspaces_id", "ix_workspaces_id"),
|
||||
("ix_searchspaces_name", "ix_workspaces_name"),
|
||||
("idx_documents_search_space_id", "idx_documents_workspace_id"),
|
||||
("idx_documents_search_space_updated", "idx_documents_workspace_updated"),
|
||||
]
|
||||
|
||||
# (old_sequence, new_sequence)
|
||||
SEQUENCE_RENAMES: list[tuple[str, str]] = [
|
||||
("searchspaces_id_seq", "workspaces_id_seq"),
|
||||
("search_space_roles_id_seq", "workspace_roles_id_seq"),
|
||||
("search_space_memberships_id_seq", "workspace_memberships_id_seq"),
|
||||
("search_space_invites_id_seq", "workspace_invites_id_seq"),
|
||||
]
|
||||
|
||||
# ---- Hardcoded OLD-shape publication (for downgrade only; finding 7). ----
|
||||
# Mirrors app.zero_publication.ZERO_PUBLICATION at revision 169 but with the
|
||||
# pre-rename ``search_space_id`` column. NEVER import the live module here: it
|
||||
# now reflects ``workspace_id`` and would silently drop the column-list tables.
|
||||
_DOWNGRADE_DOCUMENT_COLS = [
|
||||
"id",
|
||||
"title",
|
||||
"document_type",
|
||||
"search_space_id",
|
||||
"folder_id",
|
||||
"created_by_id",
|
||||
"status",
|
||||
"created_at",
|
||||
"updated_at",
|
||||
]
|
||||
_DOWNGRADE_USER_COLS = ["id", "credit_micros_balance"]
|
||||
_DOWNGRADE_AUTOMATION_RUN_COLS = [
|
||||
"id",
|
||||
"automation_id",
|
||||
"trigger_id",
|
||||
"status",
|
||||
"step_results",
|
||||
"started_at",
|
||||
"finished_at",
|
||||
"created_at",
|
||||
]
|
||||
_DOWNGRADE_AUTOMATION_COLS = ["id", "search_space_id"]
|
||||
_DOWNGRADE_NEW_CHAT_THREAD_COLS = ["id", "search_space_id"]
|
||||
_DOWNGRADE_PODCAST_COLS = [
|
||||
"id",
|
||||
"title",
|
||||
"status",
|
||||
"spec",
|
||||
"spec_version",
|
||||
"duration_seconds",
|
||||
"error",
|
||||
"search_space_id",
|
||||
"thread_id",
|
||||
"created_at",
|
||||
]
|
||||
# Ordered to match ZERO_PUBLICATION; None == all columns.
|
||||
_DOWNGRADE_PUBLICATION: list[tuple[str, list[str] | None]] = [
|
||||
("notifications", None),
|
||||
("documents", _DOWNGRADE_DOCUMENT_COLS),
|
||||
("folders", None),
|
||||
("search_source_connectors", None),
|
||||
("new_chat_threads", _DOWNGRADE_NEW_CHAT_THREAD_COLS),
|
||||
("new_chat_messages", None),
|
||||
("chat_comments", None),
|
||||
("chat_session_state", None),
|
||||
("user", _DOWNGRADE_USER_COLS),
|
||||
("automations", _DOWNGRADE_AUTOMATION_COLS),
|
||||
("automation_runs", _DOWNGRADE_AUTOMATION_RUN_COLS),
|
||||
("podcasts", _DOWNGRADE_PODCAST_COLS),
|
||||
]
|
||||
|
||||
|
||||
def _quote(identifier: str) -> str:
|
||||
return '"' + identifier.replace('"', '""') + '"'
|
||||
|
||||
|
||||
def _publication_exists(conn) -> bool:
|
||||
return (
|
||||
conn.execute(
|
||||
sa.text("SELECT 1 FROM pg_publication WHERE pubname = :n"),
|
||||
{"n": PUBLICATION_NAME},
|
||||
).fetchone()
|
||||
is not None
|
||||
)
|
||||
|
||||
|
||||
def _is_publication_member(conn, table: str) -> bool:
|
||||
return (
|
||||
conn.execute(
|
||||
sa.text(
|
||||
"SELECT 1 FROM pg_publication_tables "
|
||||
"WHERE pubname = :n AND schemaname = current_schema() "
|
||||
"AND tablename = :t"
|
||||
),
|
||||
{"n": PUBLICATION_NAME, "t": table},
|
||||
).fetchone()
|
||||
is not None
|
||||
)
|
||||
|
||||
|
||||
def _table_columns(conn, table: str) -> set[str]:
|
||||
rows = conn.execute(
|
||||
sa.text(
|
||||
"SELECT column_name FROM information_schema.columns "
|
||||
"WHERE table_schema = current_schema() AND table_name = :t"
|
||||
),
|
||||
{"t": table},
|
||||
).fetchall()
|
||||
return {row[0] for row in rows}
|
||||
|
||||
|
||||
def _neutralize_column_list_tables(conn) -> None:
|
||||
"""Remove the column-list dependency so RENAME COLUMN is permitted."""
|
||||
if not _publication_exists(conn):
|
||||
return
|
||||
for table in PUBLICATION_COLUMN_LIST_TABLES:
|
||||
if _is_publication_member(conn, table):
|
||||
conn.execute(
|
||||
sa.text(
|
||||
f"ALTER PUBLICATION {_quote(PUBLICATION_NAME)} DROP TABLE {table}"
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _rename_table(conn, old: str, new: str) -> None:
|
||||
conn.execute(sa.text(f"ALTER TABLE IF EXISTS {old} RENAME TO {new}"))
|
||||
|
||||
|
||||
def _rename_column(conn, table: str, old: str, new: str) -> None:
|
||||
conn.execute(
|
||||
sa.text(
|
||||
f"""
|
||||
DO $$
|
||||
BEGIN
|
||||
IF EXISTS (
|
||||
SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema()
|
||||
AND table_name = '{table}' AND column_name = '{old}'
|
||||
) THEN
|
||||
ALTER TABLE {table} RENAME COLUMN {old} TO {new};
|
||||
END IF;
|
||||
END$$;
|
||||
"""
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _rename_constraint(conn, table: str, old: str, new: str) -> None:
|
||||
conn.execute(
|
||||
sa.text(
|
||||
f"""
|
||||
DO $$
|
||||
BEGIN
|
||||
IF EXISTS (
|
||||
SELECT 1 FROM pg_constraint
|
||||
WHERE conname = '{old}'
|
||||
AND conrelid = to_regclass('{table}')
|
||||
) THEN
|
||||
ALTER TABLE {table} RENAME CONSTRAINT {old} TO {new};
|
||||
END IF;
|
||||
END$$;
|
||||
"""
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _rename_index(conn, old: str, new: str) -> None:
|
||||
conn.execute(sa.text(f"ALTER INDEX IF EXISTS {old} RENAME TO {new}"))
|
||||
|
||||
|
||||
def _rename_sequence(conn, old: str, new: str) -> None:
|
||||
conn.execute(sa.text(f"ALTER SEQUENCE IF EXISTS {old} RENAME TO {new}"))
|
||||
|
||||
|
||||
def _restore_downgrade_publication(conn) -> None:
|
||||
"""Re-emit the OLD (search_space_id) publication shape via plain SET TABLE.
|
||||
|
||||
Does NOT call apply_publication (which reads the live, now-workspace_id
|
||||
module) -- that would silently drop the column-list tables (finding 7).
|
||||
"""
|
||||
if not _publication_exists(conn):
|
||||
return
|
||||
# Drop the column-list tables first so the SET TABLE has no stale
|
||||
# workspace_id dependency to trip over.
|
||||
for table in PUBLICATION_COLUMN_LIST_TABLES:
|
||||
if _is_publication_member(conn, table):
|
||||
conn.execute(
|
||||
sa.text(
|
||||
f"ALTER PUBLICATION {_quote(PUBLICATION_NAME)} DROP TABLE {table}"
|
||||
)
|
||||
)
|
||||
|
||||
entries: list[str] = []
|
||||
for table, cols in _DOWNGRADE_PUBLICATION:
|
||||
actual = _table_columns(conn, table)
|
||||
if not actual:
|
||||
continue
|
||||
if cols is None:
|
||||
entries.append(_quote(table))
|
||||
continue
|
||||
expected = list(cols)
|
||||
if table in {"documents", "user", "podcasts"} and "_0_version" in actual:
|
||||
expected.append("_0_version")
|
||||
if any(col not in actual for col in expected):
|
||||
continue
|
||||
col_sql = ", ".join(_quote(col) for col in expected)
|
||||
entries.append(f"{_quote(table)} ({col_sql})")
|
||||
|
||||
table_list = ", ".join(entries)
|
||||
conn.execute(
|
||||
sa.text(f"ALTER PUBLICATION {_quote(PUBLICATION_NAME)} SET TABLE {table_list}")
|
||||
)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
conn = op.get_bind()
|
||||
|
||||
_neutralize_column_list_tables(conn)
|
||||
|
||||
for old, new in TABLE_RENAMES:
|
||||
_rename_table(conn, old, new)
|
||||
for table, old, new in COLUMN_RENAMES:
|
||||
_rename_column(conn, table, old, new)
|
||||
for table, old, new in CONSTRAINT_RENAMES:
|
||||
_rename_constraint(conn, table, old, new)
|
||||
for old, new in INDEX_RENAMES:
|
||||
_rename_index(conn, old, new)
|
||||
for old, new in SEQUENCE_RENAMES:
|
||||
_rename_sequence(conn, old, new)
|
||||
|
||||
# Reconcile to the new workspace_id shape via the blessed apply_publication
|
||||
# path (ALTER ... SET TABLE) -- never raw DROP/CREATE PUBLICATION (bug #1355,
|
||||
# migration 116). No-op if the publication does not exist.
|
||||
apply_publication(conn)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
conn = op.get_bind()
|
||||
|
||||
_neutralize_column_list_tables(conn)
|
||||
|
||||
for old, new in SEQUENCE_RENAMES:
|
||||
_rename_sequence(conn, new, old)
|
||||
for old, new in INDEX_RENAMES:
|
||||
_rename_index(conn, new, old)
|
||||
for table, old, new in CONSTRAINT_RENAMES:
|
||||
# table here is the POST-rename name; constraints/tables are still
|
||||
# renamed at this point (table rename is reversed last).
|
||||
_rename_constraint(conn, table, new, old)
|
||||
for table, old, new in COLUMN_RENAMES:
|
||||
_rename_column(conn, table, new, old)
|
||||
for old, new in TABLE_RENAMES:
|
||||
_rename_table(conn, new, old)
|
||||
|
||||
_restore_downgrade_publication(conn)
|
||||
|
|
@ -0,0 +1,78 @@
|
|||
"""Add runs and tool_output_spills tables.
|
||||
|
||||
``runs`` backs the user-facing Scraper-API logs and the agent's tool-boundary
|
||||
truncation (full scraper output stored here, model sees a capped preview + id).
|
||||
``tool_output_spills`` is the internal scratch store for main-agent context
|
||||
spills, kept separate so the customer log stays clean.
|
||||
|
||||
Revision ID: 171
|
||||
Revises: 170
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision: str = "171"
|
||||
down_revision: str | None = "170"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS runs (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id INTEGER NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE,
|
||||
user_id UUID REFERENCES "user"(id) ON DELETE CASCADE,
|
||||
thread_id VARCHAR(255),
|
||||
capability VARCHAR(100) NOT NULL,
|
||||
origin VARCHAR(16) NOT NULL,
|
||||
status VARCHAR(16) NOT NULL,
|
||||
error TEXT,
|
||||
input JSONB,
|
||||
output_text TEXT,
|
||||
item_count INTEGER NOT NULL DEFAULT 0,
|
||||
char_count INTEGER NOT NULL DEFAULT 0,
|
||||
duration_ms INTEGER,
|
||||
cost_micros BIGINT,
|
||||
created_at TIMESTAMP WITH TIME ZONE NOT NULL
|
||||
);
|
||||
"""
|
||||
)
|
||||
op.execute("CREATE INDEX IF NOT EXISTS ix_runs_workspace_id ON runs (workspace_id)")
|
||||
op.execute("CREATE INDEX IF NOT EXISTS ix_runs_user_id ON runs (user_id)")
|
||||
op.execute("CREATE INDEX IF NOT EXISTS ix_runs_capability ON runs (capability)")
|
||||
op.execute("CREATE INDEX IF NOT EXISTS ix_runs_created_at ON runs (created_at)")
|
||||
op.execute(
|
||||
"CREATE INDEX IF NOT EXISTS ix_runs_workspace_created "
|
||||
"ON runs (workspace_id, created_at)"
|
||||
)
|
||||
|
||||
op.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS tool_output_spills (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
workspace_id INTEGER REFERENCES workspaces(id) ON DELETE CASCADE,
|
||||
thread_id VARCHAR(255),
|
||||
tool_name VARCHAR(255),
|
||||
content TEXT NOT NULL,
|
||||
char_count INTEGER NOT NULL DEFAULT 0,
|
||||
created_at TIMESTAMP WITH TIME ZONE NOT NULL
|
||||
);
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
"CREATE INDEX IF NOT EXISTS ix_tool_output_spills_workspace_id "
|
||||
"ON tool_output_spills (workspace_id)"
|
||||
)
|
||||
op.execute(
|
||||
"CREATE INDEX IF NOT EXISTS ix_tool_output_spills_created_at "
|
||||
"ON tool_output_spills (created_at)"
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.execute("DROP TABLE IF EXISTS tool_output_spills")
|
||||
op.execute("DROP TABLE IF EXISTS runs")
|
||||
|
|
@ -0,0 +1,32 @@
|
|||
"""Remove AI file sort flag.
|
||||
|
||||
Revision ID: 172
|
||||
Revises: 171
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision: str = "172"
|
||||
down_revision: str | None = "171"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.drop_column("workspaces", "ai_file_sort_enabled")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.add_column(
|
||||
"workspaces",
|
||||
sa.Column(
|
||||
"ai_file_sort_enabled",
|
||||
sa.Boolean(),
|
||||
nullable=False,
|
||||
server_default=sa.text("false"),
|
||||
),
|
||||
)
|
||||
26
surfsense_backend/alembic/versions/173_add_runs_progress.py
Normal file
26
surfsense_backend/alembic/versions/173_add_runs_progress.py
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
"""Add runs.progress for streamed scraper run progress.
|
||||
|
||||
Stores the coarse (throttled) progress log captured during a run; the live
|
||||
fine-grained stream stays ephemeral (in-process bus / SSE only). Nullable so
|
||||
existing rows and the sync/agent doors that don't report progress are unaffected.
|
||||
|
||||
Revision ID: 173
|
||||
Revises: 172
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision: str = "173"
|
||||
down_revision: str | None = "172"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.execute("ALTER TABLE runs ADD COLUMN IF NOT EXISTS progress JSONB")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.execute("ALTER TABLE runs DROP COLUMN IF EXISTS progress")
|
||||
|
|
@ -1,16 +1,16 @@
|
|||
"""Minimal anonymous / free-chat agent.
|
||||
|
||||
The no-login chat experience must stay dead simple: the user asks a question
|
||||
and the model answers, optionally using ``web_search`` and an optionally
|
||||
uploaded **read-only** document. We deliberately bypass the full SurfSense deep
|
||||
agent stack (filesystem, file-intent, knowledge-base persistence, subagents,
|
||||
skills, memory) because those middlewares stage or persist "documents" that an
|
||||
anonymous session can never see again -- which produced phantom
|
||||
"I saved it to a file" answers for free users.
|
||||
and the model answers over an optionally uploaded **read-only** document. We
|
||||
deliberately bypass the full SurfSense deep agent stack (filesystem,
|
||||
file-intent, knowledge-base persistence, subagents, skills, memory) because
|
||||
those middlewares stage or persist "documents" that an anonymous session can
|
||||
never see again -- which produced phantom "I saved it to a file" answers for
|
||||
free users.
|
||||
|
||||
For any other SurfSense capability the model is instructed (via the system
|
||||
prompt built here) to tell the user to create a free account instead of
|
||||
pretending to perform the action.
|
||||
For any other SurfSense capability (including web search) the model is
|
||||
instructed (via the system prompt built here) to tell the user to create a
|
||||
free account instead of pretending to perform the action.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -22,7 +22,6 @@ from deepagents.backends import StateBackend
|
|||
from langchain.agents import create_agent
|
||||
from langchain.agents.middleware import (
|
||||
ModelCallLimitMiddleware,
|
||||
ToolCallLimitMiddleware,
|
||||
)
|
||||
from langchain_core.language_models import BaseChatModel
|
||||
from langgraph.types import Checkpointer
|
||||
|
|
@ -32,7 +31,6 @@ from app.agents.chat.shared.middleware import (
|
|||
RetryAfterMiddleware,
|
||||
create_surfsense_compaction_middleware,
|
||||
)
|
||||
from app.agents.chat.shared.tools.web_search import create_web_search_tool
|
||||
|
||||
# Cap how much of an uploaded document we inline into the system prompt. The
|
||||
# upload endpoint allows files up to several MB, but the doc is re-sent on
|
||||
|
|
@ -43,9 +41,12 @@ _MAX_DOC_CHARS = 50_000
|
|||
def build_anonymous_system_prompt(anon_doc: dict[str, Any] | None = None) -> str:
|
||||
"""Build the system prompt for the minimal anonymous chat agent.
|
||||
|
||||
The prompt keeps the assistant focused on plain Q/A + web search, inlines
|
||||
any uploaded document as read-only context, and redirects every other
|
||||
SurfSense feature to account registration.
|
||||
The prompt keeps the assistant focused on plain Q/A from model knowledge,
|
||||
inlines any uploaded document as read-only context, and treats the chat as
|
||||
a registration funnel: every other SurfSense capability (scraping, live
|
||||
data, deliverables, knowledge base, automations) redirects to sign-up, and
|
||||
the assistant softly suggests an account when the conversation reveals a
|
||||
competitive-intelligence need the platform serves.
|
||||
"""
|
||||
today = datetime.now(UTC).strftime("%A, %B %d, %Y")
|
||||
|
||||
|
|
@ -72,30 +73,56 @@ def build_anonymous_system_prompt(anon_doc: dict[str, Any] | None = None) -> str
|
|||
|
||||
return (
|
||||
"You are SurfSense's free AI assistant, available to everyone without "
|
||||
"login.\n\n"
|
||||
"login. SurfSense is the open-source competitive intelligence platform: "
|
||||
"registered users get specialist agents that pull live market data from "
|
||||
"Reddit, YouTube, Google Maps, Google Search, and the open web, turn it "
|
||||
"into cited briefs, reports, podcasts, and presentations, keep findings "
|
||||
"in a searchable knowledge base, and run scheduled monitoring "
|
||||
"automations — plus a REST scraping API and MCP server for their own "
|
||||
"agents.\n\n"
|
||||
f"Today's date is {today}.\n\n"
|
||||
"## How to help\n"
|
||||
"- Answer the user's questions directly and conversationally. You are "
|
||||
"a straightforward question-and-answer assistant.\n"
|
||||
"- When a question needs current, real-time, or factual information "
|
||||
"from the internet (news, prices, weather, recent events, live data), "
|
||||
"use the `web_search` tool. Otherwise, answer directly from your own "
|
||||
"knowledge.\n"
|
||||
"- Answer from your own knowledge. You do NOT have web access here, so "
|
||||
"for current, real-time, or fast-changing facts (news, prices, "
|
||||
"weather, recent events, live data) say you can't look them up in the "
|
||||
"free experience and may be out of date.\n"
|
||||
"- Be concise, accurate, and helpful. Use Markdown formatting when it "
|
||||
"improves readability."
|
||||
f"{doc_section}\n\n"
|
||||
"## What is not available here\n"
|
||||
"This is the free, no-login experience. You CANNOT save files or "
|
||||
"notes, generate reports, podcasts, resumes, presentations, or images, "
|
||||
"search or build a knowledge base, connect to apps (Gmail, Google "
|
||||
"Drive, Notion, Slack, Calendar, Discord, and similar), set up "
|
||||
"automations, or remember anything across sessions.\n\n"
|
||||
"This is the free, no-login experience. You CANNOT search the web or "
|
||||
"scrape any platform (Reddit, YouTube, Google Maps, Google Search, "
|
||||
"websites), save files or notes, upload additional files, generate "
|
||||
"reports, podcasts, resumes, presentations, or images, search or build "
|
||||
"a knowledge base, connect to apps (Gmail, Google Drive, Notion, "
|
||||
"Slack, Calendar, Discord, and similar), set up automations or "
|
||||
"monitoring, use the scraping API or MCP server, or remember anything "
|
||||
"across sessions.\n\n"
|
||||
"If the user asks for any of these, do NOT pretend to do them and "
|
||||
"never claim you saved, created, or stored anything. Instead, briefly "
|
||||
"let them know the feature requires a free SurfSense account and "
|
||||
"invite them to create one at https://www.surfsense.com. Then offer to "
|
||||
"help with what you can do here (answering questions and searching the "
|
||||
"web)."
|
||||
"never claim you saved, created, scraped, or stored anything. Instead, "
|
||||
"briefly say that this needs a free SurfSense account and invite them "
|
||||
"to sign up at https://www.surfsense.com/register — name the specific "
|
||||
"capability that would solve their request (e.g. 'a SurfSense account "
|
||||
"can scrape those Reddit threads live and turn them into a cited "
|
||||
"brief'). Then offer to help with what you can do here (answering from "
|
||||
"your own knowledge and about any uploaded document).\n\n"
|
||||
"## Suggesting SurfSense when it genuinely fits\n"
|
||||
"You are also the front door to SurfSense. When the conversation "
|
||||
"reveals a need the full platform serves — researching competitors, "
|
||||
"tracking pricing or rankings, monitoring brand mentions or reviews, "
|
||||
"gauging Reddit/YouTube sentiment, generating leads, needing current "
|
||||
"web data, or wanting recurring reports — first answer as well as you "
|
||||
"can from your own knowledge, then add ONE short sentence pointing out "
|
||||
"that a free SurfSense account can do that with live data, linking "
|
||||
"https://www.surfsense.com/register.\n"
|
||||
"- Be helpful first, never salesy: the answer is the product; the "
|
||||
"suggestion is a footnote.\n"
|
||||
"- At most one suggestion per response, and stop suggesting entirely "
|
||||
"if the user declines or ignores it.\n"
|
||||
"- Do not suggest it for needs SurfSense does not serve (casual chat, "
|
||||
"coding help, homework, creative writing)."
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -105,14 +132,13 @@ async def create_anonymous_chat_agent(
|
|||
checkpointer: Checkpointer,
|
||||
anon_session_id: str | None = None,
|
||||
anon_doc: dict[str, Any] | None = None,
|
||||
enable_web_search: bool = True,
|
||||
):
|
||||
"""Create a minimal Q/A agent for anonymous / free chat.
|
||||
|
||||
Unlike :func:`create_surfsense_deep_agent`, this agent has no filesystem,
|
||||
file-intent, knowledge-base persistence, subagent, skills, or memory
|
||||
middleware. Its only tool is ``web_search`` (when ``enable_web_search`` is
|
||||
True), and any uploaded document is injected into the system prompt as
|
||||
middleware -- and no tools at all. It answers purely from the model's own
|
||||
knowledge; any uploaded document is injected into the system prompt as
|
||||
read-only context.
|
||||
|
||||
Args:
|
||||
|
|
@ -120,36 +146,23 @@ async def create_anonymous_chat_agent(
|
|||
checkpointer: LangGraph checkpointer for the ephemeral anon thread.
|
||||
anon_session_id: Anonymous session id (used only for telemetry/metadata).
|
||||
anon_doc: Optional ``{"title", "content"}`` for an uploaded document.
|
||||
enable_web_search: When False, the agent runs as a pure LLM with no
|
||||
tools (used when the user toggles web search off).
|
||||
"""
|
||||
tools = (
|
||||
[create_web_search_tool(search_space_id=None, available_connectors=None)]
|
||||
if enable_web_search
|
||||
else []
|
||||
)
|
||||
|
||||
# Reliability-only middleware. Nothing here touches the database or
|
||||
# filesystem: call limits guard against loops, compaction summarises long
|
||||
# histories into in-graph state, and retry handles provider rate limits.
|
||||
# filesystem: the call limit guards against loops, compaction summarises
|
||||
# long histories into in-graph state, and retry handles provider rate
|
||||
# limits.
|
||||
middleware: list[Any] = [
|
||||
ModelCallLimitMiddleware(thread_limit=120, run_limit=80, exit_behavior="end"),
|
||||
create_surfsense_compaction_middleware(llm, StateBackend),
|
||||
RetryAfterMiddleware(max_retries=3),
|
||||
]
|
||||
if tools:
|
||||
middleware.append(
|
||||
ToolCallLimitMiddleware(
|
||||
thread_limit=300, run_limit=80, exit_behavior="continue"
|
||||
)
|
||||
)
|
||||
middleware.append(create_surfsense_compaction_middleware(llm, StateBackend))
|
||||
middleware.append(RetryAfterMiddleware(max_retries=3))
|
||||
|
||||
system_prompt = build_anonymous_system_prompt(anon_doc)
|
||||
|
||||
agent = create_agent(
|
||||
llm,
|
||||
system_prompt=system_prompt,
|
||||
tools=tools,
|
||||
tools=[],
|
||||
middleware=middleware,
|
||||
context_schema=SurfSenseContextSchema,
|
||||
checkpointer=checkpointer,
|
||||
|
|
|
|||
|
|
@ -2,43 +2,74 @@
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
# Connected apps (hosted MCP + interim-native Gmail/Calendar) all route to the
|
||||
# single ``mcp_discovery`` subagent. File connectors stay native (they enrich
|
||||
# the knowledge base). Deprecated connectors (Discord/Teams/Luma) are omitted:
|
||||
# they have no agent subagent, so their rows produce no tools.
|
||||
CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS: dict[str, str] = {
|
||||
"GOOGLE_GMAIL_CONNECTOR": "gmail",
|
||||
"COMPOSIO_GMAIL_CONNECTOR": "gmail",
|
||||
"GOOGLE_CALENDAR_CONNECTOR": "calendar",
|
||||
"COMPOSIO_GOOGLE_CALENDAR_CONNECTOR": "calendar",
|
||||
"DISCORD_CONNECTOR": "discord",
|
||||
"TEAMS_CONNECTOR": "teams",
|
||||
"LUMA_CONNECTOR": "luma",
|
||||
"LINEAR_CONNECTOR": "linear",
|
||||
"JIRA_CONNECTOR": "jira",
|
||||
"CLICKUP_CONNECTOR": "clickup",
|
||||
"SLACK_CONNECTOR": "slack",
|
||||
"AIRTABLE_CONNECTOR": "airtable",
|
||||
"NOTION_CONNECTOR": "notion",
|
||||
"CONFLUENCE_CONNECTOR": "confluence",
|
||||
"GOOGLE_GMAIL_CONNECTOR": "mcp_discovery",
|
||||
"COMPOSIO_GMAIL_CONNECTOR": "mcp_discovery",
|
||||
"GOOGLE_CALENDAR_CONNECTOR": "mcp_discovery",
|
||||
"COMPOSIO_GOOGLE_CALENDAR_CONNECTOR": "mcp_discovery",
|
||||
"LINEAR_CONNECTOR": "mcp_discovery",
|
||||
"JIRA_CONNECTOR": "mcp_discovery",
|
||||
"CLICKUP_CONNECTOR": "mcp_discovery",
|
||||
"SLACK_CONNECTOR": "mcp_discovery",
|
||||
"AIRTABLE_CONNECTOR": "mcp_discovery",
|
||||
"NOTION_CONNECTOR": "mcp_discovery",
|
||||
"CONFLUENCE_CONNECTOR": "mcp_discovery",
|
||||
"MCP_CONNECTOR": "mcp_discovery",
|
||||
"GOOGLE_DRIVE_CONNECTOR": "google_drive",
|
||||
"COMPOSIO_GOOGLE_DRIVE_CONNECTOR": "google_drive",
|
||||
"DROPBOX_CONNECTOR": "dropbox",
|
||||
"ONEDRIVE_CONNECTOR": "onedrive",
|
||||
}
|
||||
|
||||
# ``mcp_discovery`` is gated any-of: present iff the workspace has at least one
|
||||
# connected app. Tokens are searchable-type strings (Composio Gmail/Calendar
|
||||
# map to the GOOGLE_* tokens in connector_searchable_types).
|
||||
SUBAGENT_TO_REQUIRED_CONNECTOR_MAP: dict[str, frozenset[str]] = {
|
||||
"deliverables": frozenset(),
|
||||
"knowledge_base": frozenset(),
|
||||
"airtable": frozenset({"AIRTABLE_CONNECTOR"}),
|
||||
"calendar": frozenset({"GOOGLE_CALENDAR_CONNECTOR"}),
|
||||
"clickup": frozenset({"CLICKUP_CONNECTOR"}),
|
||||
"confluence": frozenset({"CONFLUENCE_CONNECTOR"}),
|
||||
"discord": frozenset({"DISCORD_CONNECTOR"}),
|
||||
"web_crawler": frozenset(),
|
||||
"youtube": frozenset(),
|
||||
"google_maps": frozenset(),
|
||||
"google_search": frozenset(),
|
||||
"reddit": frozenset(),
|
||||
"mcp_discovery": frozenset(
|
||||
{
|
||||
"SLACK_CONNECTOR",
|
||||
"JIRA_CONNECTOR",
|
||||
"LINEAR_CONNECTOR",
|
||||
"CLICKUP_CONNECTOR",
|
||||
"AIRTABLE_CONNECTOR",
|
||||
"NOTION_CONNECTOR",
|
||||
"CONFLUENCE_CONNECTOR",
|
||||
"GOOGLE_GMAIL_CONNECTOR",
|
||||
"GOOGLE_CALENDAR_CONNECTOR",
|
||||
"MCP_CONNECTOR",
|
||||
}
|
||||
),
|
||||
"dropbox": frozenset({"DROPBOX_FILE"}),
|
||||
"gmail": frozenset({"GOOGLE_GMAIL_CONNECTOR"}),
|
||||
"google_drive": frozenset({"GOOGLE_DRIVE_FILE"}),
|
||||
"jira": frozenset({"JIRA_CONNECTOR"}),
|
||||
"linear": frozenset({"LINEAR_CONNECTOR"}),
|
||||
"luma": frozenset({"LUMA_CONNECTOR"}),
|
||||
"notion": frozenset({"NOTION_CONNECTOR"}),
|
||||
"onedrive": frozenset({"ONEDRIVE_FILE"}),
|
||||
"slack": frozenset({"SLACK_CONNECTOR"}),
|
||||
"teams": frozenset({"TEAMS_CONNECTOR"}),
|
||||
}
|
||||
|
||||
# Old per-connector subagent names, kept working for checkpoint resume: a
|
||||
# ``task(subagent_type="gmail")`` paused before the MCP consolidation resolves
|
||||
# to the consolidated ``mcp_discovery`` subagent instead of hard-failing
|
||||
# "subagent does not exist". New routing never emits these names.
|
||||
LEGACY_SUBAGENT_ALIASES: dict[str, str] = {
|
||||
"gmail": "mcp_discovery",
|
||||
"calendar": "mcp_discovery",
|
||||
"linear": "mcp_discovery",
|
||||
"jira": "mcp_discovery",
|
||||
"clickup": "mcp_discovery",
|
||||
"slack": "mcp_discovery",
|
||||
"airtable": "mcp_discovery",
|
||||
"notion": "mcp_discovery",
|
||||
"confluence": "mcp_discovery",
|
||||
"discord": "mcp_discovery",
|
||||
"teams": "mcp_discovery",
|
||||
"luma": "mcp_discovery",
|
||||
}
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ def build_compiled_agent_graph_sync(
|
|||
final_system_prompt: str,
|
||||
backend_resolver: Any,
|
||||
filesystem_mode: FilesystemMode,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
user_id: str | None,
|
||||
thread_id: int | None,
|
||||
visibility: ChatVisibility,
|
||||
|
|
@ -53,7 +53,7 @@ def build_compiled_agent_graph_sync(
|
|||
tools=tools,
|
||||
backend_resolver=backend_resolver,
|
||||
filesystem_mode=filesystem_mode,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
user_id=user_id,
|
||||
thread_id=thread_id,
|
||||
visibility=visibility,
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ def build_action_log_mw(
|
|||
*,
|
||||
flags: AgentFeatureFlags,
|
||||
thread_id: int | None,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
user_id: str | None,
|
||||
) -> ActionLogMiddleware | None:
|
||||
if not enabled(flags, "enable_action_log") or thread_id is None:
|
||||
|
|
@ -25,7 +25,7 @@ def build_action_log_mw(
|
|||
# tool via ``ToolDefinition.reverse`` and can be wired here when used.
|
||||
return ActionLogMiddleware(
|
||||
thread_id=thread_id,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
user_id=user_id,
|
||||
)
|
||||
except Exception: # pragma: no cover - defensive
|
||||
|
|
|
|||
|
|
@ -84,7 +84,7 @@ class ActionLogMiddleware(AgentMiddleware):
|
|||
Args:
|
||||
thread_id: The current chat thread's primary-key id. Required to
|
||||
persist a row; if ``None`` the middleware silently no-ops.
|
||||
search_space_id: Search-space id for cascade-on-delete safety.
|
||||
workspace_id: Workspace id for cascade-on-delete safety.
|
||||
user_id: UUID string of the user driving this turn (nullable in
|
||||
anonymous mode).
|
||||
tool_definitions: Optional mapping of tool name -> :class:`ToolDefinition`
|
||||
|
|
@ -98,13 +98,13 @@ class ActionLogMiddleware(AgentMiddleware):
|
|||
self,
|
||||
*,
|
||||
thread_id: int | None,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
user_id: str | None,
|
||||
tool_definitions: dict[str, ToolDefinition] | None = None,
|
||||
) -> None:
|
||||
super().__init__()
|
||||
self._thread_id = thread_id
|
||||
self._search_space_id = search_space_id
|
||||
self._workspace_id = workspace_id
|
||||
self._user_id = user_id
|
||||
self._tool_definitions = dict(tool_definitions or {})
|
||||
|
||||
|
|
@ -187,7 +187,7 @@ class ActionLogMiddleware(AgentMiddleware):
|
|||
row = AgentActionLog(
|
||||
thread_id=thread_id,
|
||||
user_id=self._user_id,
|
||||
search_space_id=self._search_space_id,
|
||||
workspace_id=self._workspace_id,
|
||||
# ``turn_id`` is the deprecated alias of ``tool_call_id``
|
||||
# kept for one release for safe rollback. New consumers
|
||||
# should read ``tool_call_id`` directly.
|
||||
|
|
|
|||
|
|
@ -40,7 +40,7 @@ class SurfSenseCheckpointedSubAgentMiddleware(SubAgentMiddleware):
|
|||
subagents: list[SubAgent | CompiledSubAgent],
|
||||
system_prompt: str | None = TASK_SYSTEM_PROMPT,
|
||||
task_description: str | None = None,
|
||||
search_space_id: int | None = None,
|
||||
workspace_id: int | None = None,
|
||||
) -> None:
|
||||
self._surf_checkpointer = checkpointer
|
||||
super(SubAgentMiddleware, self).__init__()
|
||||
|
|
@ -50,11 +50,11 @@ class SurfSenseCheckpointedSubAgentMiddleware(SubAgentMiddleware):
|
|||
)
|
||||
self._backend = backend
|
||||
self._subagents = subagents
|
||||
# Search-space id is captured at build time (the orchestrator runs in
|
||||
# exactly one search space for its lifetime). The spawn-paused kill
|
||||
# Workspace id is captured at build time (the orchestrator runs in
|
||||
# exactly one workspace for its lifetime). The spawn-paused kill
|
||||
# switch keys on it so an operator can quarantine one workspace
|
||||
# without affecting the rest of the deployment.
|
||||
self._search_space_id = search_space_id
|
||||
self._workspace_id = workspace_id
|
||||
|
||||
# Lazy subagent compilation. Compiling a subagent graph via
|
||||
# ``create_agent`` is expensive (~250-400ms each) and there can be up
|
||||
|
|
@ -75,7 +75,7 @@ class SurfSenseCheckpointedSubAgentMiddleware(SubAgentMiddleware):
|
|||
task_tool = build_task_tool_with_parent_config(
|
||||
descriptors,
|
||||
task_description,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
resolve_subagent=self._resolve_subagent,
|
||||
)
|
||||
if system_prompt and descriptors:
|
||||
|
|
|
|||
|
|
@ -1,12 +1,12 @@
|
|||
"""Per-search-space spawn-paused kill switch for the ``task`` boundary.
|
||||
"""Per-workspace spawn-paused kill switch for the ``task`` boundary.
|
||||
|
||||
When operators see a runaway loop, a vendor outage, or a billing event
|
||||
that requires immediate cessation of subagent traffic for a specific
|
||||
workspace, they flip a Redis flag and the ``task`` tool short-circuits
|
||||
without touching downstream services. The flag is **per-search-space**
|
||||
without touching downstream services. The flag is **per-workspace**
|
||||
so one tenant's incident never silences the rest of the deployment.
|
||||
|
||||
Flag key: ``surfsense:spawn_paused:{search_space_id}``
|
||||
Flag key: ``surfsense:spawn_paused:{workspace_id}``
|
||||
Flag value: any string-truthy value (we read presence, not contents).
|
||||
TTL: set by whoever toggles the flag — this module never expires
|
||||
keys on its own, since "the flag is on" is itself the signal
|
||||
|
|
@ -43,18 +43,18 @@ _DISABLED = os.environ.get(
|
|||
}
|
||||
|
||||
|
||||
def _flag_key(search_space_id: int) -> str:
|
||||
return f"surfsense:spawn_paused:{search_space_id}"
|
||||
def _flag_key(workspace_id: int) -> str:
|
||||
return f"surfsense:spawn_paused:{workspace_id}"
|
||||
|
||||
|
||||
async def is_spawn_paused(search_space_id: int | None) -> bool:
|
||||
async def is_spawn_paused(workspace_id: int | None) -> bool:
|
||||
"""Return ``True`` iff the workspace's spawn-paused flag is set in Redis.
|
||||
|
||||
A ``None`` search-space (e.g. dev paths that did not plumb the id
|
||||
A ``None`` workspace (e.g. dev paths that did not plumb the id
|
||||
through yet) bypasses the check. So does a Redis outage — see module
|
||||
docstring for the fail-open rationale.
|
||||
"""
|
||||
if _DISABLED or search_space_id is None:
|
||||
if _DISABLED or workspace_id is None:
|
||||
return False
|
||||
try:
|
||||
# Local import keeps the cold-path import cheap and lets routes
|
||||
|
|
@ -63,7 +63,7 @@ async def is_spawn_paused(search_space_id: int | None) -> bool:
|
|||
|
||||
client = aioredis.from_url(config.REDIS_APP_URL, decode_responses=True)
|
||||
try:
|
||||
raw = await client.get(_flag_key(search_space_id))
|
||||
raw = await client.get(_flag_key(workspace_id))
|
||||
finally:
|
||||
# ``aclose()`` is the async-safe variant on redis-py >=5; fall back
|
||||
# to ``close()`` for older clients pinned in tests.
|
||||
|
|
@ -74,8 +74,8 @@ async def is_spawn_paused(search_space_id: int | None) -> bool:
|
|||
return bool(raw)
|
||||
except Exception:
|
||||
logger.warning(
|
||||
"spawn_paused check failed for search_space_id=%s; failing open.",
|
||||
search_space_id,
|
||||
"spawn_paused check failed for workspace_id=%s; failing open.",
|
||||
workspace_id,
|
||||
exc_info=True,
|
||||
)
|
||||
return False
|
||||
|
|
|
|||
|
|
@ -23,6 +23,7 @@ from langchain_core.tools import StructuredTool
|
|||
from langgraph.errors import GraphInterrupt
|
||||
from langgraph.types import Command, Interrupt
|
||||
|
||||
from app.agents.chat.multi_agent_chat.constants import LEGACY_SUBAGENT_ALIASES
|
||||
from app.agents.chat.multi_agent_chat.subagents.shared.invocation import (
|
||||
EXCLUDED_STATE_KEYS,
|
||||
subagent_invoke_config,
|
||||
|
|
@ -142,7 +143,7 @@ def build_task_tool_with_parent_config(
|
|||
subagents: list[dict[str, Any]],
|
||||
task_description: str | None = None,
|
||||
*,
|
||||
search_space_id: int | None = None,
|
||||
workspace_id: int | None = None,
|
||||
resolve_subagent: Callable[[str], Runnable] | None = None,
|
||||
) -> BaseTool:
|
||||
"""Upstream ``_build_task_tool`` + parent ``runtime.config`` propagation + resume bridging.
|
||||
|
|
@ -157,6 +158,26 @@ def build_task_tool_with_parent_config(
|
|||
case a trivial dict-backed resolver is used.
|
||||
"""
|
||||
subagent_names: set[str] = {spec["name"] for spec in subagents}
|
||||
|
||||
def _canonical_subagent_type(subagent_type: str) -> str:
|
||||
"""Resolve a legacy connector subagent name to its consolidated route.
|
||||
|
||||
Only rewrites when the requested name is unavailable but its alias is
|
||||
(checkpoint resume of a pre-consolidation ``task(...)`` call). Current
|
||||
routing never emits legacy names, so live traffic is untouched.
|
||||
"""
|
||||
if subagent_type in subagent_names:
|
||||
return subagent_type
|
||||
alias = LEGACY_SUBAGENT_ALIASES.get(subagent_type)
|
||||
if alias is not None and alias in subagent_names:
|
||||
logger.info(
|
||||
"[hitl_route] aliasing legacy subagent %r -> %r",
|
||||
subagent_type,
|
||||
alias,
|
||||
)
|
||||
return alias
|
||||
return subagent_type
|
||||
|
||||
if resolve_subagent is None:
|
||||
_eager_graphs: dict[str, Runnable] = {
|
||||
spec["name"]: spec["runnable"] for spec in subagents if "runnable" in spec
|
||||
|
|
@ -482,6 +503,7 @@ def build_task_tool_with_parent_config(
|
|||
batched HITL is intentionally out of scope.
|
||||
"""
|
||||
async with semaphore:
|
||||
subagent_type = _canonical_subagent_type(subagent_type)
|
||||
if subagent_type not in subagent_names:
|
||||
allowed_types = ", ".join([f"`{k}`" for k in subagent_names])
|
||||
return (
|
||||
|
|
@ -658,6 +680,7 @@ def build_task_tool_with_parent_config(
|
|||
"task: must provide either single-mode (`description`+`subagent_type`) "
|
||||
"or batch-mode (`tasks`)."
|
||||
)
|
||||
subagent_type = _canonical_subagent_type(subagent_type)
|
||||
if subagent_type not in subagent_names:
|
||||
allowed_types = ", ".join([f"`{k}`" for k in subagent_names])
|
||||
return (
|
||||
|
|
@ -829,10 +852,10 @@ def build_task_tool_with_parent_config(
|
|||
atask_start = time.perf_counter()
|
||||
# Ops kill switch: short-circuit every task() call for this workspace
|
||||
# so the orchestrator stops hammering downstream APIs.
|
||||
if await is_spawn_paused(search_space_id):
|
||||
if await is_spawn_paused(workspace_id):
|
||||
logger.warning(
|
||||
"[hitl_route] atask SPAWN_PAUSED: search_space_id=%s tool_call_id=%s",
|
||||
search_space_id,
|
||||
"[hitl_route] atask SPAWN_PAUSED: workspace_id=%s tool_call_id=%s",
|
||||
workspace_id,
|
||||
runtime.tool_call_id,
|
||||
)
|
||||
return (
|
||||
|
|
@ -867,6 +890,7 @@ def build_task_tool_with_parent_config(
|
|||
subagent_type,
|
||||
runtime.tool_call_id,
|
||||
)
|
||||
subagent_type = _canonical_subagent_type(subagent_type)
|
||||
if subagent_type not in subagent_names:
|
||||
allowed_types = ", ".join([f"`{k}`" for k in subagent_names])
|
||||
return (
|
||||
|
|
|
|||
|
|
@ -3,7 +3,6 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.tools import BaseTool
|
||||
|
||||
|
|
@ -25,7 +24,7 @@ def build_context_editing_mw(
|
|||
flags: AgentFeatureFlags,
|
||||
max_input_tokens: int | None,
|
||||
tools: Sequence[BaseTool],
|
||||
backend_resolver: Any,
|
||||
workspace_id: int | None = None,
|
||||
) -> SpillingContextEditingMiddleware | None:
|
||||
if not enabled(flags, "enable_context_editing") or not max_input_tokens:
|
||||
return None
|
||||
|
|
@ -46,5 +45,5 @@ def build_context_editing_mw(
|
|||
)
|
||||
return SpillingContextEditingMiddleware(
|
||||
edits=[spill_edit, clear_edit],
|
||||
backend_resolver=backend_resolver,
|
||||
workspace_id=workspace_id,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -4,25 +4,31 @@ SpillToBackendEdit + SpillingContextEditingMiddleware.
|
|||
LangChain's :class:`ClearToolUsesEdit` discards old ``ToolMessage.content``
|
||||
when the context-editing budget triggers, replacing the body with a fixed
|
||||
placeholder. That's lossy: anything the agent might want to revisit is
|
||||
gone. The spill-to-disk pattern (originally from OpenCode's
|
||||
gone. The spill pattern (originally from OpenCode's
|
||||
``opencode/packages/opencode/src/tool/truncate.ts``) keeps the prune
|
||||
behavior but writes the full original payload to the runtime backend
|
||||
under ``/tool_outputs/{thread_id}/{message_id}.txt`` first. The
|
||||
placeholder is then upgraded to point at the spill path so the agent
|
||||
(or a subagent) can read it back on demand.
|
||||
behavior but persists the full original payload first — to the
|
||||
``tool_output_spills`` table — and upgrades the placeholder to a
|
||||
``spill_<uuid>`` reference the agent can read back with the shared
|
||||
``read_run``/``search_run`` tools on demand.
|
||||
|
||||
Why the DB and not the runtime filesystem backend: the previous version
|
||||
wrote via ``backend.aupload_files``, which is a no-op on cloud
|
||||
(``KBPostgresBackend`` raises ``NotImplementedError``) and mismatched on
|
||||
desktop (paths must be ``/{mount_id}/...``), so spills were unrecoverable
|
||||
in production. A table works on every deployment and needs no sandbox.
|
||||
|
||||
Why this is a middleware subclass instead of a plain ``ContextEdit``:
|
||||
``ContextEdit.apply`` is sync, but writing to the backend is async. We
|
||||
capture the spill payloads inside ``apply`` and flush them via
|
||||
``await backend.aupload_files(...)`` from ``awrap_model_call`` *before*
|
||||
delegating to the handler, so the explore subagent can always read what
|
||||
the placeholder advertises.
|
||||
``ContextEdit.apply`` is sync, but the DB write is async. We generate the
|
||||
spill id and capture the payload inside ``apply`` (so the placeholder can
|
||||
reference the final id immediately) and flush the rows from
|
||||
``awrap_model_call`` *before* delegating to the handler.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import threading
|
||||
import uuid
|
||||
from collections.abc import Awaitable, Callable, Sequence
|
||||
from copy import deepcopy
|
||||
from dataclasses import dataclass, field
|
||||
|
|
@ -44,7 +50,6 @@ from langchain_core.messages.utils import count_tokens_approximately
|
|||
from langgraph.config import get_config
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from deepagents.backends.protocol import BackendProtocol
|
||||
from langchain.agents.middleware.types import (
|
||||
ModelRequest,
|
||||
ModelResponse,
|
||||
|
|
@ -52,24 +57,27 @@ if TYPE_CHECKING:
|
|||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_SPILL_PREFIX = "/tool_outputs"
|
||||
# Namespace for deterministic spill ids: the same (thread, message) always maps
|
||||
# to the same row, so re-running the edit on later model calls (edits apply to a
|
||||
# per-call copy of the messages, never to persisted state) re-references the
|
||||
# existing spill instead of inserting a duplicate every turn.
|
||||
_SPILL_NAMESPACE = uuid.UUID("6ba7b810-9dad-11d1-80b4-00c04fd430c8")
|
||||
|
||||
|
||||
def _build_spill_placeholder(spill_path: str) -> str:
|
||||
def _spill_id_for(thread_id: str | None, message_key: str) -> uuid.UUID:
|
||||
return uuid.uuid5(_SPILL_NAMESPACE, f"{thread_id or 'no_thread'}:{message_key}")
|
||||
|
||||
|
||||
def _build_spill_placeholder(spill_id: uuid.UUID) -> str:
|
||||
"""Build the user-facing placeholder text shown to the model."""
|
||||
return (
|
||||
f"[cleared — full output at {spill_path}; ask the explore subagent to read it]"
|
||||
f"[cleared — full output stored as spill_{spill_id}; "
|
||||
"use read_run/search_run to read it]"
|
||||
)
|
||||
|
||||
|
||||
def _get_thread_id_or_session() -> str:
|
||||
"""Best-effort thread_id discovery for the spill path.
|
||||
|
||||
Falls back to a process-stable string if no LangGraph config is
|
||||
available (e.g. unit tests). The exact value doesn't matter as long
|
||||
as it's stable within one stream so the placeholder paths line up
|
||||
with the actual upload path.
|
||||
"""
|
||||
def _get_thread_id() -> str | None:
|
||||
"""Best-effort ``configurable.thread_id`` for the spill row (``None`` if absent)."""
|
||||
try:
|
||||
config = get_config()
|
||||
thread_id = config.get("configurable", {}).get("thread_id")
|
||||
|
|
@ -77,17 +85,18 @@ def _get_thread_id_or_session() -> str:
|
|||
return str(thread_id)
|
||||
except RuntimeError:
|
||||
pass
|
||||
return "no_thread"
|
||||
return None
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class SpillToBackendEdit(ContextEdit):
|
||||
"""Capture-and-replace context edit that spills full tool output to the backend.
|
||||
"""Capture-and-replace context edit that spills full tool output to the DB.
|
||||
|
||||
Behaves like :class:`ClearToolUsesEdit` (same trigger / keep / exclude
|
||||
semantics) **and** records the original ``ToolMessage.content`` in
|
||||
:attr:`pending_spills` so the wrapping middleware can flush them
|
||||
before the model call.
|
||||
:attr:`pending_spills` so the wrapping middleware can flush the rows to
|
||||
``tool_output_spills`` before the model call. The spill id is generated up
|
||||
front so the placeholder can reference it immediately.
|
||||
|
||||
Args:
|
||||
trigger: Token threshold above which the edit fires.
|
||||
|
|
@ -97,8 +106,6 @@ class SpillToBackendEdit(ContextEdit):
|
|||
exclude_tools: Names of tools whose output is NOT spilled.
|
||||
clear_tool_inputs: Also clear the originating ``AIMessage.tool_calls``
|
||||
args when their pair is cleared.
|
||||
path_prefix: Path under the backend where spills are written.
|
||||
Default ``"/tool_outputs"``.
|
||||
"""
|
||||
|
||||
trigger: int = 100_000
|
||||
|
|
@ -106,12 +113,16 @@ class SpillToBackendEdit(ContextEdit):
|
|||
keep: int = 3
|
||||
clear_tool_inputs: bool = False
|
||||
exclude_tools: Sequence[str] = ()
|
||||
path_prefix: str = DEFAULT_SPILL_PREFIX
|
||||
|
||||
pending_spills: list[tuple[str, bytes]] = field(default_factory=list)
|
||||
# (spill_id, content_bytes, tool_name, thread_id)
|
||||
pending_spills: list[tuple[uuid.UUID, bytes, str | None, str | None]] = field(
|
||||
default_factory=list
|
||||
)
|
||||
_lock: threading.Lock = field(default_factory=threading.Lock)
|
||||
|
||||
def drain_pending(self) -> list[tuple[str, bytes]]:
|
||||
def drain_pending(
|
||||
self,
|
||||
) -> list[tuple[uuid.UUID, bytes, str | None, str | None]]:
|
||||
"""Return and clear the pending-spill list atomically."""
|
||||
with self._lock:
|
||||
out = list(self.pending_spills)
|
||||
|
|
@ -139,7 +150,7 @@ class SpillToBackendEdit(ContextEdit):
|
|||
if self.keep:
|
||||
candidates = candidates[: -self.keep]
|
||||
|
||||
thread_id = _get_thread_id_or_session()
|
||||
thread_id = _get_thread_id()
|
||||
excluded_tools = set(self.exclude_tools)
|
||||
|
||||
for idx, tool_message in candidates:
|
||||
|
|
@ -168,24 +179,23 @@ class SpillToBackendEdit(ContextEdit):
|
|||
if tool_name in excluded_tools:
|
||||
continue
|
||||
|
||||
message_id = tool_message.id or tool_message.tool_call_id or "unknown"
|
||||
spill_path = f"{self.path_prefix}/{thread_id}/{message_id}.txt"
|
||||
|
||||
message_key = tool_message.id or tool_message.tool_call_id or "unknown"
|
||||
spill_id = _spill_id_for(thread_id, message_key)
|
||||
original = tool_message.content
|
||||
payload = self._encode_payload(original)
|
||||
with self._lock:
|
||||
self.pending_spills.append((spill_path, payload))
|
||||
self.pending_spills.append((spill_id, payload, tool_name, thread_id))
|
||||
|
||||
messages[idx] = tool_message.model_copy(
|
||||
update={
|
||||
"artifact": None,
|
||||
"content": _build_spill_placeholder(spill_path),
|
||||
"content": _build_spill_placeholder(spill_id),
|
||||
"response_metadata": {
|
||||
**tool_message.response_metadata,
|
||||
"context_editing": {
|
||||
"cleared": True,
|
||||
"strategy": "spill_to_backend",
|
||||
"spill_path": spill_path,
|
||||
"strategy": "spill_to_db",
|
||||
"spill_id": str(spill_id),
|
||||
},
|
||||
},
|
||||
}
|
||||
|
|
@ -243,52 +253,30 @@ class SpillToBackendEdit(ContextEdit):
|
|||
)
|
||||
|
||||
|
||||
BackendResolver = "Callable[[Any], BackendProtocol] | BackendProtocol"
|
||||
|
||||
|
||||
class SpillingContextEditingMiddleware(ContextEditingMiddleware):
|
||||
""":class:`ContextEditingMiddleware` that flushes :class:`SpillToBackendEdit` writes.
|
||||
|
||||
Runs the configured edits as the parent does, then flushes any
|
||||
pending spills via the supplied backend resolver before delegating
|
||||
to the model handler. Spill failures are logged but never abort the
|
||||
model call — the placeholder text is already in the message, so the
|
||||
worst case is the agent gets a placeholder it cannot follow up on.
|
||||
Runs the configured edits as the parent does, then persists any pending
|
||||
spills to ``tool_output_spills`` before delegating to the model handler.
|
||||
Spill failures are logged but never abort the model call — the placeholder
|
||||
text is already in the message, so the worst case is the agent gets a
|
||||
placeholder it cannot follow up on.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
edits: Sequence[ContextEdit],
|
||||
backend_resolver: BackendResolver | None = None,
|
||||
workspace_id: int | None = None,
|
||||
token_count_method: str = "approximate",
|
||||
) -> None:
|
||||
super().__init__(edits=list(edits), token_count_method=token_count_method) # type: ignore[arg-type]
|
||||
self._backend_resolver = backend_resolver
|
||||
self._workspace_id = workspace_id
|
||||
|
||||
def _resolve_backend(self, request: ModelRequest) -> BackendProtocol | None:
|
||||
if self._backend_resolver is None:
|
||||
return None
|
||||
if callable(self._backend_resolver):
|
||||
try:
|
||||
from langchain.tools import ToolRuntime
|
||||
|
||||
tool_runtime = ToolRuntime(
|
||||
state=getattr(request, "state", {}),
|
||||
context=getattr(request.runtime, "context", None),
|
||||
stream_writer=getattr(request.runtime, "stream_writer", None),
|
||||
store=getattr(request.runtime, "store", None),
|
||||
config=getattr(request.runtime, "config", None) or {},
|
||||
tool_call_id=None,
|
||||
)
|
||||
return self._backend_resolver(tool_runtime)
|
||||
except Exception:
|
||||
logger.exception("Failed to resolve spill backend")
|
||||
return None
|
||||
return self._backend_resolver # type: ignore[return-value]
|
||||
|
||||
def _collect_pending(self) -> list[tuple[str, bytes]]:
|
||||
out: list[tuple[str, bytes]] = []
|
||||
def _collect_pending(
|
||||
self,
|
||||
) -> list[tuple[uuid.UUID, bytes, str | None, str | None]]:
|
||||
out: list[tuple[uuid.UUID, bytes, str | None, str | None]] = []
|
||||
for edit in self.edits:
|
||||
if isinstance(edit, SpillToBackendEdit):
|
||||
out.extend(edit.drain_pending())
|
||||
|
|
@ -321,28 +309,37 @@ class SpillingContextEditingMiddleware(ContextEditingMiddleware):
|
|||
|
||||
pending = self._collect_pending()
|
||||
if pending:
|
||||
backend = self._resolve_backend(request)
|
||||
if backend is not None:
|
||||
try:
|
||||
await backend.aupload_files(pending)
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"Spill-to-backend upload failed (%d files); placeholders "
|
||||
"remain in messages but content is unrecoverable",
|
||||
len(pending),
|
||||
)
|
||||
else:
|
||||
logger.warning(
|
||||
"SpillToBackendEdit produced %d pending spills but no backend "
|
||||
"resolver was configured; content is unrecoverable",
|
||||
len(pending),
|
||||
)
|
||||
await self._flush_spills(pending)
|
||||
|
||||
return await handler(request.override(messages=edited_messages))
|
||||
|
||||
async def _flush_spills(
|
||||
self, pending: list[tuple[uuid.UUID, bytes, str | None, str | None]]
|
||||
) -> None:
|
||||
"""Persist spilled tool outputs to the DB (best-effort)."""
|
||||
from app.capabilities.core.runs import record_spill
|
||||
from app.db import async_session_maker
|
||||
|
||||
try:
|
||||
async with async_session_maker() as session:
|
||||
for spill_id, payload, tool_name, thread_id in pending:
|
||||
await record_spill(
|
||||
session,
|
||||
spill_id=spill_id,
|
||||
content=payload.decode("utf-8", errors="replace"),
|
||||
workspace_id=self._workspace_id,
|
||||
thread_id=thread_id,
|
||||
tool_name=tool_name,
|
||||
)
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"Spill-to-DB flush failed (%d rows); placeholders remain in "
|
||||
"messages but content is unrecoverable",
|
||||
len(pending),
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"DEFAULT_SPILL_PREFIX",
|
||||
"ClearToolUsesEdit",
|
||||
"SpillToBackendEdit",
|
||||
"SpillingContextEditingMiddleware",
|
||||
|
|
|
|||
|
|
@ -12,14 +12,14 @@ from .middleware import (
|
|||
def build_kb_persistence_mw(
|
||||
*,
|
||||
filesystem_mode: FilesystemMode,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
user_id: str | None,
|
||||
thread_id: int | None,
|
||||
) -> KnowledgeBasePersistenceMiddleware | None:
|
||||
if filesystem_mode != FilesystemMode.CLOUD:
|
||||
return None
|
||||
return KnowledgeBasePersistenceMiddleware(
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
created_by_id=user_id,
|
||||
filesystem_mode=filesystem_mode,
|
||||
thread_id=thread_id,
|
||||
|
|
|
|||
|
|
@ -83,11 +83,11 @@ def _basename(path: str) -> str:
|
|||
async def _ensure_folder_hierarchy(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
created_by_id: str | None,
|
||||
folder_parts: list[str],
|
||||
) -> int | None:
|
||||
"""Ensure a chain of folder names exists under the search space.
|
||||
"""Ensure a chain of folder names exists under the workspace.
|
||||
|
||||
Returns the leaf folder id, or ``None`` if ``folder_parts`` is empty
|
||||
(i.e. a document directly under ``/documents/``).
|
||||
|
|
@ -98,7 +98,7 @@ async def _ensure_folder_hierarchy(
|
|||
for raw in folder_parts:
|
||||
name = safe_folder_segment(str(raw))
|
||||
query = select(Folder).where(
|
||||
Folder.search_space_id == search_space_id,
|
||||
Folder.workspace_id == workspace_id,
|
||||
Folder.name == name,
|
||||
)
|
||||
if parent_id is None:
|
||||
|
|
@ -111,9 +111,7 @@ async def _ensure_folder_hierarchy(
|
|||
sibling_query = (
|
||||
select(Folder.position).order_by(Folder.position.desc()).limit(1)
|
||||
)
|
||||
sibling_query = sibling_query.where(
|
||||
Folder.search_space_id == search_space_id
|
||||
)
|
||||
sibling_query = sibling_query.where(Folder.workspace_id == workspace_id)
|
||||
if parent_id is None:
|
||||
sibling_query = sibling_query.where(Folder.parent_id.is_(None))
|
||||
else:
|
||||
|
|
@ -124,7 +122,7 @@ async def _ensure_folder_hierarchy(
|
|||
name=name,
|
||||
position=generate_key_between(last_position, None),
|
||||
parent_id=parent_id,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
created_by_id=created_by_id,
|
||||
updated_at=datetime.now(UTC),
|
||||
)
|
||||
|
|
@ -137,7 +135,7 @@ async def _ensure_folder_hierarchy(
|
|||
async def _resolve_folder_id(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
folder_parts: list[str],
|
||||
) -> int | None:
|
||||
"""Look up an existing folder chain without creating anything.
|
||||
|
|
@ -151,7 +149,7 @@ async def _resolve_folder_id(
|
|||
for raw in folder_parts:
|
||||
name = safe_folder_segment(str(raw))
|
||||
query = select(Folder).where(
|
||||
Folder.search_space_id == search_space_id,
|
||||
Folder.workspace_id == workspace_id,
|
||||
Folder.name == name,
|
||||
)
|
||||
query = (
|
||||
|
|
@ -185,7 +183,7 @@ async def _create_document(
|
|||
*,
|
||||
virtual_path: str,
|
||||
content: str,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
created_by_id: str | None,
|
||||
) -> Document:
|
||||
"""Create a NOTE Document + Chunks for ``virtual_path``."""
|
||||
|
|
@ -194,21 +192,21 @@ async def _create_document(
|
|||
raise ValueError(f"invalid /documents path '{virtual_path}'")
|
||||
folder_id = await _ensure_folder_hierarchy(
|
||||
session,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
created_by_id=created_by_id,
|
||||
folder_parts=folder_parts,
|
||||
)
|
||||
unique_identifier_hash = generate_unique_identifier_hash(
|
||||
DocumentType.NOTE,
|
||||
virtual_path,
|
||||
search_space_id,
|
||||
workspace_id,
|
||||
)
|
||||
# Pre-check the path-derived unique_identifier_hash so a duplicate path
|
||||
# surfaces as a clean ValueError instead of an INSERT IntegrityError that
|
||||
# poisons the session. Content is intentionally not unique (cp a b).
|
||||
path_collision = await session.execute(
|
||||
select(Document.id).where(
|
||||
Document.search_space_id == search_space_id,
|
||||
Document.workspace_id == workspace_id,
|
||||
Document.unique_identifier_hash == unique_identifier_hash,
|
||||
)
|
||||
)
|
||||
|
|
@ -217,7 +215,7 @@ async def _create_document(
|
|||
f"a document already exists at path '{virtual_path}' "
|
||||
"(unique_identifier_hash collision)"
|
||||
)
|
||||
content_hash = generate_content_hash(content, search_space_id)
|
||||
content_hash = generate_content_hash(content, workspace_id)
|
||||
doc = Document(
|
||||
title=title,
|
||||
document_type=DocumentType.NOTE,
|
||||
|
|
@ -226,7 +224,7 @@ async def _create_document(
|
|||
content_hash=content_hash,
|
||||
unique_identifier_hash=unique_identifier_hash,
|
||||
source_markdown=content,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
folder_id=folder_id,
|
||||
created_by_id=created_by_id,
|
||||
updated_at=datetime.now(UTC),
|
||||
|
|
@ -261,13 +259,13 @@ async def _update_document(
|
|||
doc_id: int,
|
||||
content: str,
|
||||
virtual_path: str,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
) -> Document | None:
|
||||
"""Update an existing Document's content + chunks."""
|
||||
result = await session.execute(
|
||||
select(Document).where(
|
||||
Document.id == doc_id,
|
||||
Document.search_space_id == search_space_id,
|
||||
Document.workspace_id == workspace_id,
|
||||
)
|
||||
)
|
||||
document = result.scalar_one_or_none()
|
||||
|
|
@ -276,7 +274,7 @@ async def _update_document(
|
|||
|
||||
document.content = content
|
||||
document.source_markdown = content
|
||||
document.content_hash = generate_content_hash(content, search_space_id)
|
||||
document.content_hash = generate_content_hash(content, workspace_id)
|
||||
document.updated_at = datetime.now(UTC)
|
||||
metadata = dict(document.document_metadata or {})
|
||||
metadata["virtual_path"] = virtual_path
|
||||
|
|
@ -284,7 +282,7 @@ async def _update_document(
|
|||
document.unique_identifier_hash = generate_unique_identifier_hash(
|
||||
DocumentType.NOTE,
|
||||
virtual_path,
|
||||
search_space_id,
|
||||
workspace_id,
|
||||
)
|
||||
|
||||
summary_embedding = (await asyncio.to_thread(embed_texts, [content]))[0]
|
||||
|
|
@ -318,7 +316,7 @@ async def _update_document(
|
|||
async def _apply_move(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
created_by_id: str | None,
|
||||
move: dict[str, Any],
|
||||
doc_id_by_path: dict[str, int],
|
||||
|
|
@ -341,14 +339,14 @@ async def _apply_move(
|
|||
result = await session.execute(
|
||||
select(Document).where(
|
||||
Document.id == doc_id,
|
||||
Document.search_space_id == search_space_id,
|
||||
Document.workspace_id == workspace_id,
|
||||
)
|
||||
)
|
||||
document = result.scalar_one_or_none()
|
||||
if document is None:
|
||||
document = await virtual_path_to_doc(
|
||||
session,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
virtual_path=source,
|
||||
)
|
||||
if document is None:
|
||||
|
|
@ -364,7 +362,7 @@ async def _apply_move(
|
|||
return None
|
||||
folder_id = await _ensure_folder_hierarchy(
|
||||
session,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
created_by_id=created_by_id,
|
||||
folder_parts=folder_parts,
|
||||
)
|
||||
|
|
@ -377,7 +375,7 @@ async def _apply_move(
|
|||
document.unique_identifier_hash = generate_unique_identifier_hash(
|
||||
DocumentType.NOTE,
|
||||
dest,
|
||||
search_space_id,
|
||||
workspace_id,
|
||||
)
|
||||
document.updated_at = datetime.now(UTC)
|
||||
|
||||
|
|
@ -501,7 +499,7 @@ async def _snapshot_document_pre_write(
|
|||
*,
|
||||
doc: Document,
|
||||
action_id: int | None,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
turn_id: str | None = None,
|
||||
deferred_dispatches: list[int] | None = None,
|
||||
) -> int | None:
|
||||
|
|
@ -517,7 +515,7 @@ async def _snapshot_document_pre_write(
|
|||
payload = _doc_revision_payload(doc, chunks_before=chunks)
|
||||
rev = DocumentRevision(
|
||||
document_id=doc.id,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
created_by_turn_id=turn_id,
|
||||
agent_action_id=action_id,
|
||||
**payload,
|
||||
|
|
@ -544,7 +542,7 @@ async def _snapshot_document_pre_create(
|
|||
session: AsyncSession,
|
||||
*,
|
||||
action_id: int | None,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
turn_id: str | None = None,
|
||||
deferred_dispatches: list[int] | None = None,
|
||||
) -> int | None:
|
||||
|
|
@ -558,7 +556,7 @@ async def _snapshot_document_pre_create(
|
|||
async with session.begin_nested():
|
||||
rev = DocumentRevision(
|
||||
document_id=None,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
content_before=None,
|
||||
title_before=None,
|
||||
folder_id_before=None,
|
||||
|
|
@ -586,7 +584,7 @@ async def _snapshot_document_pre_move(
|
|||
*,
|
||||
doc: Document,
|
||||
action_id: int | None,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
turn_id: str | None = None,
|
||||
deferred_dispatches: list[int] | None = None,
|
||||
) -> int | None:
|
||||
|
|
@ -596,7 +594,7 @@ async def _snapshot_document_pre_move(
|
|||
payload = _doc_revision_payload(doc, chunks_before=None)
|
||||
rev = DocumentRevision(
|
||||
document_id=doc.id,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
created_by_turn_id=turn_id,
|
||||
agent_action_id=action_id,
|
||||
**payload,
|
||||
|
|
@ -624,7 +622,7 @@ async def _snapshot_folder_pre_mkdir(
|
|||
*,
|
||||
folder: Folder,
|
||||
action_id: int | None,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
turn_id: str | None = None,
|
||||
deferred_dispatches: list[int] | None = None,
|
||||
) -> int | None:
|
||||
|
|
@ -637,7 +635,7 @@ async def _snapshot_folder_pre_mkdir(
|
|||
async with session.begin_nested():
|
||||
rev = FolderRevision(
|
||||
folder_id=folder.id,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
name_before=None,
|
||||
parent_id_before=None,
|
||||
position_before=None,
|
||||
|
|
@ -670,7 +668,7 @@ async def _snapshot_folder_pre_mkdir(
|
|||
async def commit_staged_filesystem_state(
|
||||
state: dict[str, Any] | AgentState,
|
||||
*,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
created_by_id: str | None,
|
||||
filesystem_mode: FilesystemMode = FilesystemMode.CLOUD,
|
||||
thread_id: int | None = None,
|
||||
|
|
@ -814,7 +812,7 @@ async def commit_staged_filesystem_state(
|
|||
continue
|
||||
folder_id = await _ensure_folder_hierarchy(
|
||||
session,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
created_by_id=created_by_id,
|
||||
folder_parts=folder_parts_full,
|
||||
)
|
||||
|
|
@ -833,7 +831,7 @@ async def commit_staged_filesystem_state(
|
|||
session,
|
||||
folder=folder_row,
|
||||
action_id=action_id,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
turn_id=tcid,
|
||||
deferred_dispatches=deferred_dispatches,
|
||||
)
|
||||
|
|
@ -851,14 +849,14 @@ async def commit_staged_filesystem_state(
|
|||
res_pre = await session.execute(
|
||||
select(Document).where(
|
||||
Document.id == doc_id_pre,
|
||||
Document.search_space_id == search_space_id,
|
||||
Document.workspace_id == workspace_id,
|
||||
)
|
||||
)
|
||||
document_pre = res_pre.scalar_one_or_none()
|
||||
if document_pre is None:
|
||||
document_pre = await virtual_path_to_doc(
|
||||
session,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
virtual_path=source,
|
||||
)
|
||||
if document_pre is not None:
|
||||
|
|
@ -866,14 +864,14 @@ async def commit_staged_filesystem_state(
|
|||
session,
|
||||
doc=document_pre,
|
||||
action_id=action_id,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
turn_id=tcid,
|
||||
deferred_dispatches=deferred_dispatches,
|
||||
)
|
||||
|
||||
applied = await _apply_move(
|
||||
session,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
created_by_id=created_by_id,
|
||||
move=move,
|
||||
doc_id_by_path=doc_id_by_path,
|
||||
|
|
@ -937,7 +935,7 @@ async def commit_staged_filesystem_state(
|
|||
# INSERT (which would hit the path-derived unique hash).
|
||||
existing = await virtual_path_to_doc(
|
||||
session,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
virtual_path=path,
|
||||
)
|
||||
if existing is not None:
|
||||
|
|
@ -948,7 +946,7 @@ async def commit_staged_filesystem_state(
|
|||
result_doc = await session.execute(
|
||||
select(Document).where(
|
||||
Document.id == doc_id,
|
||||
Document.search_space_id == search_space_id,
|
||||
Document.workspace_id == workspace_id,
|
||||
)
|
||||
)
|
||||
existing_doc = result_doc.scalar_one_or_none()
|
||||
|
|
@ -957,7 +955,7 @@ async def commit_staged_filesystem_state(
|
|||
session,
|
||||
doc=existing_doc,
|
||||
action_id=action_id,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
turn_id=tcid,
|
||||
deferred_dispatches=deferred_dispatches,
|
||||
)
|
||||
|
|
@ -966,7 +964,7 @@ async def commit_staged_filesystem_state(
|
|||
doc_id=doc_id,
|
||||
content=content,
|
||||
virtual_path=path,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
)
|
||||
if updated is not None:
|
||||
committed_updates.append(
|
||||
|
|
@ -974,7 +972,7 @@ async def commit_staged_filesystem_state(
|
|||
"id": updated.id,
|
||||
"title": updated.title,
|
||||
"documentType": DocumentType.NOTE.value,
|
||||
"searchSpaceId": search_space_id,
|
||||
"workspaceId": workspace_id,
|
||||
"folderId": updated.folder_id,
|
||||
"createdById": str(created_by_id)
|
||||
if created_by_id
|
||||
|
|
@ -991,7 +989,7 @@ async def commit_staged_filesystem_state(
|
|||
placeholder_revision_id = await _snapshot_document_pre_create(
|
||||
session,
|
||||
action_id=action_id,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
turn_id=tcid,
|
||||
deferred_dispatches=deferred_dispatches,
|
||||
)
|
||||
|
|
@ -1001,7 +999,7 @@ async def commit_staged_filesystem_state(
|
|||
session,
|
||||
virtual_path=path,
|
||||
content=content,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
created_by_id=created_by_id,
|
||||
)
|
||||
except ValueError as exc:
|
||||
|
|
@ -1045,7 +1043,7 @@ async def commit_staged_filesystem_state(
|
|||
"id": new_doc.id,
|
||||
"title": new_doc.title,
|
||||
"documentType": DocumentType.NOTE.value,
|
||||
"searchSpaceId": search_space_id,
|
||||
"workspaceId": workspace_id,
|
||||
"folderId": new_doc.folder_id,
|
||||
"createdById": str(created_by_id)
|
||||
if created_by_id
|
||||
|
|
@ -1069,14 +1067,14 @@ async def commit_staged_filesystem_state(
|
|||
result = await session.execute(
|
||||
select(Document).where(
|
||||
Document.id == doc_id_for_delete,
|
||||
Document.search_space_id == search_space_id,
|
||||
Document.workspace_id == workspace_id,
|
||||
)
|
||||
)
|
||||
document_to_delete = result.scalar_one_or_none()
|
||||
if document_to_delete is None:
|
||||
document_to_delete = await virtual_path_to_doc(
|
||||
session,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
virtual_path=final,
|
||||
)
|
||||
if document_to_delete is None:
|
||||
|
|
@ -1100,7 +1098,7 @@ async def commit_staged_filesystem_state(
|
|||
)
|
||||
rev = DocumentRevision(
|
||||
document_id=doc_pk,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
created_by_turn_id=tcid,
|
||||
agent_action_id=action_id,
|
||||
**payload,
|
||||
|
|
@ -1130,7 +1128,7 @@ async def commit_staged_filesystem_state(
|
|||
"id": doc_pk,
|
||||
"title": doc_title,
|
||||
"documentType": DocumentType.NOTE.value,
|
||||
"searchSpaceId": search_space_id,
|
||||
"workspaceId": workspace_id,
|
||||
"folderId": doc_folder_id,
|
||||
"createdById": str(created_by_id) if created_by_id else None,
|
||||
"virtualPath": final,
|
||||
|
|
@ -1151,7 +1149,7 @@ async def commit_staged_filesystem_state(
|
|||
continue
|
||||
folder_id = await _resolve_folder_id(
|
||||
session,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
folder_parts=folder_parts,
|
||||
)
|
||||
if folder_id is None:
|
||||
|
|
@ -1163,7 +1161,7 @@ async def commit_staged_filesystem_state(
|
|||
docs_in_folder = await session.execute(
|
||||
select(Document.id)
|
||||
.where(Document.folder_id == folder_id)
|
||||
.where(Document.search_space_id == search_space_id)
|
||||
.where(Document.workspace_id == workspace_id)
|
||||
.limit(1)
|
||||
)
|
||||
if docs_in_folder.scalar_one_or_none() is not None:
|
||||
|
|
@ -1175,7 +1173,7 @@ async def commit_staged_filesystem_state(
|
|||
child_folders = await session.execute(
|
||||
select(Folder.id)
|
||||
.where(Folder.parent_id == folder_id)
|
||||
.where(Folder.search_space_id == search_space_id)
|
||||
.where(Folder.workspace_id == workspace_id)
|
||||
.limit(1)
|
||||
)
|
||||
if child_folders.scalar_one_or_none() is not None:
|
||||
|
|
@ -1203,7 +1201,7 @@ async def commit_staged_filesystem_state(
|
|||
if snapshot_enabled and action_id is not None:
|
||||
rev = FolderRevision(
|
||||
folder_id=folder_pk,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
name_before=folder_name,
|
||||
parent_id_before=folder_parent_id,
|
||||
position_before=folder_position,
|
||||
|
|
@ -1232,7 +1230,7 @@ async def commit_staged_filesystem_state(
|
|||
{
|
||||
"id": folder_pk,
|
||||
"name": folder_name,
|
||||
"searchSpaceId": search_space_id,
|
||||
"workspaceId": workspace_id,
|
||||
"parentId": folder_parent_id,
|
||||
"virtualPath": final,
|
||||
}
|
||||
|
|
@ -1241,9 +1239,7 @@ async def commit_staged_filesystem_state(
|
|||
|
||||
await session.commit()
|
||||
except Exception: # pragma: no cover - rollback safety net
|
||||
logger.exception(
|
||||
"kb_persistence: commit failed (search_space=%s)", search_space_id
|
||||
)
|
||||
logger.exception("kb_persistence: commit failed (workspace=%s)", workspace_id)
|
||||
# Outer commit raised: everything above rolled back, so drop the
|
||||
# deferred dispatches.
|
||||
deferred_dispatches.clear()
|
||||
|
|
@ -1402,9 +1398,9 @@ async def commit_staged_filesystem_state(
|
|||
_ = turn_id_for_revision # diagnostic-only; silence unused lint
|
||||
|
||||
logger.info(
|
||||
"kb_persistence: commit (search_space=%s) creates=%d updates=%d "
|
||||
"kb_persistence: commit (workspace=%s) creates=%d updates=%d "
|
||||
"moves=%d staged_dirs=%d deletes=%d folder_deletes=%d discarded=%d",
|
||||
search_space_id,
|
||||
workspace_id,
|
||||
len(committed_creates),
|
||||
len(committed_updates),
|
||||
len(applied_moves),
|
||||
|
|
@ -1430,12 +1426,12 @@ class KnowledgeBasePersistenceMiddleware(AgentMiddleware): # type: ignore[type-
|
|||
def __init__(
|
||||
self,
|
||||
*,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
created_by_id: str | None,
|
||||
filesystem_mode: FilesystemMode,
|
||||
thread_id: int | None = None,
|
||||
) -> None:
|
||||
self.search_space_id = search_space_id
|
||||
self.workspace_id = workspace_id
|
||||
self.created_by_id = created_by_id
|
||||
self.filesystem_mode = filesystem_mode
|
||||
self.thread_id = thread_id
|
||||
|
|
@ -1450,7 +1446,7 @@ class KnowledgeBasePersistenceMiddleware(AgentMiddleware): # type: ignore[type-
|
|||
return None
|
||||
return await commit_staged_filesystem_state(
|
||||
state,
|
||||
search_space_id=self.search_space_id,
|
||||
workspace_id=self.workspace_id,
|
||||
created_by_id=self.created_by_id,
|
||||
filesystem_mode=self.filesystem_mode,
|
||||
thread_id=self._resolve_thread_id(),
|
||||
|
|
|
|||
|
|
@ -12,13 +12,13 @@ from .middleware import KnowledgeTreeMiddleware
|
|||
def build_knowledge_tree_mw(
|
||||
*,
|
||||
filesystem_mode: FilesystemMode,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
llm: BaseChatModel,
|
||||
) -> KnowledgeTreeMiddleware | None:
|
||||
if filesystem_mode != FilesystemMode.CLOUD:
|
||||
return None
|
||||
return KnowledgeTreeMiddleware(
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
filesystem_mode=filesystem_mode,
|
||||
llm=llm,
|
||||
inject_system_message=False,
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
"""Workspace-tree middleware for the SurfSense agent.
|
||||
|
||||
Renders the full ``Folder``+``Document`` tree under ``/documents/`` once per
|
||||
turn (cloud only), caches it by ``(search_space_id, tree_version)``, and
|
||||
turn (cloud only), caches it by ``(workspace_id, tree_version)``, and
|
||||
injects the result as a ``<workspace_tree>`` system message immediately
|
||||
before the latest human turn.
|
||||
|
||||
|
|
@ -106,14 +106,14 @@ class KnowledgeTreeMiddleware(AgentMiddleware): # type: ignore[type-arg]
|
|||
def __init__(
|
||||
self,
|
||||
*,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
filesystem_mode: FilesystemMode,
|
||||
llm: BaseChatModel | None = None,
|
||||
max_entries: int = MAX_TREE_ENTRIES,
|
||||
max_tokens: int = MAX_TREE_TOKENS,
|
||||
inject_system_message: bool = True, # For backwards compatibility
|
||||
) -> None:
|
||||
self.search_space_id = search_space_id
|
||||
self.workspace_id = workspace_id
|
||||
self.filesystem_mode = filesystem_mode
|
||||
self.llm = llm
|
||||
self.max_entries = max_entries
|
||||
|
|
@ -141,7 +141,7 @@ class KnowledgeTreeMiddleware(AgentMiddleware): # type: ignore[type-arg]
|
|||
cache_outcome = "anon"
|
||||
else:
|
||||
version = int(state.get("tree_version") or 0)
|
||||
cache_key = (self.search_space_id, version, False)
|
||||
cache_key = (self.workspace_id, version, False)
|
||||
cache_outcome = "hit" if cache_key in self._cache else "miss"
|
||||
tree_msg = await self._render_kb_tree(state)
|
||||
|
||||
|
|
@ -158,7 +158,7 @@ class KnowledgeTreeMiddleware(AgentMiddleware): # type: ignore[type-arg]
|
|||
cache_outcome,
|
||||
len(tree_msg),
|
||||
time.perf_counter() - start,
|
||||
self.search_space_id,
|
||||
self.workspace_id,
|
||||
)
|
||||
return update
|
||||
|
||||
|
|
@ -190,17 +190,17 @@ class KnowledgeTreeMiddleware(AgentMiddleware): # type: ignore[type-arg]
|
|||
|
||||
async def _render_kb_tree(self, state: AgentState) -> str:
|
||||
version = int(state.get("tree_version") or 0)
|
||||
cache_key = (self.search_space_id, version, False)
|
||||
cache_key = (self.workspace_id, version, False)
|
||||
cached = self._cache.get(cache_key)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
try:
|
||||
async with shielded_async_session() as session:
|
||||
index = await build_path_index(session, self.search_space_id)
|
||||
index = await build_path_index(session, self.workspace_id)
|
||||
doc_rows = await session.execute(
|
||||
select(Document.id, Document.title, Document.folder_id).where(
|
||||
Document.search_space_id == self.search_space_id
|
||||
Document.workspace_id == self.workspace_id
|
||||
)
|
||||
)
|
||||
docs = list(doc_rows.all())
|
||||
|
|
|
|||
|
|
@ -10,11 +10,11 @@ from .middleware import MemoryInjectionMiddleware
|
|||
def build_memory_mw(
|
||||
*,
|
||||
user_id: str | None,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
visibility: ChatVisibility,
|
||||
) -> MemoryInjectionMiddleware:
|
||||
return MemoryInjectionMiddleware(
|
||||
user_id=user_id,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
thread_visibility=visibility,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ from langgraph.runtime import Runtime
|
|||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db import ChatVisibility, SearchSpace, User, shielded_async_session
|
||||
from app.db import ChatVisibility, User, Workspace, shielded_async_session
|
||||
from app.services.memory import MEMORY_HARD_LIMIT, MEMORY_SOFT_LIMIT
|
||||
from app.utils.perf import get_perf_logger
|
||||
|
||||
|
|
@ -35,11 +35,11 @@ class MemoryInjectionMiddleware(AgentMiddleware): # type: ignore[type-arg]
|
|||
self,
|
||||
*,
|
||||
user_id: str | UUID | None,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
thread_visibility: ChatVisibility | None = None,
|
||||
) -> None:
|
||||
self.user_id = UUID(user_id) if isinstance(user_id, str) else user_id
|
||||
self.search_space_id = search_space_id
|
||||
self.workspace_id = workspace_id
|
||||
self.visibility = thread_visibility or ChatVisibility.PRIVATE
|
||||
|
||||
async def abefore_agent( # type: ignore[override]
|
||||
|
|
@ -149,8 +149,8 @@ class MemoryInjectionMiddleware(AgentMiddleware): # type: ignore[type-arg]
|
|||
async def _load_team_memory(self, session: AsyncSession) -> str | None:
|
||||
try:
|
||||
result = await session.execute(
|
||||
select(SearchSpace.shared_memory_md).where(
|
||||
SearchSpace.id == self.search_space_id
|
||||
select(Workspace.shared_memory_md).where(
|
||||
Workspace.id == self.workspace_id
|
||||
)
|
||||
)
|
||||
row = result.scalar_one_or_none()
|
||||
|
|
|
|||
|
|
@ -21,7 +21,7 @@ from ..plugins.loader import (
|
|||
def build_plugin_middlewares(
|
||||
*,
|
||||
flags: AgentFeatureFlags,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
user_id: str | None,
|
||||
visibility: ChatVisibility,
|
||||
llm: BaseChatModel,
|
||||
|
|
@ -34,7 +34,7 @@ def build_plugin_middlewares(
|
|||
return []
|
||||
return load_plugin_middlewares(
|
||||
PluginContext.build(
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
user_id=user_id,
|
||||
thread_visibility=visibility,
|
||||
llm=llm,
|
||||
|
|
|
|||
|
|
@ -17,13 +17,13 @@ def build_skills_mw(
|
|||
*,
|
||||
flags: AgentFeatureFlags,
|
||||
filesystem_mode: FilesystemMode,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
) -> SkillsMiddleware | None:
|
||||
if not enabled(flags, "enable_skills"):
|
||||
return None
|
||||
try:
|
||||
skills_factory = build_skills_backend_factory(
|
||||
search_space_id=search_space_id
|
||||
workspace_id=workspace_id
|
||||
if filesystem_mode == FilesystemMode.CLOUD
|
||||
else None,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -7,6 +7,11 @@ retrieval are owned by the ``knowledge_base`` subagent and reached via the
|
|||
the main agent only sees the specialist's grounded summary. The stack here
|
||||
computes the workspace tree, commits any subagent-side staged writes at end of
|
||||
turn (cloud mode), and wires the supporting middleware.
|
||||
|
||||
One deliberate, read-only exception to the pure-router stance: the main agent
|
||||
also carries the ``read_run``/``search_run`` tools (added in ``runtime/factory``)
|
||||
so it can follow the context-editing spill placeholder — evicted tool output is
|
||||
persisted to ``tool_output_spills`` and the placeholder advertises those tools.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -99,7 +104,7 @@ def build_main_agent_deepagent_middleware(
|
|||
tools: Sequence[BaseTool],
|
||||
backend_resolver: Any,
|
||||
filesystem_mode: FilesystemMode,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
user_id: str | None,
|
||||
thread_id: int | None,
|
||||
visibility: ChatVisibility,
|
||||
|
|
@ -120,7 +125,7 @@ def build_main_agent_deepagent_middleware(
|
|||
|
||||
memory_mw = build_memory_mw(
|
||||
user_id=user_id,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
visibility=visibility,
|
||||
)
|
||||
|
||||
|
|
@ -232,19 +237,19 @@ def build_main_agent_deepagent_middleware(
|
|||
),
|
||||
build_knowledge_tree_mw(
|
||||
filesystem_mode=filesystem_mode,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
llm=llm,
|
||||
),
|
||||
build_kb_persistence_mw(
|
||||
filesystem_mode=filesystem_mode,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
user_id=user_id,
|
||||
thread_id=thread_id,
|
||||
),
|
||||
build_skills_mw(
|
||||
flags=flags,
|
||||
filesystem_mode=filesystem_mode,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
),
|
||||
SurfSenseCheckpointedSubAgentMiddleware(
|
||||
checkpointer=checkpointer,
|
||||
|
|
@ -252,7 +257,7 @@ def build_main_agent_deepagent_middleware(
|
|||
subagents=subagents,
|
||||
system_prompt=None,
|
||||
task_description=TASK_TOOL_DESCRIPTION,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
),
|
||||
resilience.model_call_limit,
|
||||
resilience.tool_call_limit,
|
||||
|
|
@ -260,7 +265,7 @@ def build_main_agent_deepagent_middleware(
|
|||
flags=flags,
|
||||
max_input_tokens=max_input_tokens,
|
||||
tools=tools,
|
||||
backend_resolver=backend_resolver,
|
||||
workspace_id=workspace_id,
|
||||
),
|
||||
build_compaction_mw(llm),
|
||||
build_noop_injection_mw(flags),
|
||||
|
|
@ -272,14 +277,14 @@ def build_main_agent_deepagent_middleware(
|
|||
build_action_log_mw(
|
||||
flags=flags,
|
||||
thread_id=thread_id,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
user_id=user_id,
|
||||
),
|
||||
build_patch_tool_calls_mw(),
|
||||
build_dedup_hitl_mw(tools),
|
||||
*build_plugin_middlewares(
|
||||
flags=flags,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
user_id=user_id,
|
||||
visibility=visibility,
|
||||
llm=llm,
|
||||
|
|
|
|||
|
|
@ -57,7 +57,7 @@ class PluginContext(dict):
|
|||
Backed by ``dict`` so plugins can inspect the keys they care about
|
||||
without coupling to a concrete dataclass shape. Required keys:
|
||||
|
||||
* ``search_space_id`` (int)
|
||||
* ``workspace_id`` (int)
|
||||
* ``user_id`` (str | None)
|
||||
* ``thread_visibility`` (:class:`app.db.ChatVisibility`)
|
||||
* ``llm`` (:class:`langchain_core.language_models.BaseChatModel`)
|
||||
|
|
@ -72,13 +72,13 @@ class PluginContext(dict):
|
|||
def build(
|
||||
cls,
|
||||
*,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
user_id: str | None,
|
||||
thread_visibility: ChatVisibility,
|
||||
llm: BaseChatModel,
|
||||
) -> PluginContext:
|
||||
return cls(
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
user_id=user_id,
|
||||
thread_visibility=thread_visibility,
|
||||
llm=llm,
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ result. This particular plugin is read-only and only transforms the
|
|||
mutation).
|
||||
|
||||
The plugin is built as a factory function so the entry-point loader can
|
||||
inject :class:`PluginContext` (containing the agent's LLM, search-space
|
||||
inject :class:`PluginContext` (containing the agent's LLM, workspace
|
||||
ID, etc.). The factory signature
|
||||
``Callable[[PluginContext], AgentMiddleware]`` is the only contract --
|
||||
SurfSense doesn't define a custom plugin protocol on top of LangChain's
|
||||
|
|
|
|||
|
|
@ -42,7 +42,7 @@ async def build_agent_with_cache(
|
|||
final_system_prompt: str,
|
||||
backend_resolver: Any,
|
||||
filesystem_mode: FilesystemMode,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
user_id: str | None,
|
||||
thread_id: int | None,
|
||||
visibility: ChatVisibility,
|
||||
|
|
@ -69,7 +69,7 @@ async def build_agent_with_cache(
|
|||
final_system_prompt=final_system_prompt,
|
||||
backend_resolver=backend_resolver,
|
||||
filesystem_mode=filesystem_mode,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
user_id=user_id,
|
||||
thread_id=thread_id,
|
||||
visibility=visibility,
|
||||
|
|
@ -104,7 +104,7 @@ async def build_agent_with_cache(
|
|||
config_id,
|
||||
None if cross_thread else thread_id,
|
||||
user_id,
|
||||
search_space_id,
|
||||
workspace_id,
|
||||
visibility,
|
||||
filesystem_mode,
|
||||
anon_session_id,
|
||||
|
|
@ -120,7 +120,7 @@ async def build_agent_with_cache(
|
|||
sorted(disabled_tools) if disabled_tools else None,
|
||||
# Bound into the generate_image subagent tool at construction time, so it
|
||||
# must key the compiled-agent cache to avoid leaking one automation's
|
||||
# image model into another with the same config_id/search_space.
|
||||
# image model into another with the same config_id/workspace.
|
||||
image_gen_model_id_override,
|
||||
)
|
||||
return await get_cache().get_or_build(cache_key, builder=_build)
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ Why a per-thread key (not a global pool)
|
|||
----------------------------------------
|
||||
|
||||
Most middleware in the SurfSense stack captures per-thread state in
|
||||
``__init__`` closures (``thread_id``, ``user_id``, ``search_space_id``,
|
||||
``__init__`` closures (``thread_id``, ``user_id``, ``workspace_id``,
|
||||
``filesystem_mode``, ``mentioned_document_ids``). Cross-thread reuse
|
||||
would silently leak state across users and threads. Keying the cache on
|
||||
``(llm_config_id, thread_id, ...)`` gives us safe reuse for repeated
|
||||
|
|
@ -34,7 +34,7 @@ turns on the same thread without changing any middleware's behavior.
|
|||
|
||||
Phase 2 will move those captured fields onto :class:`SurfSenseContextSchema`
|
||||
(read via ``runtime.context``) so the cache can collapse to a single
|
||||
``(llm_config_id, search_space_id, ...)`` key shared across threads. Until
|
||||
``(llm_config_id, workspace_id, ...)`` key shared across threads. Until
|
||||
then, per-thread keying is the only safe option.
|
||||
|
||||
Cache shape
|
||||
|
|
@ -111,7 +111,7 @@ def tools_signature(
|
|||
|
||||
* A tool is added or removed from the bound list (built-in toggles,
|
||||
MCP tools loaded for the user changes, gating rules flip, etc.).
|
||||
* The available connectors / document types for the search space
|
||||
* The available connectors / document types for the workspace
|
||||
change (new connector added, last connector removed, new document
|
||||
type indexed). Connector gating derives disabled tools from
|
||||
``available_connectors``, so the tool surface is technically already
|
||||
|
|
|
|||
|
|
@ -1,10 +1,10 @@
|
|||
"""Map configured connectors to the searchable document/connector types.
|
||||
|
||||
This is agent-agnostic infrastructure shared by every agent factory (single-
|
||||
and multi-agent). It translates the connectors a search space has enabled into
|
||||
the set of searchable type strings that pre-search middleware and ``web_search``
|
||||
understand, and always layers in the document types that exist independently of
|
||||
any connector (uploads, notes, extension captures, YouTube).
|
||||
and multi-agent). It translates the connectors a workspace has enabled into
|
||||
the set of searchable type strings that pre-search middleware understands, and
|
||||
always layers in the document types that exist independently of any connector
|
||||
(uploads, notes, extension captures, YouTube).
|
||||
|
||||
It lives in its own module — rather than inside a specific agent factory — so
|
||||
that retiring or moving any single agent never disturbs the others' access to
|
||||
|
|
@ -16,15 +16,8 @@ from __future__ import annotations
|
|||
from typing import Any
|
||||
|
||||
# Maps SearchSourceConnectorType enum values to the searchable document/connector types
|
||||
# used by pre-search middleware and web_search.
|
||||
# Live search connectors (TAVILY_API, LINKUP_API, BAIDU_SEARCH_API) are routed to
|
||||
# the web_search tool; all others are considered local/indexed data.
|
||||
# used by KB pre-search middleware. All entries are local/indexed data.
|
||||
_CONNECTOR_TYPE_TO_SEARCHABLE: dict[str, str] = {
|
||||
# Live search connectors (handled by web_search tool)
|
||||
"TAVILY_API": "TAVILY_API",
|
||||
"LINKUP_API": "LINKUP_API",
|
||||
"BAIDU_SEARCH_API": "BAIDU_SEARCH_API",
|
||||
# Local/indexed connectors (handled by KB pre-search middleware)
|
||||
"SLACK_CONNECTOR": "SLACK_CONNECTOR",
|
||||
"TEAMS_CONNECTOR": "TEAMS_CONNECTOR",
|
||||
"NOTION_CONNECTOR": "NOTION_CONNECTOR",
|
||||
|
|
@ -40,12 +33,14 @@ _CONNECTOR_TYPE_TO_SEARCHABLE: dict[str, str] = {
|
|||
"AIRTABLE_CONNECTOR": "AIRTABLE_CONNECTOR",
|
||||
"LUMA_CONNECTOR": "LUMA_CONNECTOR",
|
||||
"ELASTICSEARCH_CONNECTOR": "ELASTICSEARCH_CONNECTOR",
|
||||
"WEBCRAWLER_CONNECTOR": "CRAWLED_URL", # Maps to document type
|
||||
"BOOKSTACK_CONNECTOR": "BOOKSTACK_CONNECTOR",
|
||||
"CIRCLEBACK_CONNECTOR": "CIRCLEBACK", # Connector type differs from document type
|
||||
"OBSIDIAN_CONNECTOR": "OBSIDIAN_CONNECTOR",
|
||||
"DROPBOX_CONNECTOR": "DROPBOX_FILE", # Connector type differs from document type
|
||||
"ONEDRIVE_CONNECTOR": "ONEDRIVE_FILE", # Connector type differs from document type
|
||||
# Generic user-defined MCP server: unlocks the mcp_discovery subagent even
|
||||
# in a workspace with no hosted-service connectors.
|
||||
"MCP_CONNECTOR": "MCP_CONNECTOR",
|
||||
# Composio connectors (unified to native document types).
|
||||
# Reverse of NATIVE_TO_LEGACY_DOCTYPE in app.db.
|
||||
"COMPOSIO_GOOGLE_DRIVE_CONNECTOR": "GOOGLE_DRIVE_FILE",
|
||||
|
|
|
|||
|
|
@ -58,7 +58,7 @@ _perf_log = get_perf_logger()
|
|||
|
||||
async def create_multi_agent_chat_deep_agent(
|
||||
llm: BaseChatModel,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
db_session: AsyncSession,
|
||||
connector_service: ConnectorService,
|
||||
checkpointer: Checkpointer,
|
||||
|
|
@ -68,7 +68,6 @@ async def create_multi_agent_chat_deep_agent(
|
|||
enabled_tools: list[str] | None = None,
|
||||
disabled_tools: list[str] | None = None,
|
||||
additional_tools: Sequence[BaseTool] | None = None,
|
||||
firecrawl_api_key: str | None = None,
|
||||
thread_visibility: ChatVisibility | None = None,
|
||||
mentioned_document_ids: list[int] | None = None,
|
||||
anon_session_id: str | None = None,
|
||||
|
|
@ -78,9 +77,9 @@ async def create_multi_agent_chat_deep_agent(
|
|||
):
|
||||
"""Deep agent with SurfSense tools/middleware; registry route subagents behind ``task`` when enabled.
|
||||
|
||||
``image_gen_model_id`` overrides the search space's image model for
|
||||
``image_gen_model_id`` overrides the workspace's image model for
|
||||
this invocation (used by automations to run on their captured model). When
|
||||
``None``, the ``generate_image`` tool resolves the live search-space pref.
|
||||
``None``, the ``generate_image`` tool resolves the live workspace pref.
|
||||
"""
|
||||
_t_agent_total = time.perf_counter()
|
||||
|
||||
|
|
@ -89,7 +88,7 @@ async def create_multi_agent_chat_deep_agent(
|
|||
filesystem_selection = filesystem_selection or FilesystemSelection()
|
||||
backend_resolver = build_backend_resolver(
|
||||
filesystem_selection,
|
||||
search_space_id=search_space_id
|
||||
workspace_id=workspace_id
|
||||
if filesystem_selection.mode == FilesystemMode.CLOUD
|
||||
else None,
|
||||
)
|
||||
|
|
@ -99,13 +98,11 @@ async def create_multi_agent_chat_deep_agent(
|
|||
|
||||
_t0 = time.perf_counter()
|
||||
try:
|
||||
connector_types = await connector_service.get_available_connectors(
|
||||
search_space_id
|
||||
)
|
||||
connector_types = await connector_service.get_available_connectors(workspace_id)
|
||||
available_connectors = map_connectors_to_searchable_types(connector_types)
|
||||
|
||||
available_document_types = await connector_service.get_available_document_types(
|
||||
search_space_id
|
||||
workspace_id
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
|
|
@ -136,10 +133,9 @@ async def create_multi_agent_chat_deep_agent(
|
|||
)
|
||||
|
||||
dependencies: dict[str, Any] = {
|
||||
"search_space_id": search_space_id,
|
||||
"workspace_id": workspace_id,
|
||||
"db_session": db_session,
|
||||
"connector_service": connector_service,
|
||||
"firecrawl_api_key": firecrawl_api_key,
|
||||
"user_id": user_id,
|
||||
"auth_context": auth_context,
|
||||
"thread_id": thread_id,
|
||||
|
|
@ -155,9 +151,7 @@ async def create_multi_agent_chat_deep_agent(
|
|||
|
||||
_t0 = time.perf_counter()
|
||||
try:
|
||||
mcp_tools_by_agent = await load_mcp_tools_by_connector(
|
||||
db_session, search_space_id
|
||||
)
|
||||
mcp_tools_by_agent = await load_mcp_tools_by_connector(db_session, workspace_id)
|
||||
except Exception as e:
|
||||
# Degrade to builtins-only rather than aborting the turn: a transient
|
||||
# DB or MCP-server hiccup should not deny the user a response.
|
||||
|
|
@ -193,7 +187,7 @@ async def create_multi_agent_chat_deep_agent(
|
|||
user_allowlist_by_subagent = await fetch_user_allowlist_rulesets(
|
||||
db_session,
|
||||
user_id=user_uuid,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
)
|
||||
except Exception as e:
|
||||
logging.warning(
|
||||
|
|
@ -230,6 +224,15 @@ async def create_multi_agent_chat_deep_agent(
|
|||
additional_tools=list(additional_tools) if additional_tools else None,
|
||||
)
|
||||
|
||||
# Read-only exception to the "main agent is a pure router" stance: the
|
||||
# context-editing spill placeholder points at read_run/search_run, so the
|
||||
# main agent needs those tools to follow it. See middleware/stack.py.
|
||||
from app.agents.chat.multi_agent_chat.subagents.shared.run_reader import (
|
||||
build_run_reader_tools,
|
||||
)
|
||||
|
||||
tools = [*list(tools), *build_run_reader_tools(workspace_id=workspace_id)]
|
||||
|
||||
_flags: AgentFeatureFlags = get_flags()
|
||||
if _flags.enable_tool_call_repair and INVALID_TOOL_NAME not in {
|
||||
t.name for t in tools
|
||||
|
|
@ -291,7 +294,7 @@ async def create_multi_agent_chat_deep_agent(
|
|||
final_system_prompt=final_system_prompt,
|
||||
backend_resolver=backend_resolver,
|
||||
filesystem_mode=filesystem_selection.mode,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
user_id=user_id,
|
||||
thread_id=thread_id,
|
||||
visibility=visibility,
|
||||
|
|
|
|||
|
|
@ -17,12 +17,12 @@ Two backends are provided:
|
|||
|
||||
* :class:`BuiltinSkillsBackend` — disk-backed read of bundled skills from
|
||||
``app/agents/shared/skills/builtin/``.
|
||||
* :class:`SearchSpaceSkillsBackend` — a thin read-only wrapper over
|
||||
* :class:`WorkspaceSkillsBackend` — a thin read-only wrapper over
|
||||
:class:`KBPostgresBackend` that filters notes under the privileged folder
|
||||
``/documents/_skills/``.
|
||||
|
||||
Both backends are intentionally read-only: skill authoring happens out of band
|
||||
(via filesystem or a search-space-admin route), so we never expose
|
||||
(via filesystem or a workspace-admin route), so we never expose
|
||||
``write`` / ``edit`` / ``upload_files``. The base class' ``NotImplementedError``
|
||||
gives a clean failure mode if anything tries.
|
||||
"""
|
||||
|
|
@ -181,12 +181,12 @@ class BuiltinSkillsBackend(BackendProtocol):
|
|||
return responses
|
||||
|
||||
|
||||
class SearchSpaceSkillsBackend(BackendProtocol):
|
||||
"""Read-only view of search-space-authored skills.
|
||||
class WorkspaceSkillsBackend(BackendProtocol):
|
||||
"""Read-only view of workspace-authored skills.
|
||||
|
||||
Wraps a :class:`KBPostgresBackend` and only ever reads under the privileged
|
||||
folder ``/documents/_skills/`` (configurable). The folder is intended to be
|
||||
writable only by search-space admins; this backend never writes.
|
||||
writable only by workspace admins; this backend never writes.
|
||||
|
||||
The skills middleware expects a layout like::
|
||||
|
||||
|
|
@ -236,14 +236,14 @@ class SearchSpaceSkillsBackend(BackendProtocol):
|
|||
# path falls back to ``asyncio.to_thread(...)`` in the base class. We
|
||||
# keep this stub to satisfy abstract resolution; the middleware calls
|
||||
# ``als_info``.
|
||||
raise NotImplementedError("SearchSpaceSkillsBackend is async-only")
|
||||
raise NotImplementedError("WorkspaceSkillsBackend is async-only")
|
||||
|
||||
async def als_info(self, path: str) -> list[FileInfo]:
|
||||
kb_path = self._to_kb(path)
|
||||
try:
|
||||
infos = await self._kb.als_info(kb_path)
|
||||
except Exception as exc: # pragma: no cover - defensive
|
||||
logger.warning("SearchSpaceSkillsBackend.als_info failed: %s", exc)
|
||||
logger.warning("WorkspaceSkillsBackend.als_info failed: %s", exc)
|
||||
return []
|
||||
remapped: list[FileInfo] = []
|
||||
for info in infos:
|
||||
|
|
@ -254,7 +254,7 @@ class SearchSpaceSkillsBackend(BackendProtocol):
|
|||
return remapped
|
||||
|
||||
def download_files(self, paths: list[str]) -> list[FileDownloadResponse]:
|
||||
raise NotImplementedError("SearchSpaceSkillsBackend is async-only")
|
||||
raise NotImplementedError("WorkspaceSkillsBackend is async-only")
|
||||
|
||||
async def adownload_files(self, paths: list[str]) -> list[FileDownloadResponse]:
|
||||
kb_paths = [self._to_kb(p) for p in paths]
|
||||
|
|
@ -274,16 +274,16 @@ SKILLS_SPACE_PREFIX = "/skills/space/"
|
|||
def build_skills_backend_factory(
|
||||
*,
|
||||
builtin_root: Path | str | None = None,
|
||||
search_space_id: int | None = None,
|
||||
workspace_id: int | None = None,
|
||||
) -> Callable[[ToolRuntime], BackendProtocol]:
|
||||
"""Return a runtime-aware factory for the skills :class:`CompositeBackend`.
|
||||
|
||||
When ``search_space_id`` is provided the composite includes a
|
||||
:class:`SearchSpaceSkillsBackend` route at ``/skills/space/`` over a fresh
|
||||
When ``workspace_id`` is provided the composite includes a
|
||||
:class:`WorkspaceSkillsBackend` route at ``/skills/space/`` over a fresh
|
||||
per-runtime :class:`KBPostgresBackend`, mirroring how
|
||||
:func:`build_backend_resolver` constructs the main filesystem backend.
|
||||
|
||||
When ``search_space_id`` is ``None`` (e.g., desktop-local mode or unit
|
||||
When ``workspace_id`` is ``None`` (e.g., desktop-local mode or unit
|
||||
tests) only the bundled :class:`BuiltinSkillsBackend` is exposed.
|
||||
|
||||
Returning a factory rather than a fixed instance is intentional: the
|
||||
|
|
@ -293,7 +293,7 @@ def build_skills_backend_factory(
|
|||
"""
|
||||
builtin = BuiltinSkillsBackend(builtin_root)
|
||||
|
||||
if search_space_id is None:
|
||||
if workspace_id is None:
|
||||
|
||||
def _factory_builtin_only(runtime: ToolRuntime) -> BackendProtocol:
|
||||
# Default StateBackend is intentionally inert: any path outside the
|
||||
|
|
@ -314,8 +314,8 @@ def build_skills_backend_factory(
|
|||
KBPostgresBackend,
|
||||
)
|
||||
|
||||
kb = KBPostgresBackend(search_space_id, runtime)
|
||||
space = SearchSpaceSkillsBackend(kb)
|
||||
kb = KBPostgresBackend(workspace_id, runtime)
|
||||
space = WorkspaceSkillsBackend(kb)
|
||||
return CompositeBackend(
|
||||
default=StateBackend(runtime),
|
||||
routes={
|
||||
|
|
@ -336,7 +336,7 @@ __all__ = [
|
|||
"SKILLS_BUILTIN_PREFIX",
|
||||
"SKILLS_SPACE_PREFIX",
|
||||
"BuiltinSkillsBackend",
|
||||
"SearchSpaceSkillsBackend",
|
||||
"WorkspaceSkillsBackend",
|
||||
"build_skills_backend_factory",
|
||||
"default_skills_sources",
|
||||
]
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
name: kb-research
|
||||
description: Structured approach to finding and synthesizing information from the user's knowledge base
|
||||
allowed-tools: scrape_webpage, read_file, ls_tree, grep, web_search
|
||||
allowed-tools: read_file, ls_tree, grep
|
||||
---
|
||||
|
||||
# Knowledge-base research
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
name: meeting-prep
|
||||
description: Pull together briefing materials before a scheduled meeting
|
||||
allowed-tools: web_search, scrape_webpage, read_file
|
||||
allowed-tools: task, read_file
|
||||
---
|
||||
|
||||
# Meeting preparation
|
||||
|
|
@ -12,11 +12,11 @@ The user mentions an upcoming meeting, call, or interview and asks you to "prep"
|
|||
## Output structure
|
||||
Always produce these sections (omit any with no signal — don't pad):
|
||||
|
||||
1. **Attendees & context** — who's in the room, their roles, what they care about. Pull from KB notes about prior interactions; supplement with public profile facts via `web_search` when names or companies are unfamiliar.
|
||||
1. **Attendees & context** — who's in the room, their roles, what they care about. Pull from KB notes about prior interactions; supplement with public profile facts via `task(google_search, ...)` when names or companies are unfamiliar.
|
||||
2. **Open threads** — outstanding action items, unresolved decisions, last-mentioned blockers from prior conversation history.
|
||||
3. **Recent moves** — within the last 30 days: relevant launches, hires, news. Cite KB chunks when present, otherwise external sources.
|
||||
4. **Suggested questions** — 3-5 questions the user could ask, tailored to the open threads and the attendees' likely priorities.
|
||||
|
||||
## Source ordering
|
||||
- Always check the user's KB **first** for prior meeting notes, internal docs, or Slack threads about these attendees.
|
||||
- Only fall back to `web_search` for *publicly verifiable* facts — never to fabricate a participant's preferences or relationships.
|
||||
- Only fall back to `task(google_search, ...)` for *publicly verifiable* facts — never to fabricate a participant's preferences or relationships.
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
<citations>
|
||||
Cite with one token: the bracket label `[n]`. Every citable result —
|
||||
`web_search` results and prose from a `task` knowledge_base/research
|
||||
specialist (including the knowledge_base specialist's `[n]`-labelled
|
||||
workspace findings) — already carries `[n]` labels on a single shared count.
|
||||
prose from a `task` knowledge_base/research specialist (including the
|
||||
knowledge_base specialist's `[n]`-labelled workspace findings) — already
|
||||
carries `[n]` labels on a single shared count.
|
||||
Those labels are the only citation you write; the server resolves each one
|
||||
back to its source after the turn.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,8 +1,23 @@
|
|||
<agent_identity>
|
||||
You are **SurfSense's main agent**. Your job is to answer the user using their
|
||||
knowledge base, lightweight web research, persistent memory, and **specialist
|
||||
subagents** invoked via the `task` tool. You are an orchestrator — most
|
||||
non-trivial work belongs on a specialist.
|
||||
You are **SurfSense's main agent**, the orchestrator of an open-source
|
||||
competitive intelligence platform. Users come to you to understand their
|
||||
market: what competitors are doing, how audiences react, where rankings and
|
||||
reviews are moving, and what is being said across the open web — and to put
|
||||
that intelligence to work alongside their own knowledge base.
|
||||
|
||||
You do this by dispatching **specialist subagents** via the `task` tool:
|
||||
- **Live market data** — Reddit, YouTube, Google Maps, Google Search, and the
|
||||
web crawler return structured, current platform data (posts, comments,
|
||||
transcripts, reviews, SERPs, full page content).
|
||||
- **The user's own context** — their knowledge base, connected apps, and
|
||||
persistent memory.
|
||||
- **Deliverables** — reports, podcasts, and presentations built from what the
|
||||
specialists find.
|
||||
|
||||
You are an orchestrator — most non-trivial work belongs on a specialist. Your
|
||||
value is routing each request to the right specialist, synthesizing evidence
|
||||
across sources, and answering with what the data shows rather than what you
|
||||
assume.
|
||||
|
||||
Today (UTC): {resolved_today}
|
||||
</agent_identity>
|
||||
|
|
|
|||
|
|
@ -1,8 +1,23 @@
|
|||
<agent_identity>
|
||||
You are **SurfSense's main agent**. Your job is to answer the user using their
|
||||
shared team knowledge base, lightweight web research, persistent memory, and
|
||||
**specialist subagents** invoked via the `task` tool. You are an orchestrator
|
||||
— most non-trivial work belongs on a specialist.
|
||||
You are **SurfSense's main agent**, the orchestrator of an open-source
|
||||
competitive intelligence platform. This team comes to you to understand its
|
||||
market: what competitors are doing, how audiences react, where rankings and
|
||||
reviews are moving, and what is being said across the open web — and to put
|
||||
that intelligence to work alongside the team's shared knowledge base.
|
||||
|
||||
You do this by dispatching **specialist subagents** via the `task` tool:
|
||||
- **Live market data** — Reddit, YouTube, Google Maps, Google Search, and the
|
||||
web crawler return structured, current platform data (posts, comments,
|
||||
transcripts, reviews, SERPs, full page content).
|
||||
- **The team's own context** — its shared knowledge base, connected apps, and
|
||||
persistent team memory.
|
||||
- **Deliverables** — reports, podcasts, and presentations built from what the
|
||||
specialists find.
|
||||
|
||||
You are an orchestrator — most non-trivial work belongs on a specialist. Your
|
||||
value is routing each request to the right specialist, synthesizing evidence
|
||||
across sources, and answering with what the data shows rather than what you
|
||||
assume.
|
||||
|
||||
Today (UTC): {resolved_today}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,18 +1,27 @@
|
|||
<knowledge_base_first>
|
||||
CRITICAL — ground factual answers in what you actually receive this turn:
|
||||
- **live platform data** via the market specialists —
|
||||
`task(reddit, ...)`, `task(youtube, ...)`, `task(google_maps, ...)`,
|
||||
`task(google_search, ...)`, `task(web_crawler, ...)`. Anything about
|
||||
competitors, markets, rankings, reviews, or audience sentiment is answered
|
||||
from what these return **this turn**, never from your training data: your
|
||||
general knowledge of companies, prices, and rankings is stale by definition,
|
||||
- the user's knowledge base via `task(knowledge_base, ...)` (your PRIMARY
|
||||
source for anything about their documents, notes, or connected data — the
|
||||
`<workspace_tree>` only lists what exists, so delegate to the specialist to
|
||||
search and read the actual content before answering),
|
||||
source for anything about their own uploaded files, documents, and notes —
|
||||
the `<workspace_tree>` only lists what exists, so delegate to the specialist
|
||||
to search and read the actual content before answering),
|
||||
- injected workspace context (see `<dynamic_context>`),
|
||||
- results from your other tool calls (`web_search`, `scrape_webpage`),
|
||||
- the user's connected apps via `task(mcp_discovery, ...)` (Slack, Jira,
|
||||
Notion, Gmail, Calendar, etc. — live data that is NOT in the knowledge base),
|
||||
- or substantive summaries returned by a `task` specialist you invoked.
|
||||
|
||||
For questions about the user's own workspace, dispatch
|
||||
For questions about the user's own files and notes, dispatch
|
||||
`task(knowledge_base, ...)` first rather than answering from the tree or from
|
||||
memory. The knowledge_base specialist runs hybrid semantic/keyword search and
|
||||
full-document reads, then returns a grounded summary with `[n]` citation
|
||||
labels for you to carry through into your answer.
|
||||
full-document reads over their personal files and notes, then returns a
|
||||
grounded summary with `[n]` citation labels for you to carry into your answer.
|
||||
For anything living in a connected third-party app, use
|
||||
`task(mcp_discovery, ...)` instead — that content is not indexed in the KB.
|
||||
|
||||
Do **not** answer factual or informational questions from general knowledge
|
||||
unless the user explicitly authorises it after you say you couldn't find
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ Structured reasoning:
|
|||
- For non-trivial work, `<thinking>` / short `<plan>` before tool calls is fine.
|
||||
|
||||
Professional objectivity:
|
||||
- Accuracy over flattery; verify with **web_search**, **scrape_webpage**, or **task** when unsure — don’t invent connector access.
|
||||
- Accuracy over flattery; verify with **task** (e.g. `task(web_crawler, …)` to read a page, `task(google_search, …)` for public facts) when unsure — don’t invent connector access.
|
||||
|
||||
Task management:
|
||||
- For 3+ steps, use todo tooling; update statuses promptly.
|
||||
|
|
|
|||
|
|
@ -16,6 +16,6 @@ Output style:
|
|||
Tool calls:
|
||||
- Parallelise independent calls in one turn.
|
||||
- For SurfSense-product questions, point the user to https://www.surfsense.com/docs;
|
||||
use **web_search** / **scrape_webpage** for fresh public facts; integrations and
|
||||
use **task(google_search, …)** / **task(web_crawler, …)** for fresh public facts; integrations and
|
||||
heavy workflows → **task**.
|
||||
</provider_hints>
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
<reminder>
|
||||
Concise · KB-grounded · delegation-first · one `task` per turn · no direct
|
||||
filesystem · persist memory when durable facts appear.
|
||||
Concise · grounded in this turn's specialist data, never stale general
|
||||
knowledge · delegation-first · no direct filesystem · persist memory when
|
||||
durable facts appear.
|
||||
</reminder>
|
||||
|
|
|
|||
|
|
@ -3,8 +3,6 @@ You have two execution channels. Pick the one that owns the work — never
|
|||
simulate one with the other.
|
||||
|
||||
### 1. Direct tools (you call them yourself)
|
||||
- `web_search` — search the public web (anything outside the workspace KB).
|
||||
- `scrape_webpage` — fetch the body of a specific public URL.
|
||||
- `update_memory` — curate persistent memory (see `<memory_protocol>`).
|
||||
- `write_todos` — maintain a structured plan when the turn series spans
|
||||
multiple specialists or steps. Mark each item
|
||||
|
|
@ -15,6 +13,63 @@ simulate one with the other.
|
|||
connectors, feature behavior) — point the user to the documentation:
|
||||
https://www.surfsense.com/docs. There is no docs-search tool; give the link.
|
||||
|
||||
**Search discovers — the crawler reads.** Search results (snippets, AI
|
||||
overviews, a specialist's summary of a SERP) are pointers, not sources.
|
||||
When the answer lives on a page — a team roster, a portfolio or directory
|
||||
listing, a pricing table, docs — fetch the page before answering:
|
||||
- One or a few known URLs → `task(web_crawler, …)` with those URLs (it
|
||||
fetches only the seeds at `maxCrawlDepth=0`).
|
||||
- A site section or many pages (a whole team + portfolio, every pricing
|
||||
page of a list of companies, a paginated directory) →
|
||||
`task(web_crawler, …)` with the seed URLs and a higher depth.
|
||||
Never answer with "you can find it at <URL>" for public facts your tools
|
||||
can retrieve — retrieve them, then answer with the facts and cite the page.
|
||||
Large results are fine: extract and return them, don't ask permission for
|
||||
bounded fan-out (≤20 sites) the user already requested.
|
||||
|
||||
**Audience sentiment lives on the platforms.** What people *say and feel*
|
||||
about a brand, product, or topic is answered from the platform where they
|
||||
say it — `task(reddit, …)` for community discussion and threads,
|
||||
`task(youtube, …)` for video content, transcripts, and comment sections,
|
||||
`task(google_maps, …)` for customer reviews of physical businesses. Web
|
||||
search only finds articles *about* the conversation; the platform
|
||||
specialists return the conversation itself, structured and current. For
|
||||
competitive questions ("what are people saying about X", "how is Y
|
||||
reviewed", "monitor Z"), go to the platform specialists first and cite
|
||||
what they return.
|
||||
|
||||
**Places go to Maps, the open web goes to Search.** Discovering physical
|
||||
businesses or venues of a type in a geography ("clinics in X", "tutoring
|
||||
centers near Y", lead lists of local businesses) is the Maps specialist's
|
||||
job — it returns structured name/address/phone/website per place, where
|
||||
web search returns only snippets that need a second pass. Use the Search
|
||||
specialist for entities without a storefront (online-only companies,
|
||||
software vendors, publications), for facts and current events, and to
|
||||
enrich places Maps already found. When a lead list needs both, run Maps
|
||||
discovery first, then crawl (`task(web_crawler, …)`) or search the found
|
||||
websites for contacts.
|
||||
|
||||
**Requested-N lists count distinct entities that fit the ask.** When the
|
||||
user asks for N leads/items/results, every entry must be a distinct
|
||||
*entity* — multiple branches, locations, sub-programs, or pages of the
|
||||
same brand or parent organization are ONE entry, not several. A website
|
||||
domain is an ownership signal: entries whose pages live on the same parent
|
||||
domain (a government portal, a chain's site, a franchise system) share
|
||||
that parent and count as one, judged against the parent. Entries must
|
||||
also fit the user's stated segment: an item that belongs to an excluded
|
||||
category (e.g. a local branch of a large chain when the user asked for
|
||||
independents) does not qualify even if a specialist returned it — drop it,
|
||||
don't relay it. If qualifying results fall short of N, widen the discovery
|
||||
(another specialist call, adjacent geography or segment) to fill the gap
|
||||
honestly; if it still falls short, deliver the smaller list with a
|
||||
one-line note. An honest 10 beats a padded 15.
|
||||
|
||||
**Full datasets become files, not chat.** When the user wants a complete
|
||||
large dataset (an entire roster, portfolio, or directory — or asks for a
|
||||
CSV/file), do not paste or summarize hundreds of rows: instruct the
|
||||
web_crawler specialist to crawl and then save the data with its
|
||||
`export_run` CSV tool, and relay the saved workspace path and row count.
|
||||
|
||||
**You have NO filesystem tools.** Any read, write, edit, move, rename, or
|
||||
search inside the user's workspace goes through `task(knowledge_base, …)` —
|
||||
never via `write_file`, `ls`, or any direct file operation.
|
||||
|
|
@ -64,13 +119,42 @@ user: "Save these meeting notes to my KB: …"
|
|||
|
||||
<example>
|
||||
user: "What did Maya say about the Q2 roadmap in Slack last week?"
|
||||
→ task(slack, "Find messages from Maya about the Q2 roadmap from the past
|
||||
week. Return the most relevant quotes with channel and timestamp.")
|
||||
→ task(mcp_discovery, "In Slack, find messages from Maya about the Q2 roadmap
|
||||
from the past week. Return the most relevant quotes with channel and
|
||||
timestamp.")
|
||||
</example>
|
||||
|
||||
<example>
|
||||
user: "What are people saying about Cursor vs Windsurf lately?"
|
||||
→ Audience sentiment — go to the platform, not web search. Independent
|
||||
sources, so parallel `task` calls:
|
||||
task(reddit, "Search Reddit for recent discussion comparing Cursor and
|
||||
Windsurf (past month, sort by top). Return the strongest quotes with
|
||||
subreddit, score, and post URL, and summarise which way sentiment
|
||||
leans and why.")
|
||||
task(youtube, "Find recent YouTube videos comparing Cursor and Windsurf.
|
||||
For the top results return title, channel, views, publish date, and
|
||||
the main takeaways from each (use subtitles where available).")
|
||||
Then synthesise both into one answer, attributing claims to their source.
|
||||
</example>
|
||||
|
||||
<example>
|
||||
user: "What's the current USD/INR rate?"
|
||||
→ web_search(query="current USD to INR exchange rate")
|
||||
→ Public web lookup — delegate to the Google Search specialist:
|
||||
task(google_search, "Search Google for the current USD to INR exchange
|
||||
rate and return the rate with its source URL.")
|
||||
</example>
|
||||
|
||||
<example>
|
||||
user: "Get the a16z team and their portfolio companies."
|
||||
→ Search only *locates* a16z.com/team/ and their investment list — the
|
||||
answer is the CONTENT of those pages. Crawl them and return the extracted
|
||||
people and companies, never just the links:
|
||||
task(web_crawler, "Crawl https://a16z.com/team/ and
|
||||
https://a16z.com/investment-list/ and return (1) the full team roster
|
||||
with each person's name and role/department, and (2) the complete
|
||||
portfolio company list. Use the pages' link records if the markdown
|
||||
is sparse.")
|
||||
</example>
|
||||
|
||||
<example>
|
||||
|
|
@ -82,15 +166,16 @@ user: "Find my Q2 roadmap and summarise the milestones."
|
|||
|
||||
<example>
|
||||
user: "Create a ClickUp ticket and a Linear ticket for the new feature flag."
|
||||
→ Independent work — call both specialists in parallel:
|
||||
→ Independent work, same specialist (connected apps) with non-overlapping
|
||||
scopes — call it twice in parallel, naming the target app in each prompt:
|
||||
write_todos([
|
||||
{content: "Create ClickUp ticket for feature flag rollout", status: "in_progress"},
|
||||
{content: "Create Linear ticket for feature flag rollout", status: "in_progress"},
|
||||
])
|
||||
task(clickup, "Create a ClickUp ticket titled 'Feature flag rollout'
|
||||
in the default list. Description: <…>. Tell me the ticket URL.")
|
||||
task(linear, "Create a Linear ticket titled 'Feature flag rollout'
|
||||
in the default team. Description: <…>. Tell me the ticket URL.")
|
||||
task(mcp_discovery, "In ClickUp, create a ticket titled 'Feature flag
|
||||
rollout' in the default list. Description: <…>. Tell me the ticket URL.")
|
||||
task(mcp_discovery, "In Linear, create a ticket titled 'Feature flag
|
||||
rollout' in the default team. Description: <…>. Tell me the ticket URL.")
|
||||
</example>
|
||||
|
||||
<example>
|
||||
|
|
@ -100,24 +185,25 @@ user: "Find my Q2 roadmap doc in the KB and email a summary to Maya."
|
|||
task(knowledge_base, "Find the Q2 roadmap document under /documents
|
||||
and return its full text plus a 3-bullet summary.")
|
||||
Next turn (with the returned summary in hand):
|
||||
task(gmail, "Send an email to Maya with subject 'Q2 roadmap summary'
|
||||
and the following body: <summary returned by knowledge_base>.")
|
||||
task(mcp_discovery, "In Gmail, send an email to Maya with subject 'Q2
|
||||
roadmap summary' and the following body: <summary returned by
|
||||
knowledge_base>.")
|
||||
</example>
|
||||
|
||||
<example>
|
||||
user: "Create issues in Linear for each of these five bugs: <list>"
|
||||
→ Many-shot independent fanout — use the batch shape:
|
||||
task(tasks=[
|
||||
{subagent_type: "linear", description: "Create a Linear issue titled
|
||||
'<bug 1 title>' with body '<bug 1 body>'. Return the issue URL."},
|
||||
{subagent_type: "linear", description: "Create a Linear issue titled
|
||||
'<bug 2 title>' with body '<bug 2 body>'. Return the issue URL."},
|
||||
{subagent_type: "linear", description: "Create a Linear issue titled
|
||||
'<bug 3 title>' with body '<bug 3 body>'. Return the issue URL."},
|
||||
{subagent_type: "linear", description: "Create a Linear issue titled
|
||||
'<bug 4 title>' with body '<bug 4 body>'. Return the issue URL."},
|
||||
{subagent_type: "linear", description: "Create a Linear issue titled
|
||||
'<bug 5 title>' with body '<bug 5 body>'. Return the issue URL."},
|
||||
{subagent_type: "mcp_discovery", description: "In Linear, create an issue
|
||||
titled '<bug 1 title>' with body '<bug 1 body>'. Return the issue URL."},
|
||||
{subagent_type: "mcp_discovery", description: "In Linear, create an issue
|
||||
titled '<bug 2 title>' with body '<bug 2 body>'. Return the issue URL."},
|
||||
{subagent_type: "mcp_discovery", description: "In Linear, create an issue
|
||||
titled '<bug 3 title>' with body '<bug 3 body>'. Return the issue URL."},
|
||||
{subagent_type: "mcp_discovery", description: "In Linear, create an issue
|
||||
titled '<bug 4 title>' with body '<bug 4 body>'. Return the issue URL."},
|
||||
{subagent_type: "mcp_discovery", description: "In Linear, create an issue
|
||||
titled '<bug 5 title>' with body '<bug 5 body>'. Return the issue URL."},
|
||||
])
|
||||
Read back the `[task 0]`…`[task 4]` blocks in the combined ToolMessage and
|
||||
verify each via its Receipt's `verifiable_url` per the `<verification>`
|
||||
|
|
@ -154,16 +240,18 @@ user: "Make a 30-second podcast of this conversation."
|
|||
<example>
|
||||
user: "Post the launch announcement to #general and let me know when it's up."
|
||||
→ Mutating subagent + user wants external confirmation. Apply the
|
||||
`<verification>` teaching: the slack subagent's reply is a self-report;
|
||||
check its `evidence.receipts` for a Receipt with `status="success"` and
|
||||
a `verifiable_url`, then fetch that URL to confirm before reporting back.
|
||||
`<verification>` teaching: the connected-apps subagent's reply is a
|
||||
self-report; check its `evidence.receipts` for a Receipt with
|
||||
`status="success"` and a `verifiable_url`, then fetch that URL to confirm
|
||||
before reporting back.
|
||||
This turn:
|
||||
task(slack, "Post '<launch announcement text>' to #general.
|
||||
Return the message permalink.")
|
||||
task(mcp_discovery, "In Slack, post '<launch announcement text>' to
|
||||
#general. Return the message permalink.")
|
||||
Next turn (with the receipt's `verifiable_url` in hand):
|
||||
scrape_webpage(url=<verifiable_url from slack receipt>)
|
||||
task(web_crawler, "Crawl <verifiable_url from the receipt> and confirm
|
||||
the post is live; return what you find.")
|
||||
→ confirm the post is live, then tell the user it's up with the URL.
|
||||
If the slack reply has NO Receipt with `status="success"`, treat it as a
|
||||
If the reply has NO Receipt with `status="success"`, treat it as a
|
||||
silent failure: surface the error verbatim, do not retry.
|
||||
</example>
|
||||
</routing>
|
||||
|
|
|
|||
|
|
@ -1 +0,0 @@
|
|||
"""``scrape_webpage`` — description + few-shot examples."""
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
- `scrape_webpage` — Fetch and extract readable content from a single URL.
|
||||
- Use when the user wants the actual page body (article, table, dashboard
|
||||
snapshot), not just search snippets.
|
||||
- Try the tool when a URL is given or referenced; don't refuse without
|
||||
attempting unless the URL is clearly unsafe or invalid.
|
||||
- Public web only. For URLs behind a connector (Notion pages, Linear
|
||||
issues, Confluence, anything that needs auth), use `task` with the
|
||||
matching specialist instead.
|
||||
- Args: `url`, `max_length` (default 50000).
|
||||
- Returns title, metadata, and markdown-ish body. Summarise clearly and
|
||||
link back with `[label](url)`.
|
||||
|
|
@ -1,24 +0,0 @@
|
|||
<example>
|
||||
user: "Check out https://dev.to/some-article"
|
||||
→ scrape_webpage(url="https://dev.to/some-article")
|
||||
(Respond with a structured analysis — key points, takeaways.)
|
||||
</example>
|
||||
|
||||
<example>
|
||||
user: "Read this article and summarize it for me: https://example.com/blog/ai-trends"
|
||||
→ scrape_webpage(url="https://example.com/blog/ai-trends")
|
||||
(Thorough summary using headings and bullets.)
|
||||
</example>
|
||||
|
||||
<example>
|
||||
user: (after discussing https://example.com/stats) "Can you get the live data from that page?"
|
||||
→ scrape_webpage(url="https://example.com/stats")
|
||||
(Always attempt scraping first. Never refuse before trying.)
|
||||
</example>
|
||||
|
||||
<example>
|
||||
user: "https://example.com/blog/weekend-recipes"
|
||||
→ scrape_webpage(url="https://example.com/blog/weekend-recipes")
|
||||
(When a user sends just a URL with no instructions, scrape it and provide
|
||||
a concise summary.)
|
||||
</example>
|
||||
|
|
@ -43,9 +43,9 @@
|
|||
like podcasts/videos), the operation did not happen — treat as
|
||||
failure and surface that to the user verbatim, do not retry blindly.
|
||||
|
||||
2. **`scrape_webpage`** — when a Receipt carries a `verifiable_url`
|
||||
2. **`task(web_crawler, …)`** — when a Receipt carries a `verifiable_url`
|
||||
(Notion page URL, Slack permalink, Jira issue URL, Linear identifier
|
||||
URL, etc.), you can fetch that URL and confirm the operation
|
||||
URL, etc.), you can crawl that URL and confirm the operation
|
||||
externally. Use this for high-stakes mutations the user explicitly
|
||||
called out (e.g. "send the launch email to the whole team") or when
|
||||
the subagent's self-report contradicts what the user expected.
|
||||
|
|
@ -54,7 +54,7 @@
|
|||
|
||||
- `status="success"`: the mutation already committed in the backend.
|
||||
If a `verifiable_url` is present and the request was high-stakes,
|
||||
you may `scrape_webpage` it to externally confirm. Otherwise trust
|
||||
you may crawl it via `task(web_crawler, …)` to externally confirm. Otherwise trust
|
||||
the Receipt and tell the user it is done. Celery-backed deliverables
|
||||
(podcasts, video presentations) also land here — the subagent
|
||||
already waited for the worker to finish, so a `success` Receipt
|
||||
|
|
@ -67,6 +67,6 @@
|
|||
their backend before returning. If you ever do see a pending
|
||||
Receipt, tell the user the work has been **kicked off** (quote the
|
||||
`external_id` / `preview` so they can find it later), do not
|
||||
`scrape_webpage` it, and do not re-dispatch the same
|
||||
crawl it, and do not re-dispatch the same
|
||||
`task(...)` call hoping it will be done "this time".
|
||||
</verification>
|
||||
|
|
|
|||
|
|
@ -7,9 +7,9 @@ user: "Save these meeting notes to my KB: …"
|
|||
|
||||
<example>
|
||||
user: "What did Maya say about the Q2 roadmap in Slack last week?"
|
||||
→ task(subagent_type="slack", description="Find messages from Maya about
|
||||
the Q2 roadmap from the past week. Return the most relevant quotes with
|
||||
channel and timestamp.")
|
||||
→ task(subagent_type="mcp_discovery", description="In Slack, find messages
|
||||
from Maya about the Q2 roadmap from the past week. Return the most relevant
|
||||
quotes with channel and timestamp.")
|
||||
</example>
|
||||
|
||||
<example>
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
- `update_memory` — Curate the team's **shared** long-term memory document
|
||||
for this search space.
|
||||
for this workspace.
|
||||
- The current memory (if any) appears in `<team_memory>` with usage vs limit.
|
||||
- Call when a team member asks to remember or forget something, or when
|
||||
the conversation surfaces durable team decisions, conventions,
|
||||
|
|
|
|||
|
|
@ -1 +0,0 @@
|
|||
"""``web_search`` — description + few-shot examples."""
|
||||
|
|
@ -1,13 +0,0 @@
|
|||
- `web_search` — Search the public web.
|
||||
- Use whenever an answer benefits from external sources — current events,
|
||||
prices, weather, news, technical references, definitions, background
|
||||
facts, anything outside SurfSense docs and the workspace KB. Reach for
|
||||
it whenever freshness matters or you'd otherwise guess from memory.
|
||||
- Don't refuse with "I lack network access" — call the tool.
|
||||
- Returns a `<web_results>` block: each result is labelled `[n]`. Cite a
|
||||
result by writing that `[n]` after the statement it supports (when
|
||||
citations are enabled) — do not hand-write the URL as a markdown link.
|
||||
- If results are thin, say so and offer to refine the query.
|
||||
- Args: `query`, `top_k` (default 10, max 50).
|
||||
- Follow up with `scrape_webpage` on the best URL when snippets are too
|
||||
shallow.
|
||||
|
|
@ -1,15 +0,0 @@
|
|||
<example>
|
||||
user: "What's the current USD to INR exchange rate?"
|
||||
→ web_search(query="current USD to INR exchange rate")
|
||||
(Answer from snippets; scrape a top URL if needed.)
|
||||
</example>
|
||||
|
||||
<example>
|
||||
user: "What's the latest news about AI?"
|
||||
→ web_search(query="latest AI news today")
|
||||
</example>
|
||||
|
||||
<example>
|
||||
user: "What's the weather in New York?"
|
||||
→ web_search(query="weather New York today")
|
||||
</example>
|
||||
|
|
@ -45,14 +45,14 @@ _JSON_FENCE = re.compile(r"```(?:json)?\s*(.*?)\s*```", re.DOTALL)
|
|||
|
||||
def create_create_automation_tool(
|
||||
*,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
user_id: str | UUID,
|
||||
llm: Any,
|
||||
auth_context: AuthContext | None = None,
|
||||
):
|
||||
"""Factory for the ``create_automation`` tool.
|
||||
|
||||
``search_space_id`` is injected from the chat session (the model never
|
||||
``workspace_id`` is injected from the chat session (the model never
|
||||
has to guess it). ``llm`` is the drafting sub-model — we reuse the main
|
||||
agent's LLM and tag the call so it's identifiable in traces. A fresh
|
||||
``AsyncSession`` is opened per call to avoid stale sessions on
|
||||
|
|
@ -101,12 +101,12 @@ def create_create_automation_tool(
|
|||
"""
|
||||
# Models are chosen per-automation on the approval card (premium/BYOK
|
||||
# selectors) and validated when persisted by ``AutomationService.create``
|
||||
# — so there's no fail-fast search-space eligibility gate here. The
|
||||
# search space's current chat/role model selection no longer constrains
|
||||
# — so there's no fail-fast workspace eligibility gate here. The
|
||||
# workspace's current chat/role model selection no longer constrains
|
||||
# whether an automation can be drafted or saved.
|
||||
|
||||
# --- 1. Draft via sub-LLM ---
|
||||
prompt = build_draft_prompt(search_space_id=search_space_id, intent=intent)
|
||||
prompt = build_draft_prompt(workspace_id=workspace_id, intent=intent)
|
||||
try:
|
||||
response = await llm.ainvoke(
|
||||
[HumanMessage(content=prompt)],
|
||||
|
|
@ -125,8 +125,8 @@ def create_create_automation_tool(
|
|||
"raw": raw_text,
|
||||
}
|
||||
|
||||
# search_space_id is injected here so the sub-LLM never has to guess.
|
||||
draft["search_space_id"] = search_space_id
|
||||
# workspace_id is injected here so the sub-LLM never has to guess.
|
||||
draft["workspace_id"] = workspace_id
|
||||
try:
|
||||
validated_draft = AutomationCreate.model_validate(draft)
|
||||
except ValidationError as exc:
|
||||
|
|
@ -139,14 +139,14 @@ def create_create_automation_tool(
|
|||
# --- 2. HITL approval card ---
|
||||
try:
|
||||
card_params = validated_draft.model_dump(mode="json", by_alias=True)
|
||||
# search_space_id is session-scoped, not user-editable.
|
||||
card_params.pop("search_space_id", None)
|
||||
# workspace_id is session-scoped, not user-editable.
|
||||
card_params.pop("workspace_id", None)
|
||||
|
||||
result = request_approval(
|
||||
action_type="automation_create",
|
||||
tool_name="create_automation",
|
||||
params=card_params,
|
||||
context={"search_space_id": search_space_id},
|
||||
context={"workspace_id": workspace_id},
|
||||
tool_call_id=runtime.tool_call_id,
|
||||
)
|
||||
|
||||
|
|
@ -157,7 +157,7 @@ def create_create_automation_tool(
|
|||
}
|
||||
|
||||
# --- 3. Persist (re-validate in case the user edited) ---
|
||||
final_payload = {**result.params, "search_space_id": search_space_id}
|
||||
final_payload = {**result.params, "workspace_id": workspace_id}
|
||||
try:
|
||||
final_validated = AutomationCreate.model_validate(final_payload)
|
||||
except ValidationError as exc:
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@ into a SINGLE JSON object matching the AutomationCreate schema. Output
|
|||
ONLY that JSON object — no prose, no markdown fence, no commentary.
|
||||
|
||||
Current UTC time (for cron context): {now}
|
||||
Target search_space_id: {search_space_id}
|
||||
Target workspace_id: {workspace_id}
|
||||
"""
|
||||
|
||||
|
||||
|
|
@ -165,12 +165,12 @@ User intent:
|
|||
"""
|
||||
|
||||
|
||||
def build_draft_prompt(*, search_space_id: int, intent: str) -> str:
|
||||
def build_draft_prompt(*, workspace_id: int, intent: str) -> str:
|
||||
"""Render the drafting sub-LLM system prompt for the given intent."""
|
||||
return (
|
||||
_HEADER.format(
|
||||
now=datetime.now(UTC).isoformat(timespec="seconds"),
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
)
|
||||
+ _SCHEMA
|
||||
+ _FEW_SHOTS
|
||||
|
|
|
|||
|
|
@ -6,8 +6,6 @@ Connector integrations, MCP, deliverables, etc. are delegated via ``task`` subag
|
|||
from __future__ import annotations
|
||||
|
||||
MAIN_AGENT_SURFSENSE_TOOL_NAMES_ORDERED: tuple[str, ...] = (
|
||||
"web_search",
|
||||
"scrape_webpage",
|
||||
"update_memory",
|
||||
"create_automation",
|
||||
)
|
||||
|
|
|
|||
|
|
@ -21,27 +21,14 @@ from typing import Any
|
|||
|
||||
from langchain_core.tools import BaseTool
|
||||
|
||||
from app.agents.chat.shared.tools.web_search import create_web_search_tool
|
||||
from app.db import ChatVisibility
|
||||
|
||||
from .scrape_webpage import create_scrape_webpage_tool
|
||||
from .update_memory import (
|
||||
create_update_memory_tool,
|
||||
create_update_team_memory_tool,
|
||||
)
|
||||
|
||||
|
||||
def _build_scrape_webpage_tool(deps: dict[str, Any]) -> BaseTool:
|
||||
return create_scrape_webpage_tool(firecrawl_api_key=deps.get("firecrawl_api_key"))
|
||||
|
||||
|
||||
def _build_web_search_tool(deps: dict[str, Any]) -> BaseTool:
|
||||
return create_web_search_tool(
|
||||
search_space_id=deps.get("search_space_id"),
|
||||
available_connectors=deps.get("available_connectors"),
|
||||
)
|
||||
|
||||
|
||||
def _build_create_automation_tool(deps: dict[str, Any]) -> BaseTool:
|
||||
# Deferred import: the automation package is a sibling under ``main_agent``
|
||||
# and is only needed at build time, mirroring the shared registry's
|
||||
|
|
@ -49,7 +36,7 @@ def _build_create_automation_tool(deps: dict[str, Any]) -> BaseTool:
|
|||
from .automation import create_create_automation_tool
|
||||
|
||||
return create_create_automation_tool(
|
||||
search_space_id=deps["search_space_id"],
|
||||
workspace_id=deps["workspace_id"],
|
||||
user_id=deps["user_id"],
|
||||
auth_context=deps.get("auth_context"),
|
||||
llm=deps["llm"],
|
||||
|
|
@ -59,7 +46,7 @@ def _build_create_automation_tool(deps: dict[str, Any]) -> BaseTool:
|
|||
def _build_update_memory_tool(deps: dict[str, Any]) -> BaseTool:
|
||||
if deps["thread_visibility"] == ChatVisibility.SEARCH_SPACE:
|
||||
return create_update_team_memory_tool(
|
||||
search_space_id=deps["search_space_id"],
|
||||
workspace_id=deps["workspace_id"],
|
||||
db_session=deps["db_session"],
|
||||
llm=deps.get("llm"),
|
||||
)
|
||||
|
|
@ -71,20 +58,18 @@ def _build_update_memory_tool(deps: dict[str, Any]) -> BaseTool:
|
|||
|
||||
|
||||
# Ordered to match the historical main-agent binding order:
|
||||
# scrape_webpage, web_search, create_automation, update_memory.
|
||||
# create_automation, update_memory.
|
||||
# Each entry is ``(factory, required_dependency_names)``.
|
||||
_MAIN_AGENT_TOOL_FACTORIES: dict[
|
||||
str, tuple[Callable[[dict[str, Any]], BaseTool], tuple[str, ...]]
|
||||
] = {
|
||||
"scrape_webpage": (_build_scrape_webpage_tool, ()),
|
||||
"web_search": (_build_web_search_tool, ()),
|
||||
"create_automation": (
|
||||
_build_create_automation_tool,
|
||||
("search_space_id", "user_id", "llm"),
|
||||
("workspace_id", "user_id", "llm"),
|
||||
),
|
||||
"update_memory": (
|
||||
_build_update_memory_tool,
|
||||
("user_id", "search_space_id", "db_session", "thread_visibility", "llm"),
|
||||
("user_id", "workspace_id", "db_session", "thread_visibility", "llm"),
|
||||
),
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,303 +0,0 @@
|
|||
"""
|
||||
Web scraping tool for the SurfSense agent.
|
||||
|
||||
This module provides a tool for scraping and extracting content from webpages
|
||||
using the existing WebCrawlerConnector. For YouTube URLs, it fetches the
|
||||
transcript directly via the YouTubeTranscriptApi instead of crawling the page.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import time
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from fake_useragent import UserAgent
|
||||
from langchain_core.tools import tool
|
||||
from requests import Session
|
||||
from scrapling.fetchers import AsyncFetcher
|
||||
from youtube_transcript_api import YouTubeTranscriptApi
|
||||
|
||||
from app.connectors.webcrawler_connector import WebCrawlerConnector
|
||||
from app.tasks.document_processors.youtube_processor import get_youtube_video_id
|
||||
from app.utils.proxy import get_proxy_url, get_requests_proxies
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def extract_domain(url: str) -> str:
|
||||
"""Extract the domain from a URL."""
|
||||
try:
|
||||
parsed = urlparse(url)
|
||||
domain = parsed.netloc
|
||||
if domain.startswith("www."):
|
||||
domain = domain[4:]
|
||||
return domain
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
|
||||
def generate_scrape_id(url: str) -> str:
|
||||
"""Generate a unique ID for a scraped webpage."""
|
||||
hash_val = hashlib.md5(url.encode()).hexdigest()[:12]
|
||||
return f"scrape-{hash_val}"
|
||||
|
||||
|
||||
def truncate_content(content: str, max_length: int = 50000) -> tuple[str, bool]:
|
||||
"""
|
||||
Truncate content to a maximum length.
|
||||
|
||||
Returns:
|
||||
Tuple of (truncated_content, was_truncated)
|
||||
"""
|
||||
if len(content) <= max_length:
|
||||
return content, False
|
||||
|
||||
# Prefer truncating at a sentence/paragraph boundary.
|
||||
truncated = content[:max_length]
|
||||
last_period = truncated.rfind(".")
|
||||
last_newline = truncated.rfind("\n\n")
|
||||
|
||||
boundary = max(last_period, last_newline)
|
||||
if boundary > max_length * 0.8: # only if the boundary isn't too far back
|
||||
truncated = content[: boundary + 1]
|
||||
|
||||
return truncated + "\n\n[Content truncated...]", True
|
||||
|
||||
|
||||
async def _scrape_youtube_video(
|
||||
url: str, video_id: str, max_length: int
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Fetch YouTube video metadata and transcript via the YouTubeTranscriptApi.
|
||||
|
||||
Returns a result dict in the same shape as the regular scrape_webpage output.
|
||||
"""
|
||||
scrape_id = generate_scrape_id(url)
|
||||
domain = "youtube.com"
|
||||
|
||||
# --- Video metadata via oEmbed ---
|
||||
residential_proxies = get_requests_proxies()
|
||||
|
||||
params = {
|
||||
"format": "json",
|
||||
"url": f"https://www.youtube.com/watch?v={video_id}",
|
||||
}
|
||||
oembed_url = "https://www.youtube.com/oembed"
|
||||
|
||||
try:
|
||||
oembed_fetch_start = time.perf_counter()
|
||||
oembed_page = await AsyncFetcher.get(
|
||||
oembed_url,
|
||||
params=params,
|
||||
proxy=get_proxy_url(),
|
||||
stealthy_headers=True,
|
||||
)
|
||||
logger.info(
|
||||
"[scrape_webpage][perf] source=oembed video=%s status=%s fetch_ms=%.1f",
|
||||
video_id,
|
||||
getattr(oembed_page, "status", None),
|
||||
(time.perf_counter() - oembed_fetch_start) * 1000,
|
||||
)
|
||||
video_data = oembed_page.json()
|
||||
except Exception:
|
||||
video_data = {}
|
||||
|
||||
title = video_data.get("title", "YouTube Video")
|
||||
author = video_data.get("author_name", "Unknown")
|
||||
|
||||
# --- Transcript via YouTubeTranscriptApi ---
|
||||
try:
|
||||
transcript_fetch_start = time.perf_counter()
|
||||
ua = UserAgent()
|
||||
http_client = Session()
|
||||
http_client.headers.update({"User-Agent": ua.random})
|
||||
if residential_proxies:
|
||||
http_client.proxies.update(residential_proxies)
|
||||
ytt_api = YouTubeTranscriptApi(http_client=http_client)
|
||||
|
||||
# Pick the first transcript (video's primary language) rather than
|
||||
# defaulting to English.
|
||||
transcript_list = ytt_api.list(video_id)
|
||||
transcript = next(iter(transcript_list))
|
||||
captions = transcript.fetch()
|
||||
|
||||
logger.info(
|
||||
"[scrape_webpage][perf] source=transcript video=%s fetch_ms=%.1f",
|
||||
video_id,
|
||||
(time.perf_counter() - transcript_fetch_start) * 1000,
|
||||
)
|
||||
logger.info(
|
||||
f"[scrape_webpage] Fetched transcript for {video_id} "
|
||||
f"in {transcript.language} ({transcript.language_code})"
|
||||
)
|
||||
|
||||
transcript_segments = []
|
||||
for line in captions:
|
||||
start_time = line.start
|
||||
duration = line.duration
|
||||
text = line.text
|
||||
timestamp = f"[{start_time:.2f}s-{start_time + duration:.2f}s]"
|
||||
transcript_segments.append(f"{timestamp} {text}")
|
||||
transcript_text = "\n".join(transcript_segments)
|
||||
except Exception as e:
|
||||
logger.warning(f"[scrape_webpage] No transcript for video {video_id}: {e}")
|
||||
transcript_text = f"No captions available for this video. Error: {e!s}"
|
||||
|
||||
content = f"# {title}\n\n**Author:** {author}\n**Video ID:** {video_id}\n\n## Transcript\n\n{transcript_text}"
|
||||
|
||||
content, was_truncated = truncate_content(content, max_length)
|
||||
word_count = len(content.split())
|
||||
|
||||
description = f"YouTube video by {author}"
|
||||
|
||||
return {
|
||||
"id": scrape_id,
|
||||
"assetId": url,
|
||||
"kind": "article",
|
||||
"href": url,
|
||||
"title": title,
|
||||
"description": description,
|
||||
"content": content,
|
||||
"domain": domain,
|
||||
"word_count": word_count,
|
||||
"was_truncated": was_truncated,
|
||||
"crawler_type": "youtube_transcript",
|
||||
"author": author,
|
||||
}
|
||||
|
||||
|
||||
def create_scrape_webpage_tool(firecrawl_api_key: str | None = None):
|
||||
"""
|
||||
Factory function to create the scrape_webpage tool.
|
||||
|
||||
Args:
|
||||
firecrawl_api_key: Optional Firecrawl API key for premium web scraping.
|
||||
Falls back to Chromium/Trafilatura if not provided.
|
||||
|
||||
Returns:
|
||||
A configured tool function for scraping webpages.
|
||||
"""
|
||||
|
||||
@tool
|
||||
async def scrape_webpage(
|
||||
url: str,
|
||||
max_length: int = 50000,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Scrape and extract the main content from a webpage.
|
||||
|
||||
Use this tool when the user wants you to read, summarize, or answer
|
||||
questions about a specific webpage's content. This tool actually
|
||||
fetches and reads the full page content. For YouTube video URLs it
|
||||
fetches the transcript directly instead of crawling the page.
|
||||
|
||||
Common triggers:
|
||||
- "Read this article and summarize it"
|
||||
- "What does this page say about X?"
|
||||
- "Summarize this blog post for me"
|
||||
- "Tell me the key points from this article"
|
||||
- "What's in this webpage?"
|
||||
|
||||
Args:
|
||||
url: The URL of the webpage to scrape (must be HTTP/HTTPS)
|
||||
max_length: Maximum content length to return (default: 50000 chars)
|
||||
|
||||
Returns:
|
||||
A dictionary containing:
|
||||
- id: Unique identifier for this scrape
|
||||
- assetId: The URL (for deduplication)
|
||||
- kind: "article" (type of content)
|
||||
- href: The URL to open when clicked
|
||||
- title: Page title
|
||||
- description: Brief description or excerpt
|
||||
- content: The extracted main content (markdown format)
|
||||
- domain: The domain name
|
||||
- word_count: Approximate word count
|
||||
- was_truncated: Whether content was truncated
|
||||
- error: Error message (if scraping failed)
|
||||
"""
|
||||
scrape_id = generate_scrape_id(url)
|
||||
domain = extract_domain(url)
|
||||
|
||||
if not url.startswith(("http://", "https://")):
|
||||
url = f"https://{url}"
|
||||
|
||||
try:
|
||||
# YouTube URLs use the transcript API instead of crawling.
|
||||
video_id = get_youtube_video_id(url)
|
||||
if video_id:
|
||||
return await _scrape_youtube_video(url, video_id, max_length)
|
||||
|
||||
connector = WebCrawlerConnector(firecrawl_api_key=firecrawl_api_key)
|
||||
result, error = await connector.crawl_url(url, formats=["markdown"])
|
||||
|
||||
if error:
|
||||
return {
|
||||
"id": scrape_id,
|
||||
"assetId": url,
|
||||
"kind": "article",
|
||||
"href": url,
|
||||
"title": domain or "Webpage",
|
||||
"domain": domain,
|
||||
"error": error,
|
||||
}
|
||||
|
||||
if not result:
|
||||
return {
|
||||
"id": scrape_id,
|
||||
"assetId": url,
|
||||
"kind": "article",
|
||||
"href": url,
|
||||
"title": domain or "Webpage",
|
||||
"domain": domain,
|
||||
"error": "No content returned from crawler",
|
||||
}
|
||||
|
||||
content = result.get("content", "")
|
||||
metadata = result.get("metadata", {})
|
||||
|
||||
title = metadata.get("title", "")
|
||||
if not title:
|
||||
title = domain or url.split("/")[-1] or "Webpage"
|
||||
|
||||
description = metadata.get("description", "")
|
||||
if not description and content:
|
||||
first_para = content.split("\n\n")[0] if content else ""
|
||||
description = (
|
||||
first_para[:300] + "..." if len(first_para) > 300 else first_para
|
||||
)
|
||||
|
||||
content, was_truncated = truncate_content(content, max_length)
|
||||
word_count = len(content.split())
|
||||
|
||||
return {
|
||||
"id": scrape_id,
|
||||
"assetId": url,
|
||||
"kind": "article",
|
||||
"href": url,
|
||||
"title": title,
|
||||
"description": description,
|
||||
"content": content,
|
||||
"domain": domain,
|
||||
"word_count": word_count,
|
||||
"was_truncated": was_truncated,
|
||||
"crawler_type": result.get("crawler_type", "unknown"),
|
||||
"author": metadata.get("author"),
|
||||
"date": metadata.get("date"),
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
error_message = str(e)
|
||||
logger.error(f"[scrape_webpage] Error scraping {url}: {error_message}")
|
||||
return {
|
||||
"id": scrape_id,
|
||||
"assetId": url,
|
||||
"kind": "article",
|
||||
"href": url,
|
||||
"title": domain or "Webpage",
|
||||
"domain": domain,
|
||||
"error": f"Failed to scrape: {error_message[:100]}",
|
||||
}
|
||||
|
||||
return scrape_webpage
|
||||
|
|
@ -53,7 +53,7 @@ def create_update_memory_tool(
|
|||
|
||||
|
||||
def create_update_team_memory_tool(
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
db_session: AsyncSession,
|
||||
llm: Any | None = None,
|
||||
):
|
||||
|
|
@ -62,7 +62,7 @@ def create_update_team_memory_tool(
|
|||
|
||||
@tool
|
||||
async def update_memory(updated_memory: str) -> dict[str, Any]:
|
||||
"""Update the team's shared memory document for this search space.
|
||||
"""Update the team's shared memory document for this workspace.
|
||||
|
||||
The current team memory is shown in <team_memory>. Pass the FULL updated
|
||||
markdown document, not a diff.
|
||||
|
|
@ -71,7 +71,7 @@ def create_update_team_memory_tool(
|
|||
async with async_session_maker() as db_session:
|
||||
result = await save_memory(
|
||||
scope=MemoryScope.TEAM,
|
||||
target_id=search_space_id,
|
||||
target_id=workspace_id,
|
||||
content=updated_memory,
|
||||
session=db_session,
|
||||
llm=llm,
|
||||
|
|
|
|||
|
|
@ -1,9 +1,8 @@
|
|||
"""Render citable documents for the model: one shape for search, read, and web.
|
||||
"""Render citable documents for the model: one shape for search and read.
|
||||
|
||||
``render_document`` emits one ``<document title=… source=… view="excerpt|full">``
|
||||
block whose passages carry server-assigned ``[n]`` labels. ``render_search_context``
|
||||
wraps KB excerpt blocks in ``<retrieved_context>``; ``render_web_results`` wraps web
|
||||
excerpt blocks in ``<web_results>``. Both cite with the same ``[n]`` spine.
|
||||
wraps KB excerpt blocks in ``<retrieved_context>`` and cites with the ``[n]`` spine.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -12,7 +11,6 @@ from .document import render_document
|
|||
from .models import DocumentView, RenderableDocument, RenderablePassage
|
||||
from .search_context import render_search_context
|
||||
from .source_label import source_label
|
||||
from .web_results import render_web_results
|
||||
|
||||
__all__ = [
|
||||
"DocumentView",
|
||||
|
|
@ -20,6 +18,5 @@ __all__ = [
|
|||
"RenderablePassage",
|
||||
"render_document",
|
||||
"render_search_context",
|
||||
"render_web_results",
|
||||
"source_label",
|
||||
]
|
||||
|
|
|
|||
|
|
@ -1,46 +0,0 @@
|
|||
"""Wrap live web-search results in a ``<web_results>`` block.
|
||||
|
||||
Each result renders through the shared ``render_document`` (excerpt view), so a
|
||||
web result is cited with ``[n]`` exactly like a knowledge-base passage. Only the
|
||||
container and header differ — they tell the model these came from the public web,
|
||||
not the user's workspace.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.citations import CitationRegistry
|
||||
|
||||
from .document import render_document
|
||||
from .models import RenderableDocument
|
||||
|
||||
_HEADER = (
|
||||
"These are live results from a public web search for this query. Each\n"
|
||||
"<document> below is one result in excerpt view; cite a result with its [n]\n"
|
||||
"after the statement it supports. Scrape the URL for full context before\n"
|
||||
"making a definitive claim from a snippet."
|
||||
)
|
||||
|
||||
|
||||
def render_web_results(
|
||||
documents: list[RenderableDocument],
|
||||
registry: CitationRegistry,
|
||||
) -> str | None:
|
||||
"""Render web results as excerpt blocks inside ``<web_results>``.
|
||||
|
||||
Returns ``None`` when no result has content to show, so the caller can skip
|
||||
the block. Mutates ``registry`` (find-or-create), so a URL seen again keeps
|
||||
its original ``[n]``.
|
||||
"""
|
||||
blocks = [
|
||||
block
|
||||
for document in documents
|
||||
if (block := render_document(document, view="excerpt", registry=registry))
|
||||
is not None
|
||||
]
|
||||
if not blocks:
|
||||
return None
|
||||
|
||||
return "<web_results>\n" + _HEADER + "\n" + "\n".join(blocks) + "\n</web_results>"
|
||||
|
||||
|
||||
__all__ = ["render_web_results"]
|
||||
|
|
@ -120,8 +120,8 @@ class KBPostgresBackend(BackendProtocol):
|
|||
|
||||
_IMAGE_EXTENSIONS = frozenset({".png", ".jpg", ".jpeg", ".gif", ".webp"})
|
||||
|
||||
def __init__(self, search_space_id: int, runtime: ToolRuntime) -> None:
|
||||
self.search_space_id = search_space_id
|
||||
def __init__(self, workspace_id: int, runtime: ToolRuntime) -> None:
|
||||
self.workspace_id = workspace_id
|
||||
self.runtime = runtime
|
||||
|
||||
@property
|
||||
|
|
@ -405,7 +405,7 @@ class KBPostgresBackend(BackendProtocol):
|
|||
if not normalized_path.startswith(DOCUMENTS_ROOT):
|
||||
return [], set()
|
||||
|
||||
index = await build_path_index(session, self.search_space_id)
|
||||
index = await build_path_index(session, self.workspace_id)
|
||||
target_folder_id: int | None = None
|
||||
if normalized_path != DOCUMENTS_ROOT:
|
||||
target_path = normalized_path
|
||||
|
|
@ -418,7 +418,7 @@ class KBPostgresBackend(BackendProtocol):
|
|||
|
||||
result = await session.execute(
|
||||
select(Document.id, Document.title, Document.folder_id, Document.updated_at)
|
||||
.where(Document.search_space_id == self.search_space_id)
|
||||
.where(Document.workspace_id == self.workspace_id)
|
||||
.where(
|
||||
Document.folder_id == target_folder_id
|
||||
if target_folder_id is not None
|
||||
|
|
@ -519,7 +519,7 @@ class KBPostgresBackend(BackendProtocol):
|
|||
async with shielded_async_session() as session:
|
||||
document_row = await virtual_path_to_doc(
|
||||
session,
|
||||
search_space_id=self.search_space_id,
|
||||
workspace_id=self.workspace_id,
|
||||
virtual_path=path,
|
||||
)
|
||||
if document_row is None:
|
||||
|
|
@ -667,10 +667,10 @@ class KBPostgresBackend(BackendProtocol):
|
|||
if normalized.startswith(DOCUMENTS_ROOT) or normalized == "/":
|
||||
try:
|
||||
async with shielded_async_session() as session:
|
||||
index = await build_path_index(session, self.search_space_id)
|
||||
index = await build_path_index(session, self.workspace_id)
|
||||
rows = await session.execute(
|
||||
select(Document.id, Document.title, Document.folder_id).where(
|
||||
Document.search_space_id == self.search_space_id
|
||||
Document.workspace_id == self.workspace_id
|
||||
)
|
||||
)
|
||||
for row in rows.all():
|
||||
|
|
@ -747,11 +747,11 @@ class KBPostgresBackend(BackendProtocol):
|
|||
if normalized.startswith(DOCUMENTS_ROOT) or normalized == "/":
|
||||
try:
|
||||
async with shielded_async_session() as session:
|
||||
index = await build_path_index(session, self.search_space_id)
|
||||
index = await build_path_index(session, self.workspace_id)
|
||||
sub = (
|
||||
select(Chunk.document_id, Chunk.id, Chunk.content)
|
||||
.join(Document, Document.id == Chunk.document_id)
|
||||
.where(Document.search_space_id == self.search_space_id)
|
||||
.where(Document.workspace_id == self.workspace_id)
|
||||
.where(Chunk.content.ilike(f"%{pattern}%"))
|
||||
.order_by(Chunk.document_id, Chunk.position, Chunk.id)
|
||||
)
|
||||
|
|
@ -849,14 +849,14 @@ class KBPostgresBackend(BackendProtocol):
|
|||
|
||||
try:
|
||||
async with shielded_async_session() as session:
|
||||
index = await build_path_index(session, self.search_space_id)
|
||||
index = await build_path_index(session, self.workspace_id)
|
||||
doc_rows_raw = await session.execute(
|
||||
select(
|
||||
Document.id,
|
||||
Document.title,
|
||||
Document.folder_id,
|
||||
Document.updated_at,
|
||||
).where(Document.search_space_id == self.search_space_id)
|
||||
).where(Document.workspace_id == self.workspace_id)
|
||||
)
|
||||
doc_rows = list(doc_rows_raw.all())
|
||||
except Exception as exc: # pragma: no cover
|
||||
|
|
|
|||
|
|
@ -31,14 +31,14 @@ def _cached_multi_root_backend(
|
|||
def build_backend_resolver(
|
||||
selection: FilesystemSelection,
|
||||
*,
|
||||
search_space_id: int | None = None,
|
||||
workspace_id: int | None = None,
|
||||
) -> Callable[[ToolRuntime], BackendProtocol]:
|
||||
"""Create deepagents backend resolver for the selected filesystem mode.
|
||||
|
||||
In cloud mode the resolver returns a fresh :class:`KBPostgresBackend`
|
||||
bound to the current ``runtime`` so the backend can read staging state
|
||||
(``staged_dirs``, ``pending_moves``, ``files`` cache, ``kb_anon_doc``)
|
||||
for each tool call. When no ``search_space_id``
|
||||
for each tool call. When no ``workspace_id``
|
||||
is provided, the resolver falls back to :class:`StateBackend` (used by
|
||||
sub-agents and tests that don't need DB-backed reads).
|
||||
|
||||
|
|
@ -55,10 +55,10 @@ def build_backend_resolver(
|
|||
|
||||
return _resolve_local
|
||||
|
||||
if search_space_id is not None:
|
||||
if workspace_id is not None:
|
||||
|
||||
def _resolve_kb(runtime: ToolRuntime) -> BackendProtocol:
|
||||
return KBPostgresBackend(search_space_id, runtime)
|
||||
return KBPostgresBackend(workspace_id, runtime)
|
||||
|
||||
return _resolve_kb
|
||||
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ def build_filesystem_mw(
|
|||
*,
|
||||
backend_resolver: Any,
|
||||
filesystem_mode: FilesystemMode,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
user_id: str | None,
|
||||
thread_id: int | None,
|
||||
read_only: bool = False,
|
||||
|
|
@ -21,7 +21,7 @@ def build_filesystem_mw(
|
|||
return SurfSenseFilesystemMiddleware(
|
||||
backend=backend_resolver,
|
||||
filesystem_mode=filesystem_mode,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
created_by_id=user_id,
|
||||
thread_id=thread_id,
|
||||
read_only=read_only,
|
||||
|
|
|
|||
|
|
@ -49,14 +49,14 @@ class SurfSenseFilesystemMiddleware(FilesystemMiddleware):
|
|||
*,
|
||||
backend: Any = None,
|
||||
filesystem_mode: FilesystemMode = FilesystemMode.CLOUD,
|
||||
search_space_id: int | None = None,
|
||||
workspace_id: int | None = None,
|
||||
created_by_id: str | None = None,
|
||||
thread_id: int | str | None = None,
|
||||
tool_token_limit_before_evict: int | None = 20000,
|
||||
read_only: bool = False,
|
||||
) -> None:
|
||||
self._filesystem_mode = filesystem_mode
|
||||
self._search_space_id = search_space_id
|
||||
self._workspace_id = workspace_id
|
||||
self._created_by_id = created_by_id
|
||||
self._thread_id = thread_id
|
||||
self._read_only = read_only
|
||||
|
|
|
|||
|
|
@ -33,10 +33,13 @@ from __future__ import annotations
|
|||
|
||||
from typing import Any, Literal, TypedDict
|
||||
|
||||
# Subagent that emitted this receipt.
|
||||
# Subagent that emitted this receipt. ``mcp_discovery`` is the current
|
||||
# connected-apps route; the per-connector literals below it are retained so
|
||||
# historical receipts (persisted in old checkpoints) still type-check.
|
||||
ReceiptRoute = Literal[
|
||||
"deliverables",
|
||||
"knowledge_base",
|
||||
"mcp_discovery",
|
||||
"notion",
|
||||
"slack",
|
||||
"gmail",
|
||||
|
|
@ -102,7 +105,7 @@ class Receipt(TypedDict, total=False):
|
|||
``None`` only when the operation failed before the backend assigned one."""
|
||||
|
||||
verifiable_url: str | None
|
||||
"""URL the parent can pass to ``scrape_webpage`` to verify the
|
||||
"""URL the parent can crawl (via ``task(web_crawler, …)``) to verify the
|
||||
operation. ``None`` when no public URL exists (Gmail, KB, raw images
|
||||
stored in the DB)."""
|
||||
|
||||
|
|
|
|||
|
|
@ -1,18 +1,17 @@
|
|||
"""Knowledge-base retrieval: hybrid search rendered as citable evidence.
|
||||
|
||||
Public surface is the service (``search_knowledge_base_context``) and its input
|
||||
value object (``SearchScope``); the rest are building blocks.
|
||||
Public surface is ``build_context`` (rerank → adapt → render) and the
|
||||
``SearchScope`` input value object; the rest are building blocks.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from .models import ChunkHit, DocumentHit, SearchScope
|
||||
from .service import build_context, search_knowledge_base_context
|
||||
from .service import build_context
|
||||
|
||||
__all__ = [
|
||||
"ChunkHit",
|
||||
"DocumentHit",
|
||||
"SearchScope",
|
||||
"build_context",
|
||||
"search_knowledge_base_context",
|
||||
]
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ _SURFACE = "chunks"
|
|||
async def search_chunks(
|
||||
db_session: AsyncSession,
|
||||
*,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
query: str,
|
||||
scope: SearchScope,
|
||||
top_k: int,
|
||||
|
|
@ -44,14 +44,14 @@ async def search_chunks(
|
|||
"""
|
||||
started = time.perf_counter()
|
||||
with otel.kb_search_span(
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
query_chars=len(query),
|
||||
extra={"search.surface": _SURFACE, "search.mode": "hybrid"},
|
||||
) as span:
|
||||
try:
|
||||
documents = await _search(
|
||||
db_session,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
query=query,
|
||||
scope=scope,
|
||||
top_k=top_k,
|
||||
|
|
@ -60,14 +60,14 @@ async def search_chunks(
|
|||
finally:
|
||||
elapsed_ms = (time.perf_counter() - started) * 1000
|
||||
metrics.record_kb_search_duration(
|
||||
elapsed_ms, search_space_id=search_space_id, surface=_SURFACE
|
||||
elapsed_ms, workspace_id=workspace_id, surface=_SURFACE
|
||||
)
|
||||
span.set_attribute("result.count", len(documents))
|
||||
get_perf_logger().info(
|
||||
"[chunk_search] hybrid in %.3fs docs=%d space=%d",
|
||||
elapsed_ms / 1000,
|
||||
len(documents),
|
||||
search_space_id,
|
||||
workspace_id,
|
||||
)
|
||||
return documents
|
||||
|
||||
|
|
@ -75,7 +75,7 @@ async def search_chunks(
|
|||
async def _search(
|
||||
db_session: AsyncSession,
|
||||
*,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
query: str,
|
||||
scope: SearchScope,
|
||||
top_k: int,
|
||||
|
|
@ -91,7 +91,7 @@ async def _search(
|
|||
config.embedding_model_instance.embed, query
|
||||
)
|
||||
|
||||
conditions = _base_conditions(search_space_id, scope, document_types)
|
||||
conditions = _base_conditions(workspace_id, scope, document_types)
|
||||
rows = await _fused_chunks(
|
||||
db_session,
|
||||
query=query,
|
||||
|
|
@ -116,13 +116,13 @@ def _resolve_document_types(
|
|||
|
||||
|
||||
def _base_conditions(
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
scope: SearchScope,
|
||||
document_types: list[DocumentType] | None,
|
||||
) -> list:
|
||||
"""Filters shared by both search legs."""
|
||||
conditions = [
|
||||
Document.search_space_id == search_space_id,
|
||||
Document.workspace_id == workspace_id,
|
||||
func.coalesce(Document.status["state"].astext, "ready") != "deleting",
|
||||
]
|
||||
if document_types:
|
||||
|
|
|
|||
|
|
@ -1,54 +1,27 @@
|
|||
"""Search the knowledge base and render it as model-facing ``<retrieved_context>``.
|
||||
"""Render knowledge-base hits as model-facing ``<retrieved_context>``.
|
||||
|
||||
The retrieval spine end to end: hybrid search → rerank → adapt → render, with
|
||||
each shown passage registered for ``[n]`` citation along the way.
|
||||
The tail of the retrieval spine: rerank → adapt → render, registering each
|
||||
shown passage for ``[n]`` citation. Hybrid search itself lives in
|
||||
``hybrid_search``; callers (the ``search_knowledge_base`` tool) pass its hits
|
||||
straight into :func:`build_context`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.citations import CitationRegistry
|
||||
from app.agents.chat.multi_agent_chat.shared.document_render import (
|
||||
render_search_context,
|
||||
)
|
||||
|
||||
from .adapter import to_renderable_document
|
||||
from .hybrid_search import search_chunks
|
||||
from .models import DocumentHit, SearchScope
|
||||
from .models import DocumentHit
|
||||
from .reranking import rerank_hits
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.services.reranker_service import RerankerService
|
||||
|
||||
_DEFAULT_TOP_K = 10
|
||||
|
||||
|
||||
async def search_knowledge_base_context(
|
||||
db_session: AsyncSession,
|
||||
*,
|
||||
search_space_id: int,
|
||||
query: str,
|
||||
registry: CitationRegistry,
|
||||
scope: SearchScope | None = None,
|
||||
reranker: RerankerService | None = None,
|
||||
top_k: int = _DEFAULT_TOP_K,
|
||||
) -> str | None:
|
||||
"""Retrieve KB evidence for ``query`` and render it, registering each ``[n]``.
|
||||
|
||||
Returns ``None`` when nothing matched, so the caller can skip the block.
|
||||
"""
|
||||
hits = await search_chunks(
|
||||
db_session,
|
||||
search_space_id=search_space_id,
|
||||
query=query,
|
||||
scope=scope or SearchScope(),
|
||||
top_k=top_k,
|
||||
)
|
||||
return build_context(query, hits, registry, reranker=reranker)
|
||||
|
||||
|
||||
def build_context(
|
||||
query: str,
|
||||
|
|
@ -63,4 +36,4 @@ def build_context(
|
|||
return render_search_context(documents, registry)
|
||||
|
||||
|
||||
__all__ = ["build_context", "search_knowledge_base_context"]
|
||||
__all__ = ["build_context"]
|
||||
|
|
|
|||
|
|
@ -64,14 +64,6 @@ TOOL_CATALOG: list[ToolMetadata] = [
|
|||
name="search_knowledge_base",
|
||||
description="Search the user's knowledge base with hybrid semantic + keyword retrieval",
|
||||
),
|
||||
ToolMetadata(
|
||||
name="scrape_webpage",
|
||||
description="Scrape and extract the main content from a webpage",
|
||||
),
|
||||
ToolMetadata(
|
||||
name="web_search",
|
||||
description="Search the web for real-time information using configured search engines",
|
||||
),
|
||||
ToolMetadata(
|
||||
name="create_automation",
|
||||
description="Draft an automation from an NL intent; user approves the card; tool saves",
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@ logger = logging.getLogger(__name__)
|
|||
# artifact in the user's own workspace with no external visibility (drafts
|
||||
# aren't sent; new files aren't shared). They still call ``request_approval``,
|
||||
# which returns ``decision_type="auto_approved"`` without firing an interrupt.
|
||||
# Per-search-space ``agent_permission_rules`` can re-enable prompting.
|
||||
# Per-workspace ``agent_permission_rules`` can re-enable prompting.
|
||||
DEFAULT_AUTO_APPROVED_TOOLS: frozenset[str] = frozenset(
|
||||
{
|
||||
"create_gmail_draft",
|
||||
|
|
|
|||
|
|
@ -99,11 +99,11 @@ async def write_cached_tools(
|
|||
|
||||
def refresh_mcp_tools_cache_for_connector(
|
||||
connector_id: int,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
) -> None:
|
||||
"""Maintain the MCP tool cache after a single-connector lifecycle event.
|
||||
|
||||
Synchronously evicts the in-process LRU for the connector's search space
|
||||
Synchronously evicts the in-process LRU for the connector's workspace
|
||||
(LRU keys are per-space, so eviction cannot be scoped finer), then schedules
|
||||
a background live discovery for this connector alone so its persisted
|
||||
``cached_tools`` row is refreshed before the next user query.
|
||||
|
|
@ -116,11 +116,11 @@ def refresh_mcp_tools_cache_for_connector(
|
|||
invalidate_mcp_tools_cache,
|
||||
)
|
||||
|
||||
invalidate_mcp_tools_cache(search_space_id)
|
||||
invalidate_mcp_tools_cache(workspace_id)
|
||||
except Exception:
|
||||
logger.debug(
|
||||
"MCP in-process cache eviction skipped for space %d",
|
||||
search_space_id,
|
||||
workspace_id,
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -56,7 +56,7 @@ _MCP_CACHE_MAX_SIZE = 50
|
|||
_MCP_DISCOVERY_TIMEOUT_SECONDS = 30
|
||||
_TOOL_CALL_MAX_RETRIES = 3
|
||||
_TOOL_CALL_RETRY_DELAY = 1.5 # seconds, doubles per attempt
|
||||
# Keyed by ``(search_space_id, bypass_internal_hitl)`` so single-agent and
|
||||
# Keyed by ``(workspace_id, bypass_internal_hitl)`` so single-agent and
|
||||
# multi-agent paths cannot share tool closures with different HITL wiring.
|
||||
_MCPCacheKey = tuple[int, bool]
|
||||
_mcp_tools_cache: dict[_MCPCacheKey, tuple[float, list[StructuredTool]]] = {}
|
||||
|
|
@ -649,14 +649,31 @@ async def _load_http_mcp_tools(
|
|||
total_discovered = len(tool_definitions)
|
||||
|
||||
if allowed_set:
|
||||
tool_definitions = [td for td in tool_definitions if td["name"] in allowed_set]
|
||||
logger.info(
|
||||
"HTTP MCP server '%s' (connector %d): %d/%d tools after allowlist filter",
|
||||
url,
|
||||
connector_id,
|
||||
len(tool_definitions),
|
||||
total_discovered,
|
||||
)
|
||||
filtered = [td for td in tool_definitions if td["name"] in allowed_set]
|
||||
if not filtered and total_discovered:
|
||||
# The server renamed its tools out from under our allowlist
|
||||
# (e.g. Notion's "notion-" prefix rename) — a fully-dead
|
||||
# allowlist would silently disable the connector. Load
|
||||
# everything instead: renamed tools won't match
|
||||
# ``readonly_tools`` either, so every tool stays HITL-gated.
|
||||
logger.warning(
|
||||
"HTTP MCP server '%s' (connector %d): allowlist matched 0/%d "
|
||||
"advertised tools — server likely renamed its tools. "
|
||||
"Loading all tools (HITL-gated). Update the registry allowlist: %s",
|
||||
url,
|
||||
connector_id,
|
||||
total_discovered,
|
||||
sorted(td["name"] for td in tool_definitions),
|
||||
)
|
||||
else:
|
||||
tool_definitions = filtered
|
||||
logger.info(
|
||||
"HTTP MCP server '%s' (connector %d): %d/%d tools after allowlist filter",
|
||||
url,
|
||||
connector_id,
|
||||
len(tool_definitions),
|
||||
total_discovered,
|
||||
)
|
||||
else:
|
||||
logger.info(
|
||||
"Discovered %d tools from HTTP MCP server '%s' (connector %d) — no allowlist, loading all",
|
||||
|
|
@ -814,7 +831,7 @@ async def _refresh_connector_token(
|
|||
await session.commit()
|
||||
await session.refresh(connector)
|
||||
|
||||
invalidate_mcp_tools_cache(connector.search_space_id)
|
||||
invalidate_mcp_tools_cache(connector.workspace_id)
|
||||
|
||||
return new_access
|
||||
|
||||
|
|
@ -990,7 +1007,7 @@ async def _mark_connector_auth_expired(connector_id: int) -> None:
|
|||
"Marked MCP connector %s as auth_expired after unrecoverable 401",
|
||||
connector_id,
|
||||
)
|
||||
invalidate_mcp_tools_cache(connector.search_space_id)
|
||||
invalidate_mcp_tools_cache(connector.workspace_id)
|
||||
|
||||
except Exception:
|
||||
logger.warning(
|
||||
|
|
@ -1000,10 +1017,10 @@ async def _mark_connector_auth_expired(connector_id: int) -> None:
|
|||
)
|
||||
|
||||
|
||||
def invalidate_mcp_tools_cache(search_space_id: int | None = None) -> None:
|
||||
def invalidate_mcp_tools_cache(workspace_id: int | None = None) -> None:
|
||||
"""Invalidate cached MCP tools (both ``bypass_internal_hitl`` variants together)."""
|
||||
if search_space_id is not None:
|
||||
for key in [k for k in _mcp_tools_cache if k[0] == search_space_id]:
|
||||
if workspace_id is not None:
|
||||
for key in [k for k in _mcp_tools_cache if k[0] == workspace_id]:
|
||||
_mcp_tools_cache.pop(key, None)
|
||||
else:
|
||||
_mcp_tools_cache.clear()
|
||||
|
|
@ -1099,27 +1116,27 @@ async def discover_single_mcp_connector(connector_id: int) -> None:
|
|||
|
||||
async def load_mcp_tools(
|
||||
session: AsyncSession,
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
*,
|
||||
bypass_internal_hitl: bool = False,
|
||||
) -> list[StructuredTool]:
|
||||
"""Load all MCP tools from the user's active MCP server connectors.
|
||||
|
||||
Results are cached per ``(search_space_id, bypass_internal_hitl)`` for up
|
||||
Results are cached per ``(workspace_id, bypass_internal_hitl)`` for up
|
||||
to 5 minutes; bypass is keyed because each variant builds a different tool
|
||||
closure (with vs. without the in-wrapper ``request_approval`` gate).
|
||||
"""
|
||||
_evict_expired_mcp_cache()
|
||||
|
||||
now = time.monotonic()
|
||||
cache_key: _MCPCacheKey = (search_space_id, bypass_internal_hitl)
|
||||
cache_key: _MCPCacheKey = (workspace_id, bypass_internal_hitl)
|
||||
cached = _mcp_tools_cache.get(cache_key)
|
||||
if cached is not None:
|
||||
cached_at, cached_tools = cached
|
||||
if now - cached_at < _MCP_CACHE_TTL_SECONDS:
|
||||
logger.info(
|
||||
"Using cached MCP tools for search space %s (%d tools, age=%.0fs, bypass_hitl=%s)",
|
||||
search_space_id,
|
||||
"Using cached MCP tools for workspace %s (%d tools, age=%.0fs, bypass_hitl=%s)",
|
||||
workspace_id,
|
||||
len(cached_tools),
|
||||
now - cached_at,
|
||||
bypass_internal_hitl,
|
||||
|
|
@ -1132,7 +1149,7 @@ async def load_mcp_tools(
|
|||
# Cast JSON -> JSONB so we can use has_key to filter by the presence of "server_config".
|
||||
result = await session.execute(
|
||||
select(SearchSourceConnector).filter(
|
||||
SearchSourceConnector.search_space_id == search_space_id,
|
||||
SearchSourceConnector.workspace_id == workspace_id,
|
||||
cast(SearchSourceConnector.config, JSONB).has_key("server_config"),
|
||||
),
|
||||
)
|
||||
|
|
@ -1322,9 +1339,9 @@ async def load_mcp_tools(
|
|||
del _mcp_tools_cache[oldest_key]
|
||||
|
||||
logger.info(
|
||||
"Loaded %d MCP tools for search space %d (bypass_hitl=%s)",
|
||||
"Loaded %d MCP tools for workspace %d (bypass_hitl=%s)",
|
||||
len(tools),
|
||||
search_space_id,
|
||||
workspace_id,
|
||||
bypass_internal_hitl,
|
||||
)
|
||||
return tools
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
"""Image generation via litellm; resolves model config from the search space and returns UI-ready payloads."""
|
||||
"""Image generation via litellm; resolves model config from the workspace and returns UI-ready payloads."""
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
|
|
@ -18,7 +18,7 @@ from app.config import config
|
|||
from app.db import (
|
||||
ImageGeneration,
|
||||
Model,
|
||||
SearchSpace,
|
||||
Workspace,
|
||||
shielded_async_session,
|
||||
)
|
||||
from app.services.auto_model_pin_service import (
|
||||
|
|
@ -48,14 +48,14 @@ def _get_global_connection(connection_id: int) -> dict | None:
|
|||
|
||||
|
||||
def create_generate_image_tool(
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
db_session: AsyncSession,
|
||||
image_gen_model_id_override: int | None = None,
|
||||
):
|
||||
"""Create ``generate_image`` with bound search space; DB work uses a per-call session.
|
||||
"""Create ``generate_image`` with bound workspace; DB work uses a per-call session.
|
||||
|
||||
``image_gen_model_id_override``: when set (automations running on a
|
||||
captured model), use this model id instead of reading the search space's
|
||||
captured model), use this model id instead of reading the workspace's
|
||||
live ``image_gen_model_id``.
|
||||
"""
|
||||
del db_session # tool uses a fresh per-call session instead
|
||||
|
|
@ -102,23 +102,21 @@ def create_generate_image_tool(
|
|||
# autoflushes from a concurrent writer poison this tool too.
|
||||
async with shielded_async_session() as session:
|
||||
result = await session.execute(
|
||||
select(SearchSpace).filter(SearchSpace.id == search_space_id)
|
||||
select(Workspace).filter(Workspace.id == workspace_id)
|
||||
)
|
||||
search_space = result.scalars().first()
|
||||
if not search_space:
|
||||
workspace = result.scalars().first()
|
||||
if not workspace:
|
||||
return _failed(
|
||||
{"error": "Search space not found"},
|
||||
error="Search space not found",
|
||||
{"error": "Workspace not found"},
|
||||
error="Workspace not found",
|
||||
)
|
||||
|
||||
if image_gen_model_id_override is not None:
|
||||
# Automation run: use the captured image model, insulated from
|
||||
# later search-space changes. No search-space read needed.
|
||||
# later workspace changes. No workspace read needed.
|
||||
config_id = image_gen_model_id_override or IMAGE_GEN_AUTO_MODE_ID
|
||||
else:
|
||||
config_id = (
|
||||
search_space.image_gen_model_id or IMAGE_GEN_AUTO_MODE_ID
|
||||
)
|
||||
config_id = workspace.image_gen_model_id or IMAGE_GEN_AUTO_MODE_ID
|
||||
|
||||
# size/quality/style are intentionally omitted: valid values
|
||||
# differ per model, so we let each model use its own defaults.
|
||||
|
|
@ -129,8 +127,8 @@ def create_generate_image_tool(
|
|||
if is_image_gen_auto_mode(config_id):
|
||||
candidates = await auto_model_candidates(
|
||||
session,
|
||||
search_space_id=search_space_id,
|
||||
user_id=search_space.user_id,
|
||||
workspace_id=workspace_id,
|
||||
user_id=workspace.user_id,
|
||||
capability="image_gen",
|
||||
)
|
||||
if not candidates:
|
||||
|
|
@ -140,7 +138,7 @@ def create_generate_image_tool(
|
|||
)
|
||||
return _failed({"error": err}, error=err)
|
||||
config_id = int(
|
||||
choose_auto_model_candidate(candidates, search_space_id)["id"]
|
||||
choose_auto_model_candidate(candidates, workspace_id)["id"]
|
||||
)
|
||||
|
||||
provider_base_url: str | None = None
|
||||
|
|
@ -186,15 +184,12 @@ def create_generate_image_tool(
|
|||
return _failed({"error": err}, error=err)
|
||||
conn = db_model.connection
|
||||
if (
|
||||
conn.search_space_id is not None
|
||||
and conn.search_space_id != search_space_id
|
||||
conn.workspace_id is not None
|
||||
and conn.workspace_id != workspace_id
|
||||
):
|
||||
err = f"Image generation model {config_id} not found"
|
||||
return _failed({"error": err}, error=err)
|
||||
if (
|
||||
conn.user_id is not None
|
||||
and conn.user_id != search_space.user_id
|
||||
):
|
||||
if conn.user_id is not None and conn.user_id != workspace.user_id:
|
||||
err = f"Image generation model {config_id} not found"
|
||||
return _failed({"error": err}, error=err)
|
||||
if not has_capability(db_model, "image_gen"):
|
||||
|
|
@ -225,7 +220,7 @@ def create_generate_image_tool(
|
|||
n=n,
|
||||
image_gen_model_id=config_id,
|
||||
response_data=response_dict,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
access_token=access_token,
|
||||
)
|
||||
session.add(db_image_gen)
|
||||
|
|
|
|||
|
|
@ -28,28 +28,28 @@ def load_tools(
|
|||
d = {**(dependencies or {}), **kwargs}
|
||||
return [
|
||||
create_generate_podcast_tool(
|
||||
search_space_id=d["search_space_id"],
|
||||
workspace_id=d["workspace_id"],
|
||||
db_session=d["db_session"],
|
||||
thread_id=d["thread_id"],
|
||||
),
|
||||
create_generate_video_presentation_tool(
|
||||
search_space_id=d["search_space_id"],
|
||||
workspace_id=d["workspace_id"],
|
||||
db_session=d["db_session"],
|
||||
thread_id=d["thread_id"],
|
||||
),
|
||||
create_generate_report_tool(
|
||||
search_space_id=d["search_space_id"],
|
||||
workspace_id=d["workspace_id"],
|
||||
thread_id=d["thread_id"],
|
||||
connector_service=d.get("connector_service"),
|
||||
available_connectors=d.get("available_connectors"),
|
||||
available_document_types=d.get("available_document_types"),
|
||||
),
|
||||
create_generate_resume_tool(
|
||||
search_space_id=d["search_space_id"],
|
||||
workspace_id=d["workspace_id"],
|
||||
thread_id=d["thread_id"],
|
||||
),
|
||||
create_generate_image_tool(
|
||||
search_space_id=d["search_space_id"],
|
||||
workspace_id=d["workspace_id"],
|
||||
db_session=d["db_session"],
|
||||
image_gen_model_id_override=d.get("image_gen_model_id_override"),
|
||||
),
|
||||
|
|
|
|||
|
|
@ -28,11 +28,11 @@ logger = logging.getLogger(__name__)
|
|||
|
||||
|
||||
def create_generate_podcast_tool(
|
||||
search_space_id: int,
|
||||
workspace_id: int,
|
||||
db_session: AsyncSession,
|
||||
thread_id: int | None = None,
|
||||
):
|
||||
"""Create ``generate_podcast`` with bound search space and thread; DB writes use a tool-local session."""
|
||||
"""Create ``generate_podcast`` with bound workspace and thread; DB writes use a tool-local session."""
|
||||
del db_session # writes use a fresh tool-local session, see below
|
||||
|
||||
@tool
|
||||
|
|
@ -77,13 +77,13 @@ def create_generate_podcast_tool(
|
|||
service = PodcastService(session)
|
||||
podcast = await service.create(
|
||||
title=podcast_title,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
thread_id=resolve_root_thread_id(runtime, thread_id),
|
||||
)
|
||||
podcast.source_content = source_content
|
||||
spec = await propose_brief(
|
||||
session,
|
||||
search_space_id=search_space_id,
|
||||
workspace_id=workspace_id,
|
||||
focus=user_prompt,
|
||||
)
|
||||
await service.attach_brief(podcast, spec)
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue