description: How SurfSense web search works and how to configure it for production with residential proxies
---
# Web Search
SurfSense uses [SearXNG](https://docs.searxng.org/) as a bundled meta-search engine to provide web search across all search spaces. SearXNG aggregates results from multiple search engines (Google, DuckDuckGo, Brave, Bing, and more) without requiring any API keys.
You can also add live search connectors such as Baidu Search, Tavily, and Linkup to a search space. When those connectors are active, SurfSense queries them in parallel with SearXNG and merges the results before passing sources to the assistant.
SearXNG runs as a Docker container alongside the backend. It is never exposed to the internet. Only the backend communicates with it over the internal Docker network.
Live search connectors are optional API-backed search providers configured per search space. They are useful when you need a specialized index, authenticated search API, or stronger regional coverage.
| Connector | Best For | Setup |
|-----------|----------|-------|
| Baidu Search | Chinese web search and China-focused current information | [Baidu Search connector](/docs/connectors/baidu-search) |
| Tavily | General web research through Tavily's search API | Add the Tavily connector from the Connectors dashboard |
| Linkup | General web search through Linkup's search API | Add the Linkup connector from the Connectors dashboard |
<Callout type="info" title="Search Space Scoped">
Live search connectors only run for the search space where they are configured. They do not replace SearXNG globally.
All engines are free. SearXNG scrapes public search pages, no API keys required.
### Engine Suspension
When a search engine returns an error (CAPTCHA, rate limit, access denied), SearXNG suspends it for a configurable duration. After the suspension expires, the engine is automatically retried.
The default suspension times are tuned for use with rotating residential proxies (shorter bans since each retry goes through a different IP):
| Error Type | Suspension | Default (without override) |
| `max_request_timeout` | 20s | Maximum allowed timeout (must be ≥ `request_timeout`) |
| `extra_proxy_timeout` | 10s | Extra seconds added when using a proxy |
| `retries` | 1 | Retries on HTTP error (uses a different proxy IP per retry) |
## Production: Residential Proxies
In production, search engines may rate-limit or block your server's IP. To avoid this, configure a residential proxy so SearXNG's outgoing requests appear to come from rotating residential IPs.
### Step 1: Build the Proxy URL
SurfSense uses [anonymous-proxies.net](https://anonymous-proxies.net/) style residential proxies where the password is a base64-encoded JSON object. Build the URL using your proxy credentials:
```bash
# Encode the password (replace with your actual values)
In `docker/searxng/settings.yml`, add the proxy URL under `outgoing.proxies`:
```yaml
outgoing:
proxies:
all://:
- http://username:base64password@proxy-host:port/
```
The `all://:` key routes both HTTP and HTTPS requests through the proxy. If you have multiple proxy endpoints, list them and SearXNG will round-robin between them:
```yaml
proxies:
all://:
- http://user:pass@proxy1:port/
- http://user:pass@proxy2:port/
```
### Step 3: Restart SearXNG
```bash
docker compose restart searxng
```
### Verify
Check that SearXNG is healthy:
```bash
curl http://localhost:8888/healthz
```
## Troubleshooting
### SearXNG Fails to Start
**`ValueError: Invalid settings.yml`** - Check the error line above the traceback. Common causes:
- `extra_proxy_timeout` must be an integer (use `10`, not `10.0`)
- `KeyError: 'engine_name'` means an engine was removed but other engines reference its network. Remove all variants (e.g., removing `qwant` also requires removing `qwant news`, `qwant images`, `qwant videos`)
### Engines Getting Suspended
If an engine is suspended (visible in SearXNG logs as `suspended_time=N`), it will automatically recover after the suspension period. With residential proxies, the next request after recovery goes through a different IP and typically succeeds.
3. Verify the backend can reach SearXNG: the `SEARXNG_DEFAULT_HOST` env var should point to `http://searxng:8080` (Docker) or `http://localhost:8888` (local dev)
### Proxy Not Working
- Verify the base64 password is correctly encoded
- Check that `extra_proxy_timeout` is set (proxies add latency)
- Ensure `max_request_timeout` is high enough to accommodate `request_timeout + extra_proxy_timeout`
## Environment Variables Reference
| Variable | Location | Description | Default |
|----------|----------|-------------|---------|
| `SEARXNG_DEFAULT_HOST` | `docker/.env` | URL of the SearXNG instance | `http://searxng:8080` |
