SurfSense/surfsense_web/content/docs/observability.mdx

---
title: Observability
description: Configure backend traces and metrics for SurfSense
icon: Radar
---

SurfSense uses OpenTelemetry for backend traces and metrics. Application logs
include trace and span IDs so you can correlate logs with traces, but logs stay
on the normal container stderr path.

## Enable Locally

The development compose file reads backend settings from
`surfsense_backend/.env`. Add these values there:

```dotenv
SURFSENSE_ENABLE_OTEL=true
SURFSENSE_ENV=dev
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-lgtm:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_RESOURCE_ATTRIBUTES=service.namespace=surfsense
OTEL_METRIC_EXPORT_INTERVAL=300000
```

Then start the development stack with the local LGTM backend:

```bash
docker compose -f docker/docker-compose.dev.yml up --build
```

Grafana is exposed on `http://localhost:3001` by default.

## Enable in Production Docker Compose

Production Docker Compose reads backend and collector settings from
`docker/.env`. The API and Celery worker export telemetry to the bundled
collector at `otel-collector:4317`; the collector is the only service that uses
the Grafana Cloud credentials.

Add these values to `docker/.env`:

```dotenv
SURFSENSE_ENV=production
SURFSENSE_ENABLE_OTEL=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_RESOURCE_ATTRIBUTES=service.namespace=surfsense
OTEL_METRIC_EXPORT_INTERVAL=300000

GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-<region>.grafana.net/otlp
GRAFANA_CLOUD_INSTANCE_ID=<stack instance id>
GRAFANA_CLOUD_API_KEY=<cloud access policy token>
```

Then start the stack:

```bash
docker compose -f docker/docker-compose.yml --profile observability up -d
```

The collector receives OTLP on `otel-collector:4317`, scrubs sensitive span
attributes, applies the configured tail-sampling policy, batches exports,
retries failures, and forwards traces and metrics to Grafana Cloud over OTLP
HTTP.

When deploying `surfsense_backend/Dockerfile` directly instead of production
compose, use the same split: SurfSense containers export to a collector, and
the collector owns the Grafana Cloud credentials.

## Automatic Traces

When OpenTelemetry is enabled, the backend instruments:

- FastAPI inbound requests.
- SQLAlchemy queries from the main async engine and Celery task engine.
- Raw psycopg calls used by the LangGraph checkpointer.
- Redis commands.
- HTTPX outbound requests.
- Celery producer and worker execution.

## Manual Spans

SurfSense keeps project-specific spans behind `app.observability.otel`:

- `model.call`
- `tool.call`
- `chat.request`
- `kb.search`
- `kb.persist`
- `connector.sync`
- `subagent.invoke`
- `etl.extract`
- `etl.parse`
- `etl.ocr`
- `etl.picture.describe`
- `etl.picture.ocr`
- `compaction.run`
- `permission.asked`
- `interrupt.raised`

Keep span names and attributes low-cardinality. Do not attach user content,
prompts, document titles, file paths, user-specific URLs, secrets, or raw
queries as span attributes.

## Metrics

The OpenTelemetry instrumentors provide HTTP, HTTPX, and Celery runtime
metrics. SurfSense adds these project metrics from `app.observability.metrics`:

- `surfsense.model.call.duration`
- `gen_ai.client.token.usage`
- `surfsense.tool.call.duration`
- `surfsense.tool.call.errors`
- `surfsense.chat.request.duration`
- `surfsense.chat.request.outcome`
- `surfsense.kb.search.duration`
- `surfsense.compaction.runs`
- `surfsense.permission.asks`
- `surfsense.interrupt.raised`
- `surfsense.indexing.document.duration`
- `surfsense.indexing.document.outcome`
- `surfsense.connector.sync.duration`
- `surfsense.connector.sync.outcome`
- `surfsense.subagent.invoke.duration`
- `surfsense.subagent.invoke.outcome`
- `surfsense.etl.extract.duration`
- `surfsense.etl.extract.outcome`
- `surfsense.celery.heartbeat.refreshes`
- `surfsense.celery.heartbeat.failures`
- `surfsense.celery.queue.latency`
- `surfsense.auth.failures`
- `surfsense.rate_limit.rejections`
- `surfsense.perf.elapsed_ms`

Runtime gauges include process RSS, CPU utilization, threads, open file
descriptors, asyncio tasks, and CPython GC counters.

## Logs

`LoggingInstrumentor().instrument()` injects `otelTraceID` and `otelSpanID` into
standard Python `LogRecord`s. The root log format writes them as
`trace_id=... span_id=...`.

SurfSense intentionally does not create an OpenTelemetry `LoggerProvider`,
`LoggingHandler`, or `OTLPLogExporter`. Container stderr remains the log
transport.

## Verification

1. Hit a FastAPI endpoint and confirm an inbound server span appears in Grafana.
2. Run a chat request and confirm `model.call` and `tool.call` child spans.
3. Run a knowledge-base search and confirm `kb.search` spans and SQL child spans.
4. Run connector indexing and confirm Celery producer/worker spans share a trace
   ID and connector sync metrics increment.
5. Confirm `gen_ai.client.token.usage`, model/tool durations, request duration,
   Celery runtime, and runtime gauges appear within one export interval.
6. Confirm logs emitted inside a traced request show non-zero trace and span IDs.

## Product Analytics (PostHog)

Separate from OpenTelemetry, the backend can emit server-side product events to
PostHog. This is the authoritative source for outcome events (chats, document
ingestion, connector indexing, billing, automations) because it captures traffic
the browser never sees — MCP clients, personal-access-token scripts, and Celery
background jobs. It is fully opt-in and mirrors the OTel contract: with
`POSTHOG_API_KEY` unset, every capture is a silent no-op.

Use the **same** project key as the frontend's `NEXT_PUBLIC_POSTHOG_KEY` so
server events merge onto the same PostHog persons the web app identifies by user
id. Add these to `surfsense_backend/.env` (local) or `docker/.env` (production);
they reach the API, Celery worker, and beat services via `env_file`, so no
compose changes are needed:

```dotenv
POSTHOG_API_KEY=phc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
POSTHOG_HOST=https://us.i.posthog.com
POSTHOG_AI_PRIVACY_MODE=true
```

`POSTHOG_AI_PRIVACY_MODE` defaults to `true`; set it to `false` only if you want
LLM prompt and completion bodies shipped to PostHog's AI observability views.

Every backend event is stamped `source=backend` (so it is distinguishable from
frontend captures), carries `auth_method` / `client` for surface attribution, and
sends `disable_geoip=true` so the server IP never overwrites a person's real
location. LangGraph chat turns additionally emit `$ai_generation` / `$ai_span`
traces via the PostHog LangChain callback handler, keyed by turn and chat id.

Keep event properties low-cardinality. Never attach user content — workspace
names, connector titles, document titles, prompts, or raw queries — as event
properties; they carry no aggregation value and are a privacy risk.

## Out Of Scope

- Frontend/browser OpenTelemetry.
- OpenTelemetry log export.
- Profiling.
- Production backend selection.
- Tail-sampling collector configuration.
- Replacing LangSmith.
- Vendor SDKs.