mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-05-19 18:45:15 +02:00
chore: update E2E test documentation for clarity and local setup instructions
This commit is contained in:
Anish Sarkar
parent
68f45335bc
commit
5344fa47e6
2 changed files with 192 additions and 78 deletions
|
|
@ -1,48 +1,48 @@
|
||||||
# Backend E2E Test Harness
|
# Backend E2E Harness
|
||||||
|
|
||||||
Strict fakes + alternative entrypoints used **only** by Playwright E2E.
|
This directory contains the test-only backend entrypoints and fakes used by
|
||||||
Excluded from the production Docker image via `.dockerignore`.
|
Playwright. They are not part of the production image: `.dockerignore` excludes
|
||||||
|
`tests/`, and the E2E Docker stage copies this directory through a separate
|
||||||
|
build context.
|
||||||
|
|
||||||
## Files
|
## Files
|
||||||
|
|
||||||
| Path | Role |
|
| Path | Purpose |
|
||||||
| -------------------------------- | ------------------------------------------------------------------------------- |
|
| --- | --- |
|
||||||
| `run_backend.py` | FastAPI entrypoint that hijacks `sys.modules` before importing `app.app:app` |
|
| `run_backend.py` | Starts FastAPI after installing the test fakes into `sys.modules`. |
|
||||||
| `run_celery.py` | Celery worker entrypoint with the same hijack + patch logic |
|
| `run_celery.py` | Starts the Celery worker with the same fake setup. |
|
||||||
| `middleware/scenario.py` | `X-E2E-Scenario` header → ContextVar (read by fakes) |
|
| `middleware/scenario.py` | Reads `X-E2E-Scenario` into a request-scoped context var. |
|
||||||
| `fakes/composio_module.py` | Strict drop-in for the `composio` package; raises on unknown surface |
|
| `fakes/composio_module.py` | Fake `composio` package used by connector flows. |
|
||||||
| `fakes/llm.py` | `fake_get_user_long_context_llm` returning a `FakeListChatModel` |
|
| `fakes/llm.py` | Fake chat model factory. |
|
||||||
| `fakes/embeddings.py` | Deterministic 0.1-vector `embed_text` / `embed_texts` |
|
| `fakes/embeddings.py` | Deterministic embedding helpers. |
|
||||||
| `fakes/fixtures/drive_files.json`| Canned Drive listings + file contents (incl. canary tokens) |
|
| `fakes/fixtures/drive_files.json` | Drive fixture data and canary file contents. |
|
||||||
|
|
||||||
## Why a sys.modules hijack?
|
## Why the import hook exists
|
||||||
|
|
||||||
Production code does `from composio import Composio` at module load
|
Some production modules import SDK clients at module load time, for example
|
||||||
time. By the time the FastAPI app object exists, that binding has
|
`from composio import Composio`. By the time `app.app` has been imported, those
|
||||||
already been resolved. The hijack runs **before** any `app.*` import,
|
bindings are already fixed.
|
||||||
so the binding resolves to our strict fake. No production source
|
|
||||||
changes; fakes are physically excluded from production images.
|
|
||||||
|
|
||||||
Belt + suspenders + no internet: the strict `__getattr__` in every
|
The E2E entrypoints install fake modules in `sys.modules` before importing any
|
||||||
fake raises `NotImplementedError` if a future production code path
|
`app.*` module. That lets the normal production code run while SDK calls resolve
|
||||||
introduces a new SDK call. CI also sets `HTTPS_PROXY=http://127.0.0.1:1`
|
to local fakes.
|
||||||
plus sentinel API keys so any leaked outbound HTTP fails immediately.
|
|
||||||
|
|
||||||
## Adding a new fake
|
The fakes should fail loudly. If production starts using a new SDK method that
|
||||||
|
the fake does not implement, add that method to the fake instead of letting the
|
||||||
|
test call the real service.
|
||||||
|
|
||||||
1. Create `fakes/<sdk>_module.py` modelled on `composio_module.py`.
|
## Adding a fake
|
||||||
2. In `run_backend.py` and `run_celery.py`, register
|
|
||||||
`sys.modules["<sdk>"] = _fake_<sdk>` before the `from app.app import app`
|
1. Add `fakes/<sdk>_module.py`.
|
||||||
line.
|
2. Register it in both `run_backend.py` and `run_celery.py` before importing
|
||||||
3. If the new fake needs scenario branching, read from
|
`app.app` or `app.celery_app`.
|
||||||
|
3. If the fake needs per-test behavior, read the current scenario from
|
||||||
`tests.e2e.middleware.scenario.current_scenario()`.
|
`tests.e2e.middleware.scenario.current_scenario()`.
|
||||||
|
|
||||||
## Reused by backend integration tests
|
## Shared with backend integration tests
|
||||||
|
|
||||||
The strict fakes are not only for Playwright. Backend route integration
|
Backend integration tests can use the same fakes when they need production route
|
||||||
tests can import the same fake before importing `app.app`, so Composio
|
code without the real SDK:
|
||||||
route tests exercise production route code without touching the real
|
|
||||||
SDK:
|
|
||||||
|
|
||||||
```python
|
```python
|
||||||
from tests.e2e.fakes import composio_module as _fake_composio
|
from tests.e2e.fakes import composio_module as _fake_composio
|
||||||
|
|
@ -50,20 +50,93 @@ sys.modules["composio"] = _fake_composio
|
||||||
from app.app import app
|
from app.app import app
|
||||||
```
|
```
|
||||||
|
|
||||||
See `surfsense_backend/tests/integration/composio/conftest.py` for the
|
See `surfsense_backend/tests/integration/composio/conftest.py` for the current
|
||||||
current pattern.
|
pattern.
|
||||||
|
|
||||||
## Running locally
|
## Running locally
|
||||||
|
|
||||||
|
The recommended local flow runs only Postgres and Redis in Docker, and the
|
||||||
|
backend + Celery worker on the host. No `.env` file is required: both
|
||||||
|
entrypoints `setdefault` every variable they need (DB URL, Redis URL,
|
||||||
|
sentinel API keys, etc.) to values that match `docker-compose.deps-only.yml`.
|
||||||
|
|
||||||
|
### One-time setup
|
||||||
|
|
||||||
|
From `surfsense_web/`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd surfsense_backend
|
pnpm install
|
||||||
|
pnpm exec playwright install --with-deps chromium
|
||||||
|
```
|
||||||
|
|
||||||
|
### Each run
|
||||||
|
|
||||||
|
**1. Bring up Postgres + Redis** from the repo root (the other deps-only
|
||||||
|
services (SearXNG, Zero, pgAdmin) are not needed for E2E):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f docker/docker-compose.deps-only.yml up -d db redis
|
||||||
|
```
|
||||||
|
|
||||||
|
**2. Start the backend** in `surfsense_backend/`, terminal A:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv sync
|
||||||
|
uv run alembic upgrade head
|
||||||
uv run python tests/e2e/run_backend.py
|
uv run python tests/e2e/run_backend.py
|
||||||
# in a second shell:
|
```
|
||||||
|
|
||||||
|
**3. Start the Celery worker** in `surfsense_backend/`, terminal B:
|
||||||
|
|
||||||
|
```bash
|
||||||
uv run python tests/e2e/run_celery.py
|
uv run python tests/e2e/run_celery.py
|
||||||
```
|
```
|
||||||
|
|
||||||
Then in `surfsense_web`:
|
**4. Register the Playwright user**:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pnpm test:e2e
|
curl -X POST http://localhost:8000/auth/register \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"email":"e2e-test@surfsense.net","password":"E2eTestPassword123!"}'
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**5. Run Playwright** from `surfsense_web/`, terminal C:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pnpm test:e2e # dev server (fast iteration)
|
||||||
|
pnpm test:e2e:headed # show the browser
|
||||||
|
pnpm test:e2e:ui # Playwright UI mode
|
||||||
|
pnpm test:e2e:prod # build + start (matches CI exactly)
|
||||||
|
```
|
||||||
|
|
||||||
|
`playwright.config.ts` and the run scripts share defaults, so this works on a
|
||||||
|
fresh checkout. Set `PLAYWRIGHT_TEST_EMAIL`, `PLAYWRIGHT_TEST_PASSWORD`,
|
||||||
|
`NEXT_PUBLIC_FASTAPI_BACKEND_URL`, or any backend env (e.g. `DATABASE_URL`)
|
||||||
|
only when pointing tests at a different stack.
|
||||||
|
|
||||||
|
### Cleanup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f docker/docker-compose.deps-only.yml down
|
||||||
|
```
|
||||||
|
|
||||||
|
Add `-v` to also wipe the Postgres volume.
|
||||||
|
|
||||||
|
### Hermetic alternative (matches CI)
|
||||||
|
|
||||||
|
To reproduce the CI environment exactly — backend and Celery in containers,
|
||||||
|
network egress denied at L3 — replace steps 1–3 with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f docker/docker-compose.e2e.yml up -d --build --wait
|
||||||
|
```
|
||||||
|
|
||||||
|
Then run steps 4 (curl register) and 5 (`pnpm test:e2e:prod`) as above. Tear
|
||||||
|
down with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f docker/docker-compose.e2e.yml down -v --remove-orphans
|
||||||
|
```
|
||||||
|
|
||||||
|
This builds the ~9 GB `surfsense-e2e-backend:local` image, so the deps-only
|
||||||
|
flow above is faster for day-to-day development.
|
||||||
|
|
|
||||||
|
|
@ -5,29 +5,6 @@ Celery + Postgres + Redis). Designed to scale from one connector
|
||||||
(Composio Drive in Phase 1) to every connector + manual file upload
|
(Composio Drive in Phase 1) to every connector + manual file upload
|
||||||
without rewriting the harness.
|
without rewriting the harness.
|
||||||
|
|
||||||
## Layout
|
|
||||||
|
|
||||||
```
|
|
||||||
tests/
|
|
||||||
├── auth.setup.ts # one-time login, persists localStorage
|
|
||||||
├── smoke/ # tracer-bullet tests (dashboard renders)
|
|
||||||
├── connectors/
|
|
||||||
│ └── composio/
|
|
||||||
│ └── drive/ # Composio Google Drive — Phase 1
|
|
||||||
│ └── journey.spec.ts # connect -> select -> index -> canary assertion
|
|
||||||
├── fixtures/ # test.extend() fixtures
|
|
||||||
│ ├── index.ts # named `test` exports per spec category
|
|
||||||
│ ├── search-space.fixture.ts # apiToken + per-test search space
|
|
||||||
│ └── connectors/
|
|
||||||
│ └── composio-drive.fixture.ts
|
|
||||||
├── helpers/ # reusable building blocks
|
|
||||||
│ ├── api/ # backend HTTP helpers
|
|
||||||
│ ├── ui/ # page-object selectors
|
|
||||||
│ ├── waits/ # deterministic polling
|
|
||||||
│ └── canary.ts # canary tokens + fixed Drive file ids
|
|
||||||
└── README.md # this file
|
|
||||||
```
|
|
||||||
|
|
||||||
## How the deterministic harness works
|
## How the deterministic harness works
|
||||||
|
|
||||||
There are **three layers of defense** against accidental real-world
|
There are **three layers of defense** against accidental real-world
|
||||||
|
|
@ -47,26 +24,90 @@ calls. None of them touch production code.
|
||||||
|
|
||||||
## Running locally
|
## Running locally
|
||||||
|
|
||||||
|
The recommended flow runs only Postgres and Redis in Docker, and the backend
|
||||||
|
+ Celery worker on the host. The E2E entrypoints `setdefault` every backend
|
||||||
|
variable they need, so no `.env` file is required on a fresh checkout.
|
||||||
|
|
||||||
|
### One-time setup
|
||||||
|
|
||||||
|
From `surfsense_web/`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 1. Bring up Postgres + Redis (Docker compose, supabase, whatever you use)
|
pnpm install
|
||||||
docker compose up -d postgres redis
|
pnpm exec playwright install --with-deps chromium
|
||||||
|
|
||||||
# 2. Backend with E2E entrypoint (note: NOT `uv run main.py`)
|
|
||||||
cd surfsense_backend
|
|
||||||
uv run alembic upgrade head
|
|
||||||
uv run python tests/e2e/run_backend.py &
|
|
||||||
|
|
||||||
# 3. Celery worker with the same entrypoint pattern
|
|
||||||
uv run python tests/e2e/run_celery.py &
|
|
||||||
|
|
||||||
# 4. Run Playwright tests (auto-starts `pnpm dev` via webServer config)
|
|
||||||
cd ../surfsense_web
|
|
||||||
pnpm test:e2e
|
|
||||||
```
|
```
|
||||||
|
|
||||||
For CI behavior in one go: `pnpm test:e2e:headless`.
|
### Each run
|
||||||
|
|
||||||
To debug the Drive journey: `pnpm test:e2e -- connectors/composio/drive/journey.spec.ts --headed`.
|
**1. Bring up Postgres + Redis** from the repo root:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f docker/docker-compose.deps-only.yml up -d db redis
|
||||||
|
```
|
||||||
|
|
||||||
|
**2. Start the backend** in `surfsense_backend/`, terminal A:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv sync
|
||||||
|
uv run alembic upgrade head
|
||||||
|
uv run python tests/e2e/run_backend.py
|
||||||
|
```
|
||||||
|
|
||||||
|
**3. Start the Celery worker** in `surfsense_backend/`, terminal B:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python tests/e2e/run_celery.py
|
||||||
|
```
|
||||||
|
|
||||||
|
**4. Register the Playwright user**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/auth/register \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"email":"e2e-test@surfsense.net","password":"E2eTestPassword123!"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
**5. Run Playwright** from `surfsense_web/`, terminal C:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pnpm test:e2e # dev server (fast iteration)
|
||||||
|
pnpm test:e2e:headed # show the browser
|
||||||
|
pnpm test:e2e:ui # Playwright UI mode
|
||||||
|
pnpm test:e2e:debug # Playwright Inspector
|
||||||
|
pnpm test:e2e:prod # build + start (matches CI exactly)
|
||||||
|
pnpm test:e2e:report # open the last HTML report
|
||||||
|
```
|
||||||
|
|
||||||
|
`playwright.config.ts` and the backend run scripts share defaults, so the
|
||||||
|
above works without exporting any env vars. Override
|
||||||
|
`PLAYWRIGHT_TEST_EMAIL`, `PLAYWRIGHT_TEST_PASSWORD`, or
|
||||||
|
`NEXT_PUBLIC_FASTAPI_BACKEND_URL` only when pointing tests at a different
|
||||||
|
stack.
|
||||||
|
|
||||||
|
To debug a single journey:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pnpm test:e2e:headed connectors/composio/drive/journey.spec.ts
|
||||||
|
```
|
||||||
|
|
||||||
|
### Hermetic alternative (matches CI)
|
||||||
|
|
||||||
|
To reproduce the CI environment exactly: backend and Celery in containers
|
||||||
|
with L3 egress denied, replace steps 1–3 with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f docker/docker-compose.e2e.yml up -d --build --wait
|
||||||
|
```
|
||||||
|
|
||||||
|
Then run steps 4 (curl register) and 5 (`pnpm test:e2e:prod`) as above. Tear
|
||||||
|
down with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f docker/docker-compose.e2e.yml down -v --remove-orphans
|
||||||
|
```
|
||||||
|
|
||||||
|
This builds the ~9 GB e2e backend image, so the deps-only flow is faster for
|
||||||
|
day-to-day work.
|
||||||
|
|
||||||
## Adding a new connector
|
## Adding a new connector
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue