diff --git a/surfsense_backend/tests/e2e/README.md b/surfsense_backend/tests/e2e/README.md index 800d61dfb..caa0f89b0 100644 --- a/surfsense_backend/tests/e2e/README.md +++ b/surfsense_backend/tests/e2e/README.md @@ -1,48 +1,48 @@ -# Backend E2E Test Harness +# Backend E2E Harness -Strict fakes + alternative entrypoints used **only** by Playwright E2E. -Excluded from the production Docker image via `.dockerignore`. +This directory contains the test-only backend entrypoints and fakes used by +Playwright. They are not part of the production image: `.dockerignore` excludes +`tests/`, and the E2E Docker stage copies this directory through a separate +build context. ## Files -| Path | Role | -| -------------------------------- | ------------------------------------------------------------------------------- | -| `run_backend.py` | FastAPI entrypoint that hijacks `sys.modules` before importing `app.app:app` | -| `run_celery.py` | Celery worker entrypoint with the same hijack + patch logic | -| `middleware/scenario.py` | `X-E2E-Scenario` header → ContextVar (read by fakes) | -| `fakes/composio_module.py` | Strict drop-in for the `composio` package; raises on unknown surface | -| `fakes/llm.py` | `fake_get_user_long_context_llm` returning a `FakeListChatModel` | -| `fakes/embeddings.py` | Deterministic 0.1-vector `embed_text` / `embed_texts` | -| `fakes/fixtures/drive_files.json`| Canned Drive listings + file contents (incl. canary tokens) | +| Path | Purpose | +| --- | --- | +| `run_backend.py` | Starts FastAPI after installing the test fakes into `sys.modules`. | +| `run_celery.py` | Starts the Celery worker with the same fake setup. | +| `middleware/scenario.py` | Reads `X-E2E-Scenario` into a request-scoped context var. | +| `fakes/composio_module.py` | Fake `composio` package used by connector flows. | +| `fakes/llm.py` | Fake chat model factory. | +| `fakes/embeddings.py` | Deterministic embedding helpers. | +| `fakes/fixtures/drive_files.json` | Drive fixture data and canary file contents. | -## Why a sys.modules hijack? +## Why the import hook exists -Production code does `from composio import Composio` at module load -time. By the time the FastAPI app object exists, that binding has -already been resolved. The hijack runs **before** any `app.*` import, -so the binding resolves to our strict fake. No production source -changes; fakes are physically excluded from production images. +Some production modules import SDK clients at module load time, for example +`from composio import Composio`. By the time `app.app` has been imported, those +bindings are already fixed. -Belt + suspenders + no internet: the strict `__getattr__` in every -fake raises `NotImplementedError` if a future production code path -introduces a new SDK call. CI also sets `HTTPS_PROXY=http://127.0.0.1:1` -plus sentinel API keys so any leaked outbound HTTP fails immediately. +The E2E entrypoints install fake modules in `sys.modules` before importing any +`app.*` module. That lets the normal production code run while SDK calls resolve +to local fakes. -## Adding a new fake +The fakes should fail loudly. If production starts using a new SDK method that +the fake does not implement, add that method to the fake instead of letting the +test call the real service. -1. Create `fakes/_module.py` modelled on `composio_module.py`. -2. In `run_backend.py` and `run_celery.py`, register - `sys.modules[""] = _fake_` before the `from app.app import app` - line. -3. If the new fake needs scenario branching, read from +## Adding a fake + +1. Add `fakes/_module.py`. +2. Register it in both `run_backend.py` and `run_celery.py` before importing + `app.app` or `app.celery_app`. +3. If the fake needs per-test behavior, read the current scenario from `tests.e2e.middleware.scenario.current_scenario()`. -## Reused by backend integration tests +## Shared with backend integration tests -The strict fakes are not only for Playwright. Backend route integration -tests can import the same fake before importing `app.app`, so Composio -route tests exercise production route code without touching the real -SDK: +Backend integration tests can use the same fakes when they need production route +code without the real SDK: ```python from tests.e2e.fakes import composio_module as _fake_composio @@ -50,20 +50,93 @@ sys.modules["composio"] = _fake_composio from app.app import app ``` -See `surfsense_backend/tests/integration/composio/conftest.py` for the -current pattern. +See `surfsense_backend/tests/integration/composio/conftest.py` for the current +pattern. ## Running locally +The recommended local flow runs only Postgres and Redis in Docker, and the +backend + Celery worker on the host. No `.env` file is required: both +entrypoints `setdefault` every variable they need (DB URL, Redis URL, +sentinel API keys, etc.) to values that match `docker-compose.deps-only.yml`. + +### One-time setup + +From `surfsense_web/`: + ```bash -cd surfsense_backend +pnpm install +pnpm exec playwright install --with-deps chromium +``` + +### Each run + +**1. Bring up Postgres + Redis** from the repo root (the other deps-only +services (SearXNG, Zero, pgAdmin) are not needed for E2E): + +```bash +docker compose -f docker/docker-compose.deps-only.yml up -d db redis +``` + +**2. Start the backend** in `surfsense_backend/`, terminal A: + +```bash +uv sync +uv run alembic upgrade head uv run python tests/e2e/run_backend.py -# in a second shell: +``` + +**3. Start the Celery worker** in `surfsense_backend/`, terminal B: + +```bash uv run python tests/e2e/run_celery.py ``` -Then in `surfsense_web`: +**4. Register the Playwright user**: ```bash -pnpm test:e2e +curl -X POST http://localhost:8000/auth/register \ + -H "Content-Type: application/json" \ + -d '{"email":"e2e-test@surfsense.net","password":"E2eTestPassword123!"}' ``` + +**5. Run Playwright** from `surfsense_web/`, terminal C: + +```bash +pnpm test:e2e # dev server (fast iteration) +pnpm test:e2e:headed # show the browser +pnpm test:e2e:ui # Playwright UI mode +pnpm test:e2e:prod # build + start (matches CI exactly) +``` + +`playwright.config.ts` and the run scripts share defaults, so this works on a +fresh checkout. Set `PLAYWRIGHT_TEST_EMAIL`, `PLAYWRIGHT_TEST_PASSWORD`, +`NEXT_PUBLIC_FASTAPI_BACKEND_URL`, or any backend env (e.g. `DATABASE_URL`) +only when pointing tests at a different stack. + +### Cleanup + +```bash +docker compose -f docker/docker-compose.deps-only.yml down +``` + +Add `-v` to also wipe the Postgres volume. + +### Hermetic alternative (matches CI) + +To reproduce the CI environment exactly — backend and Celery in containers, +network egress denied at L3 — replace steps 1–3 with: + +```bash +docker compose -f docker/docker-compose.e2e.yml up -d --build --wait +``` + +Then run steps 4 (curl register) and 5 (`pnpm test:e2e:prod`) as above. Tear +down with: + +```bash +docker compose -f docker/docker-compose.e2e.yml down -v --remove-orphans +``` + +This builds the ~9 GB `surfsense-e2e-backend:local` image, so the deps-only +flow above is faster for day-to-day development. diff --git a/surfsense_web/tests/README.md b/surfsense_web/tests/README.md index 51fd35050..89aab1f9b 100644 --- a/surfsense_web/tests/README.md +++ b/surfsense_web/tests/README.md @@ -5,29 +5,6 @@ Celery + Postgres + Redis). Designed to scale from one connector (Composio Drive in Phase 1) to every connector + manual file upload without rewriting the harness. -## Layout - -``` -tests/ -├── auth.setup.ts # one-time login, persists localStorage -├── smoke/ # tracer-bullet tests (dashboard renders) -├── connectors/ -│ └── composio/ -│ └── drive/ # Composio Google Drive — Phase 1 -│ └── journey.spec.ts # connect -> select -> index -> canary assertion -├── fixtures/ # test.extend() fixtures -│ ├── index.ts # named `test` exports per spec category -│ ├── search-space.fixture.ts # apiToken + per-test search space -│ └── connectors/ -│ └── composio-drive.fixture.ts -├── helpers/ # reusable building blocks -│ ├── api/ # backend HTTP helpers -│ ├── ui/ # page-object selectors -│ ├── waits/ # deterministic polling -│ └── canary.ts # canary tokens + fixed Drive file ids -└── README.md # this file -``` - ## How the deterministic harness works There are **three layers of defense** against accidental real-world @@ -47,26 +24,90 @@ calls. None of them touch production code. ## Running locally +The recommended flow runs only Postgres and Redis in Docker, and the backend ++ Celery worker on the host. The E2E entrypoints `setdefault` every backend +variable they need, so no `.env` file is required on a fresh checkout. + +### One-time setup + +From `surfsense_web/`: + ```bash -# 1. Bring up Postgres + Redis (Docker compose, supabase, whatever you use) -docker compose up -d postgres redis - -# 2. Backend with E2E entrypoint (note: NOT `uv run main.py`) -cd surfsense_backend -uv run alembic upgrade head -uv run python tests/e2e/run_backend.py & - -# 3. Celery worker with the same entrypoint pattern -uv run python tests/e2e/run_celery.py & - -# 4. Run Playwright tests (auto-starts `pnpm dev` via webServer config) -cd ../surfsense_web -pnpm test:e2e +pnpm install +pnpm exec playwright install --with-deps chromium ``` -For CI behavior in one go: `pnpm test:e2e:headless`. +### Each run -To debug the Drive journey: `pnpm test:e2e -- connectors/composio/drive/journey.spec.ts --headed`. +**1. Bring up Postgres + Redis** from the repo root: + +```bash +docker compose -f docker/docker-compose.deps-only.yml up -d db redis +``` + +**2. Start the backend** in `surfsense_backend/`, terminal A: + +```bash +uv sync +uv run alembic upgrade head +uv run python tests/e2e/run_backend.py +``` + +**3. Start the Celery worker** in `surfsense_backend/`, terminal B: + +```bash +uv run python tests/e2e/run_celery.py +``` + +**4. Register the Playwright user**: + +```bash +curl -X POST http://localhost:8000/auth/register \ + -H "Content-Type: application/json" \ + -d '{"email":"e2e-test@surfsense.net","password":"E2eTestPassword123!"}' +``` + +**5. Run Playwright** from `surfsense_web/`, terminal C: + +```bash +pnpm test:e2e # dev server (fast iteration) +pnpm test:e2e:headed # show the browser +pnpm test:e2e:ui # Playwright UI mode +pnpm test:e2e:debug # Playwright Inspector +pnpm test:e2e:prod # build + start (matches CI exactly) +pnpm test:e2e:report # open the last HTML report +``` + +`playwright.config.ts` and the backend run scripts share defaults, so the +above works without exporting any env vars. Override +`PLAYWRIGHT_TEST_EMAIL`, `PLAYWRIGHT_TEST_PASSWORD`, or +`NEXT_PUBLIC_FASTAPI_BACKEND_URL` only when pointing tests at a different +stack. + +To debug a single journey: + +```bash +pnpm test:e2e:headed connectors/composio/drive/journey.spec.ts +``` + +### Hermetic alternative (matches CI) + +To reproduce the CI environment exactly: backend and Celery in containers +with L3 egress denied, replace steps 1–3 with: + +```bash +docker compose -f docker/docker-compose.e2e.yml up -d --build --wait +``` + +Then run steps 4 (curl register) and 5 (`pnpm test:e2e:prod`) as above. Tear +down with: + +```bash +docker compose -f docker/docker-compose.e2e.yml down -v --remove-orphans +``` + +This builds the ~9 GB e2e backend image, so the deps-only flow is faster for +day-to-day work. ## Adding a new connector