mirror of
https://github.com/willchen96/mike.git
synced 2026-07-26 23:51:08 +02:00
e6d53c4f0f
The suite is green with no secret (LLM specs skip). Update docs/e2e-ci.md and the workflow header to reflect the migrations step, the production build, and the optional (not required) ANTHROPIC_API_KEY. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
93 lines
4.6 KiB
Markdown
93 lines
4.6 KiB
Markdown
# End-to-end tests in CI
|
|
|
|
The Playwright suite (`e2e/`) runs on every pull request through
|
|
`.github/workflows/e2e.yml`. This document covers the one repository secret it
|
|
needs and the **branch-protection step that turns a red run into a blocked
|
|
merge** — the workflow reports pass/fail on its own, but only branch protection
|
|
makes that check *required*.
|
|
|
|
## What the workflow does
|
|
|
|
On every `pull_request` targeting `main` (or `upstream-main`, the fork mirror),
|
|
and on manual `workflow_dispatch`, the `e2e / playwright` job:
|
|
|
|
1. installs the root (Playwright), `backend/`, and `frontend/` dependencies;
|
|
2. boots **MinIO** (S3-compatible object storage — several specs upload documents);
|
|
3. boots **local Supabase** (Auth + Postgres) via the Supabase CLI, loads
|
|
`backend/schema.sql`, then applies every dated migration in `backend/migrations/`
|
|
on top. `schema.sql` is meant to be the latest shape but in practice lags the
|
|
migrations (e.g. it is missing `workflow_open_source_submissions`, which
|
|
`GET /workflows/:id` queries — a 500 without the migrations). It also grants
|
|
`service_role` full access to the `public` tables afterward, because
|
|
`schema.sql` revokes client grants assuming a hosted Supabase where
|
|
`service_role` is already privileged;
|
|
4. writes `backend/.env` and `frontend/.env.local` from the live Supabase values;
|
|
5. **builds** the web app (`next build`) and serves it with `next start` — a
|
|
production build, not `next dev`, so there is no on-demand compilation (which
|
|
makes first-hit page loads slow enough to time out specs) and no dev
|
|
hydration-error overlay (whose injected DOM pollutes text locators). Starts the
|
|
backend API (`:3001`) and the web server (`:3000`) and waits for both healthy;
|
|
6. runs `npx playwright test` and uploads the HTML report + traces as an artifact
|
|
(`playwright-report`) on pass, fail, or timeout.
|
|
|
|
`e2e/auth.setup.ts` bootstraps the shared test user (`e2e@mike.local`) against
|
|
the local Supabase admin API, so no login secret is needed — the credentials
|
|
baked into that file are the single source of truth.
|
|
|
|
Typical run: **~7 minutes**, **23 passed / 4 skipped / 0 failed** with no secret.
|
|
|
|
## Optional secret (fuller coverage)
|
|
|
|
| Secret | What it unlocks | Without it |
|
|
|---|---|---|
|
|
| `ANTHROPIC_API_KEY` | The 4 LLM-dependent specs (chat rename/delete/submit, critical-path "ask a question") send a message and assert a **streamed** answer. With the key set they run and must pass. | Those 4 specs **skip** (see `e2e/llm.ts`) instead of hanging, so the run is still green on the other ~23 specs. |
|
|
|
|
The suite is green **without** any secret — the LLM specs skip themselves via
|
|
`test.skip(!process.env.ANTHROPIC_API_KEY, …)`, which keeps keyless runs (local,
|
|
and fork PRs with no secret access) green and fast. Add the key under
|
|
**Settings → Secrets and variables → Actions → New repository secret** to also
|
|
run and enforce the LLM specs. For fork PRs, GitHub withholds secrets until a
|
|
maintainer approves the run.
|
|
|
|
## Make it merge-blocking
|
|
|
|
The workflow failing is not enough on its own — GitHub will still allow the merge
|
|
unless the check is **required**. Enable branch protection once you have seen the
|
|
suite go green a few times (it is environment-sensitive by nature):
|
|
|
|
1. **Settings → Branches → Add branch protection rule** (or edit the rule for
|
|
`main`).
|
|
2. Enable **Require status checks to pass before merging**.
|
|
3. Enable **Require branches to be up to date before merging**.
|
|
4. In the checks search box add **`e2e / playwright`** (the job appears in the
|
|
list after it has run at least once on a PR).
|
|
5. Recommended alongside it: the unit/build check `backend` and the `license/cla`
|
|
check.
|
|
6. Save. From now on a red e2e run blocks the **Merge** button.
|
|
|
|
Equivalent via the GitHub CLI (repo admin token required):
|
|
|
|
```bash
|
|
gh api -X PUT repos/OWNER/REPO/branches/main/protection \
|
|
-H "Accept: application/vnd.github+json" \
|
|
-f 'required_status_checks[strict]=true' \
|
|
-f 'required_status_checks[contexts][]=e2e / playwright' \
|
|
-f 'enforce_admins=true' \
|
|
-f 'required_pull_request_reviews[required_approving_review_count]=1' \
|
|
-f 'restrictions='
|
|
```
|
|
|
|
## Running the suite locally
|
|
|
|
Locally, `playwright.config.ts` starts the backend and web dev servers for you
|
|
(`webServer` is only disabled when `CI=true`), so a full local stack plus:
|
|
|
|
```bash
|
|
npm ci
|
|
npx playwright install --with-deps chromium
|
|
npm run test:e2e # or test:e2e:ui / test:e2e:headed
|
|
```
|
|
|
|
`e2e/auth.setup.ts` reads `SUPABASE_URL` / `SUPABASE_SECRET_KEY` from the
|
|
environment or `backend/.env`, so a running local Supabase + a populated
|
|
`backend/.env` is all the setup needs.
|