docs(e2e): document keyless-green behavior, migrations, and prod build

The suite is green with no secret (LLM specs skip). Update docs/e2e-ci.md and the
workflow header to reflect the migrations step, the production build, and the
optional (not required) ANTHROPIC_API_KEY.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
QA Runner 2026-07-17 16:16:32 -07:00
parent 42346d6900
commit 094fe5653d
2 changed files with 31 additions and 20 deletions

View file

@ -7,11 +7,11 @@
# this job "e2e / playwright" as a required status check in branch protection — # this job "e2e / playwright" as a required status check in branch protection —
# see docs/e2e-ci.md ("Make it merge-blocking"). # see docs/e2e-ci.md ("Make it merge-blocking").
# #
# Required repository secret for a full pass: # The suite is green with NO secret: the 4 LLM-dependent specs (chat + critical
# ANTHROPIC_API_KEY — the critical-path / chat specs send a message and expect a # path) skip themselves when ANTHROPIC_API_KEY is absent (see e2e/llm.ts), so a
# streamed answer, which needs a live model key. Add it under # keyless run passes the other ~23 specs. Set the ANTHROPIC_API_KEY secret under
# Settings > Secrets and variables > Actions. Without it the # Settings > Secrets and variables > Actions to also run and enforce those 4.
# chat specs fail; every other spec runs key-less. # See docs/e2e-ci.md for the full picture.
# #
# The e2e user (e2e@mike.local) is bootstrapped in-job by e2e/auth.setup.ts against # The e2e user (e2e@mike.local) is bootstrapped in-job by e2e/auth.setup.ts against
# the local Supabase admin API, so no E2E_PASSWORD secret is needed — the defaults # the local Supabase admin API, so no E2E_PASSWORD secret is needed — the defaults

View file

@ -12,31 +12,42 @@ On every `pull_request` targeting `main` (or `upstream-main`, the fork mirror),
and on manual `workflow_dispatch`, the `e2e / playwright` job: and on manual `workflow_dispatch`, the `e2e / playwright` job:
1. installs the root (Playwright), `backend/`, and `frontend/` dependencies; 1. installs the root (Playwright), `backend/`, and `frontend/` dependencies;
2. boots **MinIO** (S3-compatible object storage — three specs upload documents); 2. boots **MinIO** (S3-compatible object storage — several specs upload documents);
3. boots **local Supabase** (Auth + Postgres) via the Supabase CLI and loads 3. boots **local Supabase** (Auth + Postgres) via the Supabase CLI, loads
`backend/schema.sql` — the same fresh-database schema the README tells a human `backend/schema.sql`, then applies every dated migration in `backend/migrations/`
to run; on top. `schema.sql` is meant to be the latest shape but in practice lags the
migrations (e.g. it is missing `workflow_open_source_submissions`, which
`GET /workflows/:id` queries — a 500 without the migrations). It also grants
`service_role` full access to the `public` tables afterward, because
`schema.sql` revokes client grants assuming a hosted Supabase where
`service_role` is already privileged;
4. writes `backend/.env` and `frontend/.env.local` from the live Supabase values; 4. writes `backend/.env` and `frontend/.env.local` from the live Supabase values;
5. starts the backend API (`:3001`) and the Next.js web app (`:3000`), waits for 5. **builds** the web app (`next build`) and serves it with `next start` — a
both to be healthy; production build, not `next dev`, so there is no on-demand compilation (which
makes first-hit page loads slow enough to time out specs) and no dev
hydration-error overlay (whose injected DOM pollutes text locators). Starts the
backend API (`:3001`) and the web server (`:3000`) and waits for both healthy;
6. runs `npx playwright test` and uploads the HTML report + traces as an artifact 6. runs `npx playwright test` and uploads the HTML report + traces as an artifact
(`playwright-report`) on both pass and fail. (`playwright-report`) on pass, fail, or timeout.
`e2e/auth.setup.ts` bootstraps the shared test user (`e2e@mike.local`) against `e2e/auth.setup.ts` bootstraps the shared test user (`e2e@mike.local`) against
the local Supabase admin API, so no login secret is needed — the credentials the local Supabase admin API, so no login secret is needed — the credentials
baked into that file are the single source of truth. baked into that file are the single source of truth.
## Required secret Typical run: **~7 minutes**, **23 passed / 4 skipped / 0 failed** with no secret.
| Secret | Why | Without it | ## Optional secret (fuller coverage)
| Secret | What it unlocks | Without it |
|---|---|---| |---|---|---|
| `ANTHROPIC_API_KEY` | The critical-path / chat specs send a message and assert a **streamed** answer, which needs a live model key. | Those chat specs fail; every other spec still runs. | | `ANTHROPIC_API_KEY` | The 4 LLM-dependent specs (chat rename/delete/submit, critical-path "ask a question") send a message and assert a **streamed** answer. With the key set they run and must pass. | Those 4 specs **skip** (see `e2e/llm.ts`) instead of hanging, so the run is still green on the other ~23 specs. |
Add it under **Settings → Secrets and variables → Actions → New repository The suite is green **without** any secret — the LLM specs skip themselves via
secret**. For pull requests opened from a **fork**, GitHub withholds secrets by `test.skip(!process.env.ANTHROPIC_API_KEY, …)`, which keeps keyless runs (local,
default — a maintainer approves the run (or re-runs from the branch) so the key and fork PRs with no secret access) green and fast. Add the key under
is available. Treat that approval as the point where the chat specs become **Settings → Secrets and variables → Actions → New repository secret** to also
enforceable for external contributions. run and enforce the LLM specs. For fork PRs, GitHub withholds secrets until a
maintainer approves the run.
## Make it merge-blocking ## Make it merge-blocking