From e6d53c4f0ff32a95dc78323defec1ba8208f54f6 Mon Sep 17 00:00:00 2001 From: QA Runner Date: Fri, 17 Jul 2026 16:16:32 -0700 Subject: [PATCH] docs(e2e): document keyless-green behavior, migrations, and prod build The suite is green with no secret (LLM specs skip). Update docs/e2e-ci.md and the workflow header to reflect the migrations step, the production build, and the optional (not required) ANTHROPIC_API_KEY. Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/e2e.yml | 10 +++++----- docs/e2e-ci.md | 41 +++++++++++++++++++++++++-------------- 2 files changed, 31 insertions(+), 20 deletions(-) diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index 7b5ee2e8..f322fcd1 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -7,11 +7,11 @@ # this job "e2e / playwright" as a required status check in branch protection — # see docs/e2e-ci.md ("Make it merge-blocking"). # -# Required repository secret for a full pass: -# ANTHROPIC_API_KEY — the critical-path / chat specs send a message and expect a -# streamed answer, which needs a live model key. Add it under -# Settings > Secrets and variables > Actions. Without it the -# chat specs fail; every other spec runs key-less. +# The suite is green with NO secret: the 4 LLM-dependent specs (chat + critical +# path) skip themselves when ANTHROPIC_API_KEY is absent (see e2e/llm.ts), so a +# keyless run passes the other ~23 specs. Set the ANTHROPIC_API_KEY secret under +# Settings > Secrets and variables > Actions to also run and enforce those 4. +# See docs/e2e-ci.md for the full picture. # # The e2e user (e2e@mike.local) is bootstrapped in-job by e2e/auth.setup.ts against # the local Supabase admin API, so no E2E_PASSWORD secret is needed — the defaults diff --git a/docs/e2e-ci.md b/docs/e2e-ci.md index 8435df46..ed6f60a9 100644 --- a/docs/e2e-ci.md +++ b/docs/e2e-ci.md @@ -12,31 +12,42 @@ On every `pull_request` targeting `main` (or `upstream-main`, the fork mirror), and on manual `workflow_dispatch`, the `e2e / playwright` job: 1. installs the root (Playwright), `backend/`, and `frontend/` dependencies; -2. boots **MinIO** (S3-compatible object storage — three specs upload documents); -3. boots **local Supabase** (Auth + Postgres) via the Supabase CLI and loads - `backend/schema.sql` — the same fresh-database schema the README tells a human - to run; +2. boots **MinIO** (S3-compatible object storage — several specs upload documents); +3. boots **local Supabase** (Auth + Postgres) via the Supabase CLI, loads + `backend/schema.sql`, then applies every dated migration in `backend/migrations/` + on top. `schema.sql` is meant to be the latest shape but in practice lags the + migrations (e.g. it is missing `workflow_open_source_submissions`, which + `GET /workflows/:id` queries — a 500 without the migrations). It also grants + `service_role` full access to the `public` tables afterward, because + `schema.sql` revokes client grants assuming a hosted Supabase where + `service_role` is already privileged; 4. writes `backend/.env` and `frontend/.env.local` from the live Supabase values; -5. starts the backend API (`:3001`) and the Next.js web app (`:3000`), waits for - both to be healthy; +5. **builds** the web app (`next build`) and serves it with `next start` — a + production build, not `next dev`, so there is no on-demand compilation (which + makes first-hit page loads slow enough to time out specs) and no dev + hydration-error overlay (whose injected DOM pollutes text locators). Starts the + backend API (`:3001`) and the web server (`:3000`) and waits for both healthy; 6. runs `npx playwright test` and uploads the HTML report + traces as an artifact - (`playwright-report`) on both pass and fail. + (`playwright-report`) on pass, fail, or timeout. `e2e/auth.setup.ts` bootstraps the shared test user (`e2e@mike.local`) against the local Supabase admin API, so no login secret is needed — the credentials baked into that file are the single source of truth. -## Required secret +Typical run: **~7 minutes**, **23 passed / 4 skipped / 0 failed** with no secret. -| Secret | Why | Without it | +## Optional secret (fuller coverage) + +| Secret | What it unlocks | Without it | |---|---|---| -| `ANTHROPIC_API_KEY` | The critical-path / chat specs send a message and assert a **streamed** answer, which needs a live model key. | Those chat specs fail; every other spec still runs. | +| `ANTHROPIC_API_KEY` | The 4 LLM-dependent specs (chat rename/delete/submit, critical-path "ask a question") send a message and assert a **streamed** answer. With the key set they run and must pass. | Those 4 specs **skip** (see `e2e/llm.ts`) instead of hanging, so the run is still green on the other ~23 specs. | -Add it under **Settings → Secrets and variables → Actions → New repository -secret**. For pull requests opened from a **fork**, GitHub withholds secrets by -default — a maintainer approves the run (or re-runs from the branch) so the key -is available. Treat that approval as the point where the chat specs become -enforceable for external contributions. +The suite is green **without** any secret — the LLM specs skip themselves via +`test.skip(!process.env.ANTHROPIC_API_KEY, …)`, which keeps keyless runs (local, +and fork PRs with no secret access) green and fast. Add the key under +**Settings → Secrets and variables → Actions → New repository secret** to also +run and enforce the LLM specs. For fork PRs, GitHub withholds secrets until a +maintainer approves the run. ## Make it merge-blocking