docs(e2e): reconcile the docs with what this branch actually contains

The two doc commits were written against a lineage where the
selectClaudeModel spec fix lived on a separate branch, which left three
stale claims after the rebase:

- e2e/llm.ts still called selectDemoModel a known gap and pointed at a
  docs section title that no longer exists; the specs in this very
  branch already use selectClaudeModel. Replace the paragraph with the
  current behavior.
- docs/e2e-ci.md attributed the fix to "the e2e suite branch (#220)";
  this branch carries that commit itself, so say so and name the commit.
- The 23/4 keyless and 27/0 with-key figures were presented as bare
  facts ("verified locally"). Attribute them to where they were actually
  measured — the spec-fix commit's own verification against a full local
  stack — and state they have not been re-measured since the rebase onto
  current main. Drop the unattributed "~7 minutes" typical-run figure in
  favor of the structural 27-spec / 4-gated breakdown (confirmed by
  `npx playwright test --list`: 27 tests in 7 files).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
QA Runner 2026-07-22 11:41:27 -07:00
parent 0cc635bd5e
commit 72242ff0eb
2 changed files with 16 additions and 15 deletions

View file

@ -34,7 +34,8 @@ and on manual `workflow_dispatch`, the `e2e / playwright` job:
the local Supabase admin API, so no login secret is needed — the credentials
baked into that file are the single source of truth.
Typical run: **~7 minutes**, **23 passed / 4 skipped / 0 failed** with no secret.
A keyless run is expected to end **23 passed / 4 skipped / 0 failed** — the
suite has 27 specs, 4 of them LLM-gated (see "Confirm the specs ran" below).
## Optional secret (fuller coverage)
@ -105,18 +106,21 @@ Open the **Run Playwright** step in the Actions log:
The uploaded `playwright-report` artifact shows the same per-spec statuses.
### Model selection (gap fixed on the e2e suite branch)
### Model selection (gap fixed in this branch)
Earlier revisions of the four specs drove the model picker to a keyless
**"Demo (no key needed)"** entry that exists in the amal66 fork but **not in
this repository** (no `mike-demo` model id, no demo provider), so setting the
secret unskipped them and they then failed at model selection. This is fixed on
the e2e suite branch (#220): the specs' `selectClaudeModel` helper picks
**Claude Sonnet 4.6** in the ModelToggle whenever the key is set, and the
critical-path response assertion checks for a nonempty streamed assistant
answer instead of the fork's canned demo reply. With that fix, setting the
`ANTHROPIC_API_KEY` secret yields **27 passed / 0 skipped** (verified locally:
keyless 23 passed / 4 skipped; with the key 27 / 0). The secret setup above is
secret unskipped them and they then failed at model selection. This branch
carries the fix (commit `test(e2e): select a real Claude model in LLM-gated
specs`): the specs' `selectClaudeModel` helper picks **Claude Sonnet 4.6** in
the ModelToggle whenever the key is set, and the critical-path response
assertion checks for a nonempty streamed assistant answer instead of the
fork's canned demo reply. The **23 passed / 4 skipped** keyless and
**27 passed / 0 skipped** with-key figures come from that commit's own
verification against a full local stack (backend, prod-build frontend, local
Supabase + MinIO — see its commit message); they have not been re-measured
since this branch was rebased onto current `main`. The secret setup above is
all that is needed.
## Make it merge-blocking

View file

@ -26,12 +26,9 @@
* specs, while still running and enforcing the LLM specs whenever the key
* is present. Setup steps: docs/e2e-ci.md, "Enable the LLM specs".
*
* Known gap: the specs' selectDemoModel helper still targets a keyless
* "Demo (no key needed)" model that exists in the amal66 fork but not in this
* repository, so once unskipped they additionally need that helper pointed at
* a Claude model (see docs/e2e-ci.md, "Known gap: model selection"). The
* skip-guard itself is correct either way keyless, no model in this repo
* can send.
* When the key IS set, the specs' selectClaudeModel helper picks "Claude
* Sonnet 4.6" in the ModelToggle (see docs/e2e-ci.md, "Model selection"), so
* the unskipped specs submit against a model this repository actually ships.
*/
export const hasLlmKey = Boolean(process.env.ANTHROPIC_API_KEY);