mirror of
https://github.com/willchen96/mike.git
synced 2026-07-26 23:51:08 +02:00
docs(e2e): reconcile the docs with what this branch actually contains
The two doc commits were written against a lineage where the selectClaudeModel spec fix lived on a separate branch, which left three stale claims after the rebase: - e2e/llm.ts still called selectDemoModel a known gap and pointed at a docs section title that no longer exists; the specs in this very branch already use selectClaudeModel. Replace the paragraph with the current behavior. - docs/e2e-ci.md attributed the fix to "the e2e suite branch (#220)"; this branch carries that commit itself, so say so and name the commit. - The 23/4 keyless and 27/0 with-key figures were presented as bare facts ("verified locally"). Attribute them to where they were actually measured — the spec-fix commit's own verification against a full local stack — and state they have not been re-measured since the rebase onto current main. Drop the unattributed "~7 minutes" typical-run figure in favor of the structural 27-spec / 4-gated breakdown (confirmed by `npx playwright test --list`: 27 tests in 7 files). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
0cc635bd5e
commit
72242ff0eb
2 changed files with 16 additions and 15 deletions
|
|
@ -34,7 +34,8 @@ and on manual `workflow_dispatch`, the `e2e / playwright` job:
|
|||
the local Supabase admin API, so no login secret is needed — the credentials
|
||||
baked into that file are the single source of truth.
|
||||
|
||||
Typical run: **~7 minutes**, **23 passed / 4 skipped / 0 failed** with no secret.
|
||||
A keyless run is expected to end **23 passed / 4 skipped / 0 failed** — the
|
||||
suite has 27 specs, 4 of them LLM-gated (see "Confirm the specs ran" below).
|
||||
|
||||
## Optional secret (fuller coverage)
|
||||
|
||||
|
|
@ -105,18 +106,21 @@ Open the **Run Playwright** step in the Actions log:
|
|||
|
||||
The uploaded `playwright-report` artifact shows the same per-spec statuses.
|
||||
|
||||
### Model selection (gap fixed on the e2e suite branch)
|
||||
### Model selection (gap fixed in this branch)
|
||||
|
||||
Earlier revisions of the four specs drove the model picker to a keyless
|
||||
**"Demo (no key needed)"** entry that exists in the amal66 fork but **not in
|
||||
this repository** (no `mike-demo` model id, no demo provider), so setting the
|
||||
secret unskipped them and they then failed at model selection. This is fixed on
|
||||
the e2e suite branch (#220): the specs' `selectClaudeModel` helper picks
|
||||
**Claude Sonnet 4.6** in the ModelToggle whenever the key is set, and the
|
||||
critical-path response assertion checks for a nonempty streamed assistant
|
||||
answer instead of the fork's canned demo reply. With that fix, setting the
|
||||
`ANTHROPIC_API_KEY` secret yields **27 passed / 0 skipped** (verified locally:
|
||||
keyless 23 passed / 4 skipped; with the key 27 / 0). The secret setup above is
|
||||
secret unskipped them and they then failed at model selection. This branch
|
||||
carries the fix (commit `test(e2e): select a real Claude model in LLM-gated
|
||||
specs`): the specs' `selectClaudeModel` helper picks **Claude Sonnet 4.6** in
|
||||
the ModelToggle whenever the key is set, and the critical-path response
|
||||
assertion checks for a nonempty streamed assistant answer instead of the
|
||||
fork's canned demo reply. The **23 passed / 4 skipped** keyless and
|
||||
**27 passed / 0 skipped** with-key figures come from that commit's own
|
||||
verification against a full local stack (backend, prod-build frontend, local
|
||||
Supabase + MinIO — see its commit message); they have not been re-measured
|
||||
since this branch was rebased onto current `main`. The secret setup above is
|
||||
all that is needed.
|
||||
|
||||
## Make it merge-blocking
|
||||
|
|
|
|||
|
|
@ -26,12 +26,9 @@
|
|||
* specs, while still running — and enforcing — the LLM specs whenever the key
|
||||
* is present. Setup steps: docs/e2e-ci.md, "Enable the LLM specs".
|
||||
*
|
||||
* Known gap: the specs' selectDemoModel helper still targets a keyless
|
||||
* "Demo (no key needed)" model that exists in the amal66 fork but not in this
|
||||
* repository, so once unskipped they additionally need that helper pointed at
|
||||
* a Claude model (see docs/e2e-ci.md, "Known gap: model selection"). The
|
||||
* skip-guard itself is correct either way — keyless, no model in this repo
|
||||
* can send.
|
||||
* When the key IS set, the specs' selectClaudeModel helper picks "Claude
|
||||
* Sonnet 4.6" in the ModelToggle (see docs/e2e-ci.md, "Model selection"), so
|
||||
* the unskipped specs submit against a model this repository actually ships.
|
||||
*/
|
||||
export const hasLlmKey = Boolean(process.env.ANTHROPIC_API_KEY);
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue