mirror of
https://github.com/willchen96/mike.git
synced 2026-07-26 23:51:08 +02:00
282b2dd271
Investigated whether the four LLM-gated specs (chat rename, chat delete, project-assistant create+submit, critical path) could run keyless with the demo model they select. They cannot on this codebase: no keyless model exists here — ModelToggle's MODELS are all Anthropic/Google/OpenAI entries gated on a configured provider key (env or user-stored, backend/src/lib/userApiKeys.ts), ChatInput.handleSubmit hard-blocks a send for an unavailable model, and the backend model set (backend/src/lib/llm/models.ts ALL_MODELS) has no demo id either. The 'Demo (no key needed)' model the specs target lives only in the amal66 fork. So the blanket test.skip(!hasLlmKey, ...) guard stays as-is — the evidence does not support narrowing it. What this change does instead: - docs/e2e-ci.md: new 'Enable the LLM specs' section with the exact UI path and gh CLI command for the ANTHROPIC_API_KEY repository secret, the fork-PR secrets caveat (use a maintainer branch or workflow_dispatch to exercise the specs), expected per-run cost, and how to confirm in the Actions log that the specs ran (27 passed, no skipped line) rather than skipped (4 skipped / 23 passed). Also documents the known gap: until the specs' selectDemoModel helper is pointed at a Claude model (or the demo provider is upstreamed), the secret unskips the specs but they fail at model selection. - e2e/llm.ts: rewrite the doc comment with the precise gate rationale (no keyless model in this repo; title-generation 500s are best-effort and not the reason) and pointers to the setup docs. - .github/workflows/e2e.yml: comment-only updates aligning the skip narrative (23 keyless specs, not ~20/~23) and pointing at the new docs section. No behavioral change anywhere. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
39 lines
2.1 KiB
TypeScript
39 lines
2.1 KiB
TypeScript
/**
|
|
* Helpers for specs that require a live LLM turn.
|
|
*
|
|
* Four specs (chat rename, chat delete, project-assistant create+submit, and
|
|
* the critical-path "ask a question" flow) create/populate a chat by sending a
|
|
* message. This codebase ships no keyless model: every entry in the model
|
|
* picker (frontend/src/app/components/assistant/ModelToggle.tsx MODELS) is an
|
|
* Anthropic/Google/OpenAI model whose availability requires a configured
|
|
* provider key — backend env var or user-stored key
|
|
* (backend/src/lib/userApiKeys.ts) — and ChatInput.handleSubmit refuses to
|
|
* send when the selected model's key is missing (ApiKeyMissingPopup). The
|
|
* backend enforces the same model set server-side
|
|
* (backend/src/lib/llm/models.ts ALL_MODELS/providerForModel). So with no key
|
|
* there is no way for these specs to send a message at all; unguarded they
|
|
* would hang to their timeout.
|
|
*
|
|
* The auto title-generation call (POST /chat/:id/generate-title) is NOT why
|
|
* the gate exists: keyless it just returns 500, and the specs already treat it
|
|
* as best-effort (`.catch(() => null)`).
|
|
*
|
|
* In CI the key is the `ANTHROPIC_API_KEY` repository secret, which
|
|
* `.github/workflows/e2e.yml` exposes both to the backend (backend/.env) and
|
|
* to the Playwright process. Guarding with
|
|
* `test.skip(!hasLlmKey, LLM_SKIP_REASON)` keeps a keyless run (a plain local
|
|
* run, or a fork PR with no secret access) green and fast on the other 23
|
|
* specs, while still running — and enforcing — the LLM specs whenever the key
|
|
* is present. Setup steps: docs/e2e-ci.md, "Enable the LLM specs".
|
|
*
|
|
* Known gap: the specs' selectDemoModel helper still targets a keyless
|
|
* "Demo (no key needed)" model that exists in the amal66 fork but not in this
|
|
* repository, so once unskipped they additionally need that helper pointed at
|
|
* a Claude model (see docs/e2e-ci.md, "Known gap: model selection"). The
|
|
* skip-guard itself is correct either way — keyless, no model in this repo
|
|
* can send.
|
|
*/
|
|
export const hasLlmKey = Boolean(process.env.ANTHROPIC_API_KEY);
|
|
|
|
export const LLM_SKIP_REASON =
|
|
"requires a model key — set the ANTHROPIC_API_KEY secret to run LLM-dependent specs";
|