mirror of
https://github.com/willchen96/mike.git
synced 2026-07-26 23:51:08 +02:00
test(e2e): select a real Claude model in LLM-gated specs (demo model is fork-only)
The 4 LLM-gated specs (chat-management rename/delete/project-assistant and
the critical-path project flow) selected a "Demo (no key needed)" model in
the ModelToggle. That model exists only in a fork's demo provider — upstream
ModelToggle.MODELS has no such entry — so on this repo, setting the
ANTHROPIC_API_KEY secret unskipped the specs and they then failed at the
model-selection step.
Fix:
- Replace the selectDemoModel helpers with selectClaudeModel, which picks
"Claude Sonnet 4.6" (the cheapest Anthropic entry in ModelToggle.MODELS).
With ANTHROPIC_API_KEY exported to the backend, userApiKeys.envApiKey()
reports the claude provider as configured, so the model is available and
the submit is not blocked by the ApiKeyMissingModal.
- critical-path Step 8: the canned-reply assertion getByText("Demo mode")
becomes a presence + nonempty assertion on the assistant answer container
(MarkdownContent's "div.prose.font-serif.text-gray-900", unique to
assistant answer content) — deterministic against nondeterministic real
LLM output, same 60s budget.
- Comments rewritten to describe the upstream reality: specs run only when
ANTHROPIC_API_KEY is set (e2e/llm.ts), the backend uses the env key, and
title generation resolves to claude-haiku-4-5 via resolveTitleModel.
Verified against a local prod-build stack (backend :3001, frontend :3002,
local Supabase + MinIO), only ANTHROPIC_API_KEY configured:
- keyless Playwright env: 23 passed / 4 skipped
- ANTHROPIC_API_KEY exported: 27 passed / 0 skipped (1.2m)
- npx tsc --noEmit clean
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
QA Runner
parent
86a0d78157
commit
039e433a74
2 changed files with 77 additions and 60 deletions
|
|
@ -32,23 +32,26 @@ async function ensureSidebarOpen(page: Page) {
|
|||
}
|
||||
|
||||
/**
|
||||
* Select the built-in keyless "demo" model in the chat input's ModelToggle so
|
||||
* the first submit actually creates a chat instead of opening the
|
||||
* ApiKeyMissingModal.
|
||||
* Select a Claude model in the chat input's ModelToggle so the first submit
|
||||
* actually creates a chat instead of opening the ApiKeyMissingModal.
|
||||
*
|
||||
* The default model is "gemini-3-flash-preview" (ModelToggle.DEFAULT_MODEL_ID),
|
||||
* for which no key is configured; ChatInput.handleSubmit (ChatInput.tsx:116-119)
|
||||
* then refuses to send. The suite runs WITHOUT any provider key (the CI stack
|
||||
* leaves ANTHROPIC_API_KEY empty), so no Anthropic/Gemini/OpenAI model is
|
||||
* available — only the demo model (DEMO_MODEL_ID "mike-demo", label "Demo (no
|
||||
* key needed)") is always available and streams a canned response via
|
||||
* providers/demo.ts. ModelToggle renders a Radix DropdownMenu: the trigger is a
|
||||
* button whose title is "Choose model" (current model available) or "API key
|
||||
* missing for selected model" (current model not available — the default-Gemini
|
||||
* case). We open it, pick the Demo item, and confirm the trigger now shows
|
||||
* "Demo (no key needed)".
|
||||
* The specs that call this run only when ANTHROPIC_API_KEY is set in the
|
||||
* Playwright environment (test.skip(!hasLlmKey, ...) — e2e/llm.ts). The CI
|
||||
* stack exports the same secret to the backend, whose key resolution
|
||||
* (userApiKeys.ts envApiKey()) falls back to the ANTHROPIC_API_KEY env var, so
|
||||
* the "claude" provider reports as configured and ModelToggle shows the
|
||||
* Anthropic models as available. The default model, however, is
|
||||
* "gemini-3-flash-preview" (ModelToggle.DEFAULT_MODEL_ID), for which no key is
|
||||
* configured in CI; ChatInput.handleSubmit (ChatInput.tsx:116-119) then refuses
|
||||
* to send. ModelToggle renders a Radix DropdownMenu: the trigger is a button
|
||||
* whose title is "Choose model" (current model available) or "API key missing
|
||||
* for selected model" (current model not available — the default-Gemini case).
|
||||
* We open it, pick "Claude Sonnet 4.6" (the cheapest Anthropic entry in
|
||||
* ModelToggle.MODELS), and confirm the trigger now shows that label.
|
||||
*/
|
||||
async function selectDemoModel(page: Page) {
|
||||
const CLAUDE_MODEL_LABEL = "Claude Sonnet 4.6";
|
||||
|
||||
async function selectClaudeModel(page: Page) {
|
||||
const trigger = page
|
||||
.locator(
|
||||
'button[title="Choose model"], button[title="API key missing for selected model"]',
|
||||
|
|
@ -56,12 +59,10 @@ async function selectDemoModel(page: Page) {
|
|||
.first();
|
||||
await expect(trigger).toBeVisible({ timeout: 10_000 });
|
||||
await trigger.click();
|
||||
await page
|
||||
.getByRole("menuitem", { name: "Demo (no key needed)" })
|
||||
.click();
|
||||
await page.getByRole("menuitem", { name: CLAUDE_MODEL_LABEL }).click();
|
||||
// After selection the trigger label reflects the chosen model.
|
||||
await expect(
|
||||
page.getByRole("button", { name: /Demo \(no key needed\)/ }),
|
||||
page.getByRole("button", { name: CLAUDE_MODEL_LABEL }),
|
||||
).toBeVisible({ timeout: 5_000 });
|
||||
}
|
||||
|
||||
|
|
@ -128,9 +129,10 @@ test("rename chat: sidebar rename interaction updates the title", async ({ page
|
|||
await page.goto("/assistant");
|
||||
const textarea = page.getByPlaceholder("How can I help?");
|
||||
await expect(textarea).toBeVisible({ timeout: 10_000 });
|
||||
// Pick the keyless demo model so the submit isn't blocked by the
|
||||
// ApiKeyMissingModal (no provider key is configured in this run).
|
||||
await selectDemoModel(page);
|
||||
// Pick a Claude model (available via the backend's ANTHROPIC_API_KEY) so
|
||||
// the submit isn't blocked by the ApiKeyMissingModal — the default Gemini
|
||||
// model has no key configured in this run.
|
||||
await selectClaudeModel(page);
|
||||
await textarea.fill(message);
|
||||
|
||||
// Sending the first message triggers auto title-generation
|
||||
|
|
@ -238,9 +240,10 @@ test("delete chat: sidebar delete action removes the chat from history", async (
|
|||
)
|
||||
.catch(() => null);
|
||||
|
||||
// Pick the keyless demo model so the submit isn't blocked by the
|
||||
// ApiKeyMissingModal, then send the first message.
|
||||
await selectDemoModel(page);
|
||||
// Pick a Claude model (available via the backend's ANTHROPIC_API_KEY) so
|
||||
// the submit isn't blocked by the ApiKeyMissingModal, then send the first
|
||||
// message.
|
||||
await selectClaudeModel(page);
|
||||
await textarea.fill(message);
|
||||
await textarea.press("Enter");
|
||||
await page.waitForURL(newChatUrl, { timeout: 20_000 });
|
||||
|
|
@ -372,12 +375,14 @@ test("project assistant: create a new chat and submit a question", async ({ page
|
|||
// prior response is in flight; otherwise it returns early. Two transient gates
|
||||
// exist under load:
|
||||
// • useSelectedModel (useSelectedModel.ts:16-20) seeds DEFAULT (Gemini) for
|
||||
// one render before its localStorage-read effect restores the demo model,
|
||||
// so a submit racing a ChatInput remount can momentarily see Gemini → the
|
||||
// ApiKeyMissingModal ("API key required") pops and the submit no-ops.
|
||||
// one render before its localStorage-read effect restores the Claude
|
||||
// model, so a submit racing a ChatInput remount can momentarily see
|
||||
// Gemini → the ApiKeyMissingModal ("API key required") pops and the
|
||||
// submit no-ops.
|
||||
// • A transient in-flight response (isResponseLoading) also no-ops the submit.
|
||||
// Re-select the demo model and re-submit until the textarea clears. A genuinely
|
||||
// broken send never clears on ANY attempt, so a real regression is still caught.
|
||||
// Re-select the Claude model and re-submit until the textarea clears. A
|
||||
// genuinely broken send never clears on ANY attempt, so a real regression is
|
||||
// still caught.
|
||||
const question = "What is in this project?";
|
||||
const apiKeyModalHeading = page.getByRole("heading", {
|
||||
name: "API key required",
|
||||
|
|
@ -393,8 +398,8 @@ test("project assistant: create a new chat and submit a question", async ({ page
|
|||
.click()
|
||||
.catch(() => {});
|
||||
}
|
||||
// Re-assert the demo model as the active (available) model after any remount.
|
||||
await selectDemoModel(page);
|
||||
// Re-assert the Claude model as the active (available) model after any remount.
|
||||
await selectClaudeModel(page);
|
||||
await chatInput.fill(question);
|
||||
// ChatInput.handleKeyDown: Enter (no Shift) → handleSubmit().
|
||||
await chatInput.press("Enter");
|
||||
|
|
|
|||
|
|
@ -13,17 +13,23 @@ import path from "path";
|
|||
const PDF_FIXTURE = path.join(__dirname, "fixtures/test.pdf");
|
||||
|
||||
/**
|
||||
* Select the built-in keyless "demo" model in the chat input's ModelToggle.
|
||||
* Select a Claude model in the chat input's ModelToggle.
|
||||
*
|
||||
* The default model (Gemini) has no key configured in this environment, so a
|
||||
* submit would be blocked by the ApiKeyMissingModal. The demo model
|
||||
* (ModelToggle DEMO_MODEL_ID) is always available and streams a canned response
|
||||
* via providers/demo.ts, letting the "receive a response" assertion run
|
||||
* deterministically without any provider key. The Radix DropdownMenu trigger's
|
||||
* title is "Choose model" (current model available) or "API key missing for
|
||||
* selected model" (default-Gemini case).
|
||||
* This spec runs only when ANTHROPIC_API_KEY is set in the Playwright
|
||||
* environment (test.skip(!hasLlmKey, ...) — e2e/llm.ts). The CI stack exports
|
||||
* the same secret to the backend, whose key resolution (userApiKeys.ts
|
||||
* envApiKey()) falls back to the ANTHROPIC_API_KEY env var, so the "claude"
|
||||
* provider reports as configured and ModelToggle shows the Anthropic models as
|
||||
* available. The default model (Gemini) has no key configured in CI, so a
|
||||
* submit with it would be blocked by the ApiKeyMissingModal. We pick
|
||||
* "Claude Sonnet 4.6" (the cheapest Anthropic entry in ModelToggle.MODELS) so
|
||||
* the request streams a real response. The Radix DropdownMenu trigger's title
|
||||
* is "Choose model" (current model available) or "API key missing for selected
|
||||
* model" (default-Gemini case).
|
||||
*/
|
||||
async function selectDemoModel(page: Page) {
|
||||
const CLAUDE_MODEL_LABEL = "Claude Sonnet 4.6";
|
||||
|
||||
async function selectClaudeModel(page: Page) {
|
||||
const trigger = page
|
||||
.locator(
|
||||
'button[title="Choose model"], button[title="API key missing for selected model"]',
|
||||
|
|
@ -31,12 +37,10 @@ async function selectDemoModel(page: Page) {
|
|||
.first();
|
||||
await expect(trigger).toBeVisible({ timeout: 10_000 });
|
||||
await trigger.click();
|
||||
await page
|
||||
.getByRole("menuitem", { name: "Demo (no key needed)" })
|
||||
.click();
|
||||
await page.getByRole("menuitem", { name: CLAUDE_MODEL_LABEL }).click();
|
||||
// After selection the trigger label reflects the chosen model.
|
||||
await expect(
|
||||
page.getByRole("button", { name: /Demo \(no key needed\)/ }),
|
||||
page.getByRole("button", { name: CLAUDE_MODEL_LABEL }),
|
||||
).toBeVisible({ timeout: 5_000 });
|
||||
}
|
||||
|
||||
|
|
@ -124,31 +128,39 @@ test("create project, upload PDF, ask a question and receive a response", async
|
|||
await page.waitForURL(/\/projects\/.+\/assistant/, { timeout: 10_000 });
|
||||
await page.waitForLoadState("networkidle", { timeout: 20_000 }).catch(() => {});
|
||||
|
||||
/* ── Step 7: select the keyless demo model, type a question, submit ───── */
|
||||
/* ── Step 7: select a Claude model, type a question, submit ───────────── */
|
||||
const chatInput = page.getByPlaceholder("How can I help?");
|
||||
await expect(chatInput).toBeVisible({ timeout: 10_000 });
|
||||
|
||||
/* The default Gemini model has no key configured, so submitting it would be
|
||||
blocked by the ApiKeyMissingModal. Select the keyless demo model so the
|
||||
request actually streams a response. */
|
||||
await selectDemoModel(page);
|
||||
blocked by the ApiKeyMissingModal. Select a Claude model (backed by the
|
||||
ANTHROPIC_API_KEY the backend reads from its environment) so the request
|
||||
actually streams a response. */
|
||||
await selectClaudeModel(page);
|
||||
await chatInput.fill("What is this document about?");
|
||||
/* This ChatInput submits on Enter (Shift+Enter inserts a newline). */
|
||||
await chatInput.press("Enter");
|
||||
|
||||
/* ── Step 8: verify the assistant streams a response ─────────────────── */
|
||||
/* The demo provider always opens its reply with "Demo mode …"
|
||||
(providers/demo.ts buildDemoAnswer). Its appearance proves the message
|
||||
was sent, streamed, and rendered end-to-end — deterministically and
|
||||
without any provider key.
|
||||
/* With a real Claude model the reply content is nondeterministic, so assert
|
||||
presence + nonempty rather than matching text. The assistant's answer
|
||||
renders through MarkdownContent (message/MarkdownContent.tsx), whose
|
||||
wrapper div carries "text-gray-900 … prose … font-serif" — a combination
|
||||
unique to assistant answer content on this page (user messages render a
|
||||
plain <p>, and the gray pre-response EventBlocks prose uses
|
||||
text-gray-400). Its appearance with nonempty text proves the message was
|
||||
sent, streamed, and rendered end-to-end.
|
||||
|
||||
The reply is preceded by a POST that persists the chat and a client-side
|
||||
route change to /assistant/chat/<id>; under the local-Supabase load that
|
||||
round-trip alone can outlast a 30s budget, so allow the same headroom the
|
||||
rest of this flow gets. */
|
||||
await expect(page.getByText("Demo mode").first()).toBeVisible({
|
||||
timeout: 60_000,
|
||||
});
|
||||
The reply is preceded by a POST that persists the chat, a client-side
|
||||
route change to /assistant/chat/<id>, and a real LLM round-trip; under
|
||||
local-Supabase load that can outlast a 30s budget, so allow the same
|
||||
headroom the rest of this flow gets. */
|
||||
const assistantAnswer = page
|
||||
.locator("div.prose.font-serif.text-gray-900")
|
||||
.first();
|
||||
await expect(assistantAnswer).toBeVisible({ timeout: 60_000 });
|
||||
/* Nonempty streamed text (any non-whitespace character). */
|
||||
await expect(assistantAnswer).toContainText(/\S/);
|
||||
});
|
||||
|
||||
/* ─── Test 3: login-page redirect for unauthenticated users ──────────────── */
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue