mirror of
https://github.com/rowboatlabs/rowboat.git
synced 2026-07-12 21:02:17 +02:00
cda925ad9d
Anthropic caching is opt-in, and nothing sent breakpoints: observed sessions show 0% cache hits on Claude models (~1.27M of 1.36M sampled input tokens billed at full rate) vs ~82% implicit hits on Gemini. Two ephemeral breakpoints fix that: the system prompt (whose cache prefix also covers the tool schemas — both immutable per turn by construction) and the last message (Anthropic's incremental-conversation pattern). Conservative simulation on the sampled traffic floors the saving at 44% with no cross-turn reuse; realistic reuse lands 70-85%. Applied in the model registry bridge just before streamText, gated by provider flavor or model id (covers direct Anthropic, OpenRouter, and the gateways — the installed OpenRouter provider reads the same providerOptions.anthropic key). Transport-only: nothing is persisted, message content is untouched, and non-Anthropic requests pass through byte-identical. Verify with cachedInputTokens on model_call_completed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| docs | ||
| scripts | ||
| src | ||
| .gitignore | ||
| package.json | ||
| tsconfig.build.json | ||
| tsconfig.json | ||
| vitest.config.ts | ||