apunkt/rowboat
1
0
Fork 0
mirror of https://github.com/rowboatlabs/rowboat.git synced 2026-07-15 21:11:08 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

feat(core): Anthropic prompt-cache breakpoints in the model registry

Browse source 
Anthropic caching is opt-in, and nothing sent breakpoints: observed
sessions show 0% cache hits on Claude models (~1.27M of 1.36M sampled
input tokens billed at full rate) vs ~82% implicit hits on Gemini. Two
ephemeral breakpoints fix that: the system prompt (whose cache prefix
also covers the tool schemas — both immutable per turn by construction)
and the last message (Anthropic's incremental-conversation pattern).
Conservative simulation on the sampled traffic floors the saving at 44%
with no cross-turn reuse; realistic reuse lands 70-85%.

Applied in the model registry bridge just before streamText, gated by
provider flavor or model id (covers direct Anthropic, OpenRouter, and
the gateways — the installed OpenRouter provider reads the same
providerOptions.anthropic key). Transport-only: nothing is persisted,
message content is untouched, and non-Anthropic requests pass through
byte-identical. Verify with cachedInputTokens on model_call_completed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Ramnique Singh 2026-07-07 21:03:35 +05:30
parent 7000f34c48
commit cda925ad9d
5 changed files with 273 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

6
apps/x/packages/core/docs/turn-runtime-design.md
View file

@ -418,6 +418,12 @@ bytes than were transmitted; the inspect CLI prints the active policy so
the divergence is visible. If exact-bytes replay ever becomes a hard
requirement, record the applied policy on the turn and compose from that.
Relatedly, Anthropic-family requests get cache_control breakpoints stamped
at the transport layer (`models/prompt-caching.ts`, applied inside the
model registry bridge just before streamText). These are provider metadata
only — message content is untouched, nothing is persisted, and inspect
does not render them; non-Anthropic requests pass through byte-identical.
### 6.7 Agent snapshot inheritance
Session turns whose resolved system prompt and tool set are byte-identical