Prompt Caching
Reusing the processed version of a repeated prompt prefix to cut cost and latency.
Prompt caching allows providers to save the KV cache (the model's internal representation) of a prompt prefix so that subsequent calls reusing the same prefix skip reprocessing those tokens.
Cost impact: Anthropic charges 10% of normal input price for cache hits; OpenAI charges 50%. For a 4,000-token system prompt sent 1 million times: uncached = $8/M input × 4k tokens × 1M calls = $32,000. Cached = $3,200.
How to use it (Anthropic):
Mark static prompt sections with "cache_control": {"type": "ephemeral"}. The cache persists for 5 minutes and refreshes on each hit. Cache breakpoints can be set at up to 4 positions in a prompt.
What to cache: System prompts, large static documents (product manuals, codebases, transcripts), few-shot examples, tool definitions — anything identical across many calls.
Latency benefit: Cache hits also reduce time-to-first-token because the provider skips processing thousands of tokens.
When it's not applicable: Short prompts (< 1,024 tokens for Anthropic), highly dynamic content that changes every call, one-off requests.
In plain terms
Like saving a compiled version of a document. The first time you compile it takes work; every subsequent use of that compiled version is instant.