KV cache and prompt caching: a developer's guide

TL;DR

The KV cache stores each token's attention keys and values during one generation so the model never recomputes them while it writes the next token. Prompt caching keeps that same KV state after the request ends and reuses it for later requests that start with an identical prefix.

One is a data structure that lives for a single request; the other is a storage policy that lives for minutes to an hour across many requests. The KV cache speeds up decoding (output tokens). Prompt caching speeds up prefill (reading the prompt) and cuts input cost by up to 90%.

Background: how an LLM serves a request

Every request runs in two phases with very different performance profiles: prefill reads the whole prompt at once, then decode generates output one token at a time.

Attention in one paragraph

In each transformer layer, every token is projected into three vectors: a query (Q, "what am I looking for?"), a key (K, "what do I contain?") and a value (V, "what do I contribute?"). A token's output is a weighted sum of all earlier tokens' values, weighted by how well its query matches their keys:

\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

The key point for caching: a token's K and V depend only on that token and the tokens before it. Once computed, they never change, because causal attention never looks ahead.

Prefill vs decode

Prefill cost grows with prompt length: a 100K-token prompt can take several seconds before the first token appears. Decode cost grows with output length. The KV cache targets decode; prompt caching targets prefill.

KV cache: reuse within one generation

The KV cache stores the key and value vectors of every token processed so far, per layer, so each new output token computes only its own K and V and reads the rest from memory.

Analogy: the running meeting notes

Imagine writing meeting minutes where every new sentence must account for everything said before. Without notes, you replay the full recording before writing each sentence. With notes, you glance at your summary of each earlier point and write the next line. The KV cache is those notes: one entry per token, written once, read many times.

Without vs with the cache

Without a cache, generating token t means re-running attention over all t − 1 earlier tokens and recomputing their K and V. Generating n tokens therefore does roughly 1 + 2 + … + n token-projections, which is O(n²) work.

With a cache, step t computes K and V for one token, appends them, and attends over the stored ones. Projection work drops to O(n) total. Attention itself still reads all t cached entries at each step, which is why long contexts get slower per token, but nothing is recomputed.

The cache grows by one K and V per layer each step; nothing already stored is ever recomputed.

What a step looks like

# Simplified single-layer decode step with a KV cache
def decode_step(x_new, cache, W_q, W_k, W_v):
    q = x_new @ W_q            # query for the new token only
    k = x_new @ W_k            # key for the new token only
    v = x_new @ W_v            # value for the new token only

    cache.k = concat(cache.k, k)   # append, never recompute
    cache.v = concat(cache.v, v)

    scores = softmax(q @ cache.k.T / sqrt(d_k))
    return scores @ cache.v

In Hugging Face Transformers this is the past_key_values object, enabled by default via use_cache=True. Serving engines such as vLLM, TensorRT-LLM and SGLang manage it for you.

Properties worth remembering

• Scope: one request (one sequence in a batch). Freed when generation ends.
• Location: GPU memory (HBM), alongside the model weights.
• Visibility: none to an API user. It is always on and never billed separately.
• Trade-off: it swaps compute for memory, and that memory becomes the main limit on context length and batch size (next section).

KV cache memory math

KV cache size grows linearly with context length and batch size, and at long contexts it can exceed the model weights themselves.

The formula

\text{KV bytes} = 2 \times L \times H_{kv} \times d_{head} \times T \times B \times p

The 2 covers keys and values. L = layers, H_kv = KV heads, d_head = head dimension, T = tokens in context, B = sequences in the batch, p = bytes per element (2 for FP16/BF16, 1 for FP8).

Worked examples (BF16, one sequence)

Model shapes below are from the published Llama 3 configs; totals are computed from the formula.

The 8B model's weights are about 16 GB in BF16, so a single 128K-token conversation needs as much memory for its cache as for the model. Batch 8 such users and the cache needs 128 GiB.

How the industry shrinks it

• Grouped-query attention (GQA) and multi-query attention (MQA): many query heads share fewer K/V heads. Llama 3 uses 8 KV heads for 32 query heads, a 4× saving (row 1 vs row 2).
• Multi-head latent attention (MLA): used in DeepSeek V2/V3; stores a compressed latent vector per token instead of full K/V and expands it at attention time.
• KV quantization: storing the cache in FP8 or INT4 halves or quarters it, at a small accuracy cost.
• Sliding-window attention: some layers only keep the last N tokens (Mistral, Gemma), capping cache growth.
• PagedAttention (vLLM): allocates the cache in fixed-size blocks like OS virtual memory pages, instead of one contiguous buffer per sequence sized for the maximum length. This removes most fragmentation and lets sequences share identical blocks, which is exactly what makes prefix caching cheap (next section).

Prompt caching: reuse across requests

Prompt caching saves the KV state of a prompt's prefix after a request finishes, then skips prefill for that prefix when a later request starts with exactly the same tokens.

Analogy: the briefing pack

A consultant gets a 200-page briefing pack before every client call. Without caching, they reread the whole pack before each call, even though only the client's latest question changed. With caching, they keep their annotated copy on the desk for a while. Each call they only read the new question. If someone edits page 3 of the pack, every annotation after page 3 is stale, and they must reread from there.

How it works

1. First request (cache write): the provider runs a normal prefill, then stores the KV blocks for the prefix, keyed by a hash of the tokens.
2. Later request (cache read): the provider hashes the incoming prompt prefix block by block, finds the longest matching stored prefix, loads its KV state, and prefills only the new suffix.
3. Expiry: entries are evicted after a time-to-live (TTL) with no hits, or under memory pressure. Each hit usually refreshes the TTL.

The prefix rule

Matching is by exact token prefix, because each token's K/V depends on every token before it. Change one token at position 500 and positions 500 onward must be recomputed, even if the remaining text is identical. Order therefore matters:

• Stable first: tool definitions, system prompt, reference documents, few-shot examples.
• Growing middle: conversation history (each turn extends the cached prefix).
• Volatile last: the current user message, timestamps, per-request IDs.

Where the cache lives

Serving systems tier it: hot entries stay in GPU memory, warm ones spill to CPU RAM, and some systems persist to SSD or a distributed store. Loading a cached prefix from CPU is still far faster than recomputing it on long prompts. These details are hidden behind the API, but they explain why caches are short-lived by default: holding gigabytes of KV state per prefix costs real memory.

Prompt caching in practice

The three common setups differ mainly in who decides where the cache boundary sits: you (Anthropic explicit breakpoints, OpenAI explicit mode), the provider (Anthropic automatic, OpenAI implicit), or the serving engine (vLLM).

Anthropic Claude API

Two modes. Automatic: a top-level cache_control caches everything up to the last cacheable block and moves the breakpoint forward as a conversation grows. Explicit: put cache_control on up to 4 content blocks. The prefix order is always tools → system → messages. (docs)

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=1024,
    system=[
        {"type": "text", "text": "You analyse legal contracts."},
        {
            "type": "text",
            "text": LONG_CONTRACT_TEXT,             # stable, large
            "cache_control": {"type": "ephemeral"},  # breakpoint here
        },
    ],
    messages=[{"role": "user", "content": "List the termination clauses."}],
)

u = response.usage
print(u.cache_creation_input_tokens,  # written this call
      u.cache_read_input_tokens,      # served from cache
      u.input_tokens)                 # after the last breakpoint

Key rules:

• TTL: 5 minutes by default, refreshed free on every hit. Add "ttl": "1h" for the 1-hour cache.
• Minimum size: 512 to 4,096 tokens depending on the model (512 for the current Opus 5.5 and Sonnet 5.5; 4,096 for Haiku 4.5). Shorter prefixes are silently not cached.
• Lookback: reads search up to 20 blocks back from a breakpoint for an earlier write.
• Concurrency: an entry becomes readable only once the first response starts, so fire one request before a parallel fan-out.
• Pre-warming: send the prefix with max_tokens: 0 to write the cache without generating output.
• Isolation: caches are per workspace; organizations never share them.

OpenAI

Caching is on by default for supported models; you only reorder your prompt. On GPT-5.6 and later you can also set prompt_cache_options.mode to explicit and mark prompt_cache_breakpoint on content blocks. Minimum is 1,024 tokens, and entries live at least 30 minutes after the last write or hit. (docs)

from openai import OpenAI

client = OpenAI()

resp = client.responses.create(
    model="gpt-5.6",
    prompt_cache_options={"mode": "explicit"},
    input=[
        {"role": "developer", "content": [{
            "type": "input_text",
            "text": STABLE_INSTRUCTIONS,
            "prompt_cache_breakpoint": {"mode": "explicit"},
        }]},
        {"role": "user", "content": user_question},
    ],
)

d = resp.usage.input_tokens_details
print(d.cached_tokens, d.cache_write_tokens)

Self-hosted: vLLM automatic prefix caching

vLLM hashes each fixed-size KV block (by default 16 tokens) together with all blocks before it. Any new request whose leading blocks hash the same reuses those physical blocks, with LRU eviction when GPU memory fills. It is enabled by default in the V1 engine; the flag below makes it explicit.

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct",
          enable_prefix_caching=True)

prefix = SYSTEM_PROMPT + DOCUMENT          # shared, long
prompts = [prefix + q for q in questions]  # only the tail differs

outputs = llm.generate(prompts, SamplingParams(max_tokens=256))
# The first prompt pays full prefill; the rest reuse the cached blocks.

The same idea appears as RadixAttention in SGLang (a radix tree of cached prefixes) and as KV cache reuse in TensorRT-LLM. Self-hosting means no per-token discount, but the GPU time saved is real throughput.

Cost and latency numbers

On Anthropic's API a cache read costs 10% of normal input on most models (5% on Opus 5.5), so a 5-minute cache pays for itself on the very first reuse.

Anthropic pricing (USD per million input tokens)

Output tokens are priced normally; caching never changes them. The multipliers stack with the Batch API discount. Source: Anthropic prompt caching docs. OpenAI on GPT-5.6+ uses the same shape: 1.25× write, 0.1× read on most models (OpenAI docs).

Break-even

Cost of sending a prefix N times, in units of one uncached pass, at the standard 0.1× read rate:

The 5-minute cache wins from the 2nd request; the 1-hour cache wins from the 3rd. A single-use prompt marked for caching costs 25% (or 100%) more, so never cache content you won't reuse.

Worked example: document Q&A bot

Assume a 50,000-token system prompt plus reference document on Claude Sonnet 5.5, 1,000 questions per hour, traffic steady enough that the 5-minute cache never expires.

• Without caching: 1,000 × 50K = 50M tokens × $2.00/M = $100.00 per hour.
• With caching: one write of 50K × $2.50/M = $0.13, plus 999 reads × 50K = 49.95M × $0.20/M = $9.99. Total ≈ $10.12 per hour, about 90% less.

User messages and outputs cost the same in both cases and are left out. In real traffic, gaps longer than the TTL add extra writes; measure your actual hit rate.

Latency

Caching cuts time to first token, not generation speed. The saving scales with the cached length: a few hundred milliseconds for a 5K-token prefix, several seconds for 100K+ tokens. Anthropic reported up to 85% lower latency on long prompts when it launched the feature (approximate, secondary source). The 5-minute and 1-hour caches give the same latency benefit.

One more Anthropic-specific benefit: cache reads do not count against input-token rate limits, so caching also raises effective throughput.

Design patterns and pitfalls

Most caching failures come from one cause: something early in the prompt changes between requests without anyone noticing.

Patterns that work

• Layer by change frequency. Tools (change on deploy) → system instructions (change weekly) → reference documents (change daily) → conversation (changes every turn). With Anthropic, a breakpoint at the end of each layer lets a document update keep the tools and instructions cached.
• Append, never rewrite. In multi-turn chats and agent loops, add messages to the end. Summarising, truncating or editing earlier turns resets the cache from the edit onward.
• Breakpoint on the last stable block. Not on the user's message. A breakpoint on a block that changes every request writes a new entry every time and never reads one.
• Warm before fan-out. Send one request (or a max_tokens: 0 pre-warm on Anthropic) before launching parallel calls that share a prefix.
• Pick TTL by traffic gap. Requests more often than every 5 minutes: default TTL, refreshed free. Gaps of 5 to 60 minutes (users thinking, long agent sub-tasks): 1-hour TTL.
• Pad to the minimum when it pays. A 900-token prefix sent thousands of times is worth growing to the model's cache minimum with useful examples.

Common cache breakers

Monitoring

Log the cache fields from every response and track token hit rate = cached tokens ÷ total input tokens, broken down by endpoint and day.

def cache_stats(usage):
    read = usage.cache_read_input_tokens or 0
    write = usage.cache_creation_input_tokens or 0
    fresh = usage.input_tokens
    total = read + write + fresh
    return {"hit_rate": read / total if total else 0.0,
            "write_share": write / total if total else 0.0}

Healthy chat and agent workloads usually show hit rates well above 70%. A high write share with a low hit rate means your breakpoint is on content that changes. Anthropic and OpenAI both offer cache diagnostics that report where two consecutive prompts diverged.

Security note

Caches are isolated per organization (and per workspace on Anthropic's API), so other customers cannot probe yours. Inside your own multi-tenant app, timing or usage differences could in principle reveal whether another user sent the same prefix. If that matters, keep tenant data out of shared prefixes or use per-tenant cache keys where the provider supports them.

Side-by-side comparison

The KV cache is the data; prompt caching is the decision to keep that data and share it between requests.

Five things to remember

1. K and V for a token never change once computed, which is what makes both kinds of caching possible.
2. The KV cache trades compute for memory; at long contexts its memory can match the model's weights.
3. Prompt caching only works on an exact token prefix, so put stable content first and volatile content last.
4. With a 0.1× read price, a 5-minute cache pays off on the second request.
5. Measure cache read tokens on every response; a silent miss looks exactly like a working request.

Sources

• Anthropic: Prompt caching (pricing, TTLs, minimums, lookback, usage fields)
• OpenAI: Prompt caching (implicit and explicit modes, 30-minute TTL, pricing multipliers)
• Introl: Prompt caching infrastructure (latency reduction figure)
• KV cache sizes are computed from the published Llama 3 model configs; vLLM and SGLang behaviour is described from their documentation and papers, not re-verified for this doc.