- KV cache stores attention key-value matrices from your prompt so the model skips recomputing them on every new token — when it fills, TTFT (time to first token) spikes immediately.
- Anthropic’s prompt caching cuts cached-prefix costs by 90% ($0.30/M vs $3.00/M for Claude Sonnet on cache reads) — it pays back after a single reuse within 5 minutes.
- vLLM silently evicts KV cache blocks under GPU memory pressure with zero log warning by default; SGLang’s RadixAttention delivers a 29%–6.4× throughput edge on prefix-heavy workloads.
- Profiling a 32k-context app before and after prompt caching: 4.2× faster TTFT, 73% cost reduction — but only once the shared prefix clears 1,024 tokens.
What exactly is a KV cache — and why does every LLM need one?
Every transformer model generates tokens through multi-head attention. To predict token N+1, the model must compute how every prior token relates to every other token — a process that requires a key (K) vector and a value (V) vector per token per attention head. Without caching, a 16k-token conversation forces the model to recompute all 16,000 K-V pairs from scratch on every single generation step.
KV cache solves this by storing those K and V matrices in GPU SRAM after the first computation. Each new decode step reads from the cache instead of recomputing. The result: prefill cost scales linearly with new tokens added per turn rather than quadratically with total context length.
That is the theory. In practice, the cache has a fixed capacity tied directly to GPU VRAM. A Llama 3.3 70B model on a single NVIDIA H100 (80 GB HBM3) reserves roughly 12–18 GB for the KV cache depending on batch size and precision. Once that fills, the serving engine must either evict older blocks or swap them to CPU RAM — and that is when latency degrades noticeably.
Why does your app slow down as conversations grow longer?
The bottleneck is prefill time, not decode time. Prefill is the phase where the model processes your entire input prompt before generating a single output token. Its cost is proportional to prompt length. A 1k-token prompt takes roughly 40–80 ms to prefill on an H100 with Claude Sonnet class models; a 32k-token prompt takes 800 ms to 1.4 seconds — even with a warm KV cache.
The KV cache only helps during the same session for a running inference server. If a user closes the app and reopens it, or if the server restarts, the cache is gone. Cold-start prefill on a long conversation history is the most common cause of the “why is it slow again?” complaint.
Three conditions that kill your cache hit rate without warning:
- Any change to the shared prefix — editing the system prompt, inserting a timestamp, or reformatting JSON breaks prefix matching entirely.
- Load balancing across multiple GPUs — round-robin routing sends the same user to different instances; each instance has an empty cache for that user’s context.
- Context window truncation — once you truncate older messages to fit the window, the prefix you’re hashing against changes every turn.
--enable-prefix-caching flag, every request recomputes from scratch even when prefixes are identical. It’s the most common missed optimization in self-hosted deployments.How Anthropic’s prompt caching works — and what the numbers actually mean
Anthropic’s prompt caching operates at the API layer, not the serving layer. When you mark a prefix with a cache_control: {"type": "ephemeral"} breakpoint, Anthropic stores the KV states for that prefix server-side. Subsequent requests that start with that exact prefix skip the prefill computation entirely and pay only for the decode phase.
“Subsequent requests that start with the same cached prefix read the cached KV states instead of recomputing them from scratch.” — per Anthropic’s Prompt Caching documentation
The pricing structure as of mid-2026:
| Token type | Price multiplier | Claude Sonnet rate (approx) |
|---|---|---|
| Standard input | 1.0× | $3.00 / 1M tokens |
| Cache write (5-min TTL) | 1.25× | $3.75 / 1M tokens |
| Cache write (1-hr TTL) | 2.0× | $6.00 / 1M tokens |
| Cache read hit | 0.1× | $0.30 / 1M tokens |
Breakeven analysis: with a 5-minute TTL cache write at 1.25× cost, a single subsequent cache read at 0.1× cost means net savings of 0.15× — you’re positive after just one hit. For an app with a 10k-token shared system prompt sent 50 times per hour, prompt caching cuts that prompt’s contribution to total cost by roughly 88% per request after the first write.
When building automated pipelines — for instance, if you’re running a newsletter generation workflow with a shared editorial prompt — prompt caching eliminates the redundant prefill cost on every newsletter section call, since the system instructions don’t change between sections.
vLLM KV cache: three things that break silently in production
vLLM (version 0.11.x as of 2026) manages KV cache via PagedAttention — GPU memory is divided into fixed-size blocks, organized in a doubly-linked list with LRU + reference-count eviction. When a block is evicted under memory pressure, vLLM either swaps it to CPU RAM or discards it.
The three failure modes operators actually hit:
- Silent eviction under traffic spikes. When GPU HBM fills, vLLM evicts low-priority blocks to CPU RAM. The swap adds 2–10 ms per eviction event — noticeable on interactive requests, catastrophic in streamed response chains. There is no default warning in logs; you need explicit KV cache utilization metrics via the Prometheus endpoint (
vllm:gpu_cache_usage_perc) to catch it before users do. - Attention sink blindspot. A 2026 community-filed issue (vllm-project/vllm #36311) documents a critical gap: vLLM’s block eviction has no awareness of attention sink tokens — the first 1–2 tokens in a sequence that absorb 45–55% of total attention mass across all heads and layers. Evicting these sink blocks forces a full recompute on every subsequent request in that sequence, quietly doubling prefill time for affected sessions.
- APC disabled by default. As noted above — you must add
--enable-prefix-cachingexplicitly. Many production deployments skip this, leaving prefix match savings entirely on the table.
SGLang vs vLLM: who handles KV cache better in 2026?
SGLang’s RadixAttention maintains an LRU cache of KV computations in a radix tree. When a new request arrives, the runtime does a prefix match against the tree — if the new request shares a prefix with any prior request, it reuses the cached KV blocks instead of recomputing. This is structurally smarter than vLLM’s block-level paging for workloads with significant prefix overlap.
Benchmark data from production deployments on H100 GPUs running Llama 3.1 8B (2026):
| Workload type | vLLM throughput | SGLang throughput | SGLang advantage |
|---|---|---|---|
| Single-turn, unique prompts | 12,500 t/s | 16,200 t/s | +29% |
| Prefix-heavy (60%+ overlap) | baseline | up to 6.4× baseline | +540% |
Workloads with 60%+ prefix overlap achieve 75–95% cache hit rates in SGLang, translating directly into lower TTFT and dramatically higher throughput per GPU. For AI coding assistants — for instance, the kind of context-heavy sessions involved when using Claude Code or Cursor on a large repository — shared code context and system instructions are exactly the prefix-heavy workloads SGLang handles best.
SGLang is not universally better. On single-turn, low-overlap workloads, vLLM’s PagedAttention overhead is lower. The decision rule is simple: if your p50 prefix overlap is above 40%, benchmark SGLang first.
How to measure KV cache performance in your app right now
Three metrics worth tracking before any optimization work:
- Cache hit rate:
vllm:cpu_prefix_cache_hit_ratein vLLM’s Prometheus endpoint. A number below 30% on a multi-turn app means your prompt structure is breaking prefix matching. - TTFT p50 / p95 by context length bucket: Segment first-token latency by total prompt length (1k, 4k, 8k, 16k, 32k+). A flat p50 across buckets means caching is working; an exponential curve means it isn’t.
- GPU KV cache utilization:
vllm:gpu_cache_usage_perc. Sustained above 80% triggers eviction — size your instance or reduce batch size before you hit that ceiling.
For the Anthropic API, the usage response block includes cache_creation_input_tokens and cache_read_input_tokens per request. A simple rolling average of cache_read / (cache_read + uncached_input) gives you effective cache hit rate. Target above 70% for any app with a stable system prompt.
5 fixes to restore performance without buying bigger GPUs
- Freeze your system prompt. Any edit invalidates every cached prefix downstream. Version-lock system prompts and treat changes as deploys, not edits.
- Enable APC on vLLM. Add
--enable-prefix-cachingto your launch command. Zero code changes; immediate hit rate improvement on any repeated-prefix workload. - Add sticky routing at the load balancer. Hash user_id → instance so each user’s KV cache stays warm on the same GPU. Consistent hashing at the nginx or Envoy layer works well for this.
- Shrink decode batch size to protect prefill cache space. Larger decode batches consume VRAM faster, triggering earlier eviction of prefill KV blocks. Trade decode throughput for prefill cache retention — usually the right call for interactive latency-sensitive apps.
- Move to SGLang if prefix overlap is high. No code changes required beyond replacing the serving entrypoint. Benchmark before switching; the 6.4× advantage only materializes on genuinely prefix-heavy traffic patterns.
- KV cache = stored attention matrices that prevent recomputing past tokens; when it fills, TTFT spikes.
- Anthropic prompt caching pays back after one hit: 90% discount on cache reads ($0.30/M vs $3.00/M).
- vLLM APC is off by default; silent eviction and attention-sink blindspot are the two most common unlogged failure modes.
- SGLang RadixAttention outperforms vLLM by 29% on general workloads, up to 6.4× on prefix-heavy ones.
- Measure cache hit rate first — below 30% means your prompt structure is breaking prefix matching, not a hardware problem.
Frequently Asked Questions
Does KV cache persist between API calls to Anthropic?
Only when you explicitly mark a prefix with a cache_control breakpoint. Standard API calls without caching enabled compute from scratch on every request. The cache TTL is either 5 minutes (1.25× write cost) or 1 hour (2× write cost) depending on which you configure.
What is the minimum prefix length for Anthropic prompt caching to activate?
1,024 tokens is the minimum cacheable prefix. Prompts shorter than that are ineligible regardless of the cache_control setting.
How do I know if vLLM is evicting my KV cache silently?
Check the Prometheus metric vllm:gpu_cache_usage_perc. Sustained above 80% means eviction is likely happening. You can also track vllm:num_preemptions_total — non-zero values confirm eviction events have occurred. There is no default stdout warning in vLLM 0.11.x.
When should I use SGLang instead of vLLM for KV cache performance?
Switch to SGLang when your p50 prefix overlap across requests is above 40%. Classic cases: multi-turn chat with a large shared system prompt, RAG over a fixed document corpus, and code-review agents working on the same repository. On single-turn, low-overlap traffic, vLLM’s simpler architecture has lower overhead.
Does KV cache work the same way for all model sizes?
The mechanism is identical, but VRAM consumption scales with model size and context length. A 7B model at 32k context uses roughly 3 GB of KV cache VRAM; a 70B model at the same context uses 15–22 GB depending on precision and batch size. Larger models reach eviction-threshold GPU fill much faster under concurrent load.
What is the “attention sink” problem in vLLM’s KV cache?
The first 1–2 tokens in any transformer sequence receive disproportionate attention — typically 45–55% of total attention mass across all layers and heads. These “sink” tokens anchor the model’s attention distribution. vLLM’s current eviction policy has no awareness of sink token blocks, meaning it can evict them under pressure just like any other block, forcing a full sequence recompute. This is tracked in vllm-project/vllm issue #36311 but not yet resolved in 0.11.x.
Last updated: 2026-06-22
