A KV cache is the memory an autoregressive language model keeps while it generates a response. It stores the key and value tensors already calculated for earlier tokens, so the model does not have to rebuild them at every next-token step.

The result is a tradeoff rather than free speed. Reusing those tensors reduces repeated computation, but the cache grows with the sequence and occupies accelerator memory. That is why context length, batch size and serving capacity are closely connected even when the model weights do not change.

On this page

Why generation repeats work without a cache

A causal language model produces one token, adds it to the sequence and then predicts the next token. The new token can attend to earlier tokens, but not to future ones.

Inside each attention layer, the model projects token representations into queries, keys and values. Once an earlier token has been processed, its key and value do not change just because a later token is added. Without caching, however, an implementation would calculate the earlier keys and values again for every generation step.

The Hugging Face Transformers documentation describes the difference directly: without caching, each step recomputes the previous keys and values; with caching, it computes only the current token’s keys and values and appends them to the stored tensors.

This is an inference optimization. Hugging Face warns that the cache should be used for inference rather than training.

What the KV cache stores

The cache does not store the original prompt as plain text. It stores numerical tensors produced by the model’s attention layers.

Hugging Face documents the common tensor shape as:

[batch size, attention heads, sequence length, head dimension]

There is a key tensor and a value tensor for each layer. When a new token is processed, its key and value are added along the sequence-length dimension. The next token’s query is then compared with the cached keys, and the resulting attention weights are applied to the cached values.

Because caching happens in every relevant attention layer, memory use depends on more than the number of prompt tokens. Batch size, layer count, the number of key-value heads, head dimension and numeric precision also matter.

Why it speeds up decoding

The cache removes one particular source of repeated work: recalculating keys and values for tokens the model has already processed.

According to the Transformers documentation, attention cost per generation step changes from quadratic growth with sequence length when earlier keys and values are recomputed to linear growth when those tensors are cached. The model still has to compare the new query with the available keys. A longer sequence therefore still costs more than a short one.

This distinction matters when reading latency claims. A KV cache does not make long-context generation constant-cost, and it does not remove the initial pass over the prompt. It makes repeated decoding work cheaper after the prompt has been processed.

For the broader distinction between building and running a model, see our AI model explainers. Hardware specifications are collected in AI chips compared.

Why long contexts consume more memory

A dynamic cache grows as new tokens are added. Longer prompts and longer outputs therefore retain more key-value state.

That memory competes with the model weights, temporary tensors and other requests on the same accelerator. A server may support a model’s advertised maximum context window but handle fewer simultaneous requests at that length. The exact limit depends on the model architecture, precision, serving software and hardware.

Grouped-query attention and multi-query attention reduce the number of key-value head groups compared with conventional multi-head attention. NVIDIA’s TensorRT-LLM documentation says its cache allocates state for the key-value head groups actually used: one group for multi-query attention, one per head for multi-head attention, with grouped-query attention between them. Fewer key-value groups can reduce cache memory.

Dynamic, static and sliding-window caches

A cache implementation does not have to grow in only one way.

Dynamic caches append key-value tensors as the sequence grows. They are straightforward and fit generation where the final length is not known in advance.

Static caches allocate a fixed capacity up front. Hugging Face notes that fixed-size cache layers can work with torch.compile, but unused capacity may consume memory.

Sliding-window caches retain only the active attention window for layers that do not attend to the full sequence. When old tokens fall outside the window, their cache state can be shifted out or freed. This reduces memory for models that use local attention, but it also means those layers cannot attend to the discarded positions.

The correct choice comes from the model architecture and serving workload, not from a universal ranking.

Reusing a prefix across requests

Some serving systems can reuse cached state between requests that begin with the same token prefix.

TensorRT-LLM stores filled cache blocks in a search structure and looks for matching blocks when a new request arrives. A matched block can be reused instead of recomputed. This can help when many requests share a long system prompt, a document prefix or another identical opening.

The match must be exact at the token and cache-key level used by the serving system. Similar-looking text is not enough. The serving layer also needs an eviction policy because memory is finite. TensorRT-LLM documents prioritized least-recently-used eviction, with lower-priority blocks removed before higher-priority ones.

Cross-request reuse creates an isolation question as well as a performance opportunity. TensorRT-LLM documents cache salting so blocks are shared only between requests with the same salt. Operators should follow the serving system’s security model rather than treating prefix reuse as a purely local optimization.

Offloading and eviction

When accelerator memory fills, a serving engine may evict cache blocks or move them to host memory.

NVIDIA documents optional KV-cache offloading from GPU to CPU memory. An offloaded block remains reusable, but it must be copied back before use. That preserves more cached work at the cost of transfer time and host-memory capacity.

Eviction, offloading and retention priorities are workload controls. A chat service with many short conversations may prefer different settings from a document-analysis service with long shared prefixes. A larger cache can improve reuse but also leaves less accelerator memory for concurrent requests.

What the KV cache does not prove

A product saying it uses a KV cache does not establish a specific speedup. Performance depends on prompt length, output length, batch size, model architecture, precision, memory bandwidth, cache policy and whether prefixes actually repeat.

It also does not mean the model remembers a conversation after the request or session state is removed. The KV cache is runtime state used by the serving process. It is different from saved chat history, retrieval systems and long-term product memory.

Finally, cache capacity is not the same as a model’s advertised context window. The context window is an architectural or product limit on tokens available to the model. Cache allocation is one serving constraint that determines how many of those tokens and how many requests can fit on available hardware.

Sources and documentation date

Documentation checked October 11, 2026.