Back to Curriculum
Advanced•MLOps
Model Serving
High-throughput LLM hosting, KV caching, vLLM, speculative decoding, and quantization.
Interactive Playground
Initializing Interactive Playground...
Research-Level Deep Dive & Equations
During autoregressive generation in Transformer models, each newly generated token depends on all previous tokens in the sequence. Re-computing the Key () and Value () projections for all past tokens at every decoding step creates an computation bottleneck (where is the current sequence length). To mitigate this, **Key-Value (KV) Caching** stores the past key and value vectors in memory.
**Mathematical Dimensions and Memory Profile**:
Let:
* be the batch size.
* be the number of KV heads (equal to the number of query heads in Multi-Head Attention; or smaller in Multi-Query or Grouped-Query Attention).
* be the head dimension.
* be the number of transformer layers.
* be the sequence length (prompt length + generation length).
At precision bytes per parameter (typically 2 for FP16/BF16), the KV Cache memory footprint is:
*The factor of 2 accounts for storing both key and value tensors.*
For a 70B parameter model (, , ) at batch size 32 with a sequence length of 4096 tokens using FP16:
This massive memory requirement makes LLM serving strictly **memory-bandwidth bound** during the decoding phase.
Test Your Knowledge
Check whether you have mastered this concept with a quick quiz.
Was this lesson helpful?
Your feedback helps us continuously improve the curriculum and interactive visualizations.