Back to Curriculum
Advanced•Deep Learning
Attention Variants & Norms
Grouped-Query Attention (GQA), Multi-Query (MQA), RoPE embeddings, and RMSNorm topology.
Interactive Playground
Initializing Interactive Playground...
Research-Level Deep Dive & Equations
Standard Multi-Head Attention (MHA) in LLMs uses independent Query heads, Key heads, and Value heads where (e.g. 32 heads in Llama-2 7B).
•The KV-Cache VRAM Bottleneck: During autoregressive text generation, to avoid recomputing Key and Value vectors for past context tokens at every step, all previous and vectors are cached in GPU VRAM memory.
•Exact KV-Cache Memory Formula: For a model with batch size , sequence length , layers, key-value heads, head dimension , and precision bytes (e.g., for FP16):
•Scale Example: For Llama-2 70B () at context length and batch size :
This far exceeds single GPU capacity (80 GB A100), bottlenecking serving throughput.
Key Equations
Test Your Knowledge
Check whether you have mastered this concept with a quick quiz.
Was this lesson helpful?
Your feedback helps us continuously improve the curriculum and interactive visualizations.