Back to Curriculum
Advanced•Deep Learning
Transformers
The attention mechanism that powers modern AI.
Interactive Playground
Initializing Interactive Playground...
Research-Level Deep Dive & Equations
The core mechanism of the Transformer architecture (Vaswani et al., 2017) is **Scaled Dot-Product Attention**. Given query matrix , key matrix , and value matrix , attention is formulated as:
where is the dimensionality of queries and keys.
### Mathematical Proof of the Scaling Factor:
To understand the necessity of the scaling factor , let's analyze the variance of the dot product before softmax.
Let and be query and key vectors representing a single row in and . Assume that the components and are independent random variables, each with mean 0 and variance 1:
We assume that and are mutually independent for all indices .
The dot product is:
1. **Mean of the Dot Product**:
Using the linearity of expectation:
2. **Variance of the Dot Product**:
Since the elements are mutually independent:
For two independent random variables and with mean 0:
Thus:
Summing across all dimensions:
### Softmax Gradient Saturation:
As the query/key dimension grows large, the variance of the dot products scales linearly with . This means the dot products are highly likely to take on extremely large positive or negative values.
Recall that the derivative of softmax is . When the input logits have high variance, the softmax distribution becomes extremely sharp, allocating a probability near to the maximal element and near to all other elements. In this regime, the derivatives vanish completely, causing **gradient death** during backpropagation.
Dividing the dot product by scales the variance back to 1:
This keeps the inputs in the active, high-gradient region of the softmax function, stabilizing optimization.
Key Equations
Test Your Knowledge
Check whether you have mastered this concept with a quick quiz.
Was this lesson helpful?
Your feedback helps us continuously improve the curriculum and interactive visualizations.