Back to Curriculum
Intermediate•Deep Learning
Backpropagation
How neural networks learn using the chain rule.
Interactive Playground
Initializing Interactive Playground...
Research-Level Deep Dive & Equations
At the core of deep learning optimization is the concept of a **Computational Graph**, a directed acyclic graph (DAG) where nodes represent mathematical operations (variables, tensors, or operators) and directed edges represent the flow of variables. Backpropagation is a highly specialized, computationally optimal implementation of **reverse-mode Automatic Differentiation (AD)** applied to these graphs.
To appreciate reverse-mode AD, we must compare it to forward-mode AD. Suppose we have a function .
- **Forward-Mode AD** computes the derivatives of all intermediate variables with respect to a single input variable in a single forward pass. To compute the full Jacobian , forward-mode requires separate passes, scaling as .
- **Reverse-Mode AD** computes the derivatives of a single output variable (the scalar loss , where ) with respect to all intermediate and input variables in a single backward pass. Its computational complexity scales as , which is for a scalar loss.
For a modern deep neural network with parameters and a scalar loss , naive forward-mode AD or numerical finite-difference approximation would require forward passes, making gradient computation mathematically impossible under realistic time constraints. Reverse-mode AD (backpropagation) allows us to compute the exact gradient with respect to *all* parameters in just **one backward pass**, taking roughly twice the time of a single forward pass.
Let a computational graph consist of vertices ordered such that represent the inputs, represent intermediate variables, and represents the final scalar loss. The forward pass computes:
The backward pass computes the **adjoint** (or error signal) for each variable, defined as:
Using the multivariate chain rule, we compute the adjoints in reverse order ():
By caching the intermediate activations during the forward pass, we can compute the partial derivatives analytically during the backward pass without re-evaluating the operations, achieving linear time complexity .
Key Equations
Test Your Knowledge
Check whether you have mastered this concept with a quick quiz.
Was this lesson helpful?
Your feedback helps us continuously improve the curriculum and interactive visualizations.