Back to Curriculum
Advanced•Modern AI
State Space Models & Mamba
Continuous-time SSMs, selective scan parameters, discretization, and linear-time sequence modeling.
Interactive Playground
Initializing Interactive Playground...
Research-Level Deep Dive & Equations
State Space Models (SSMs) map a continuous 1D input sequence to a continuous output sequence through an -dimensional hidden state representation .
•Continuous Linear Time-Invariant (LTI) Differential Equations:
•Matrix Dimensions: controls hidden state memory dynamics, projects scalar inputs into state space, projects hidden states to output, and is the feedthrough skip connection.
•HiPPO Initialization: Standard random initialization causes vanishing/exploding state dynamics over long context windows. The High-order Polynomial Projection Operators (HiPPO) framework initializes matrix such that tracks sliding memory polynomial projections of historical inputs:
Key Equations
PyTorch HiPPO Matrix & Continuous SSM Initializationpython
import torch
import torch.nn as nn
import math
def make_hippo_matrix(N: int) -> torch.Tensor:
"""Generates the continuous HiPPO-LegS memory matrix A."""
P = torch.sqrt(1 + 2 * torch.arange(N, dtype=torch.float32))
A = P.unsqueeze(1) * P.unsqueeze(0)
A = torch.tril(A, diagonal=-1) + torch.diag(torch.arange(N, dtype=torch.float32) + 1)
return -A
class ContinuousSSM(nn.Module):
def __init__(self, state_dim: int = 64):
super().__init__()
self.state_dim = state_dim
self.A = nn.Parameter(make_hippo_matrix(state_dim))
self.B = nn.Parameter(torch.randn(state_dim, 1) / math.sqrt(state_dim))
self.C = nn.Parameter(torch.randn(1, state_dim) / math.sqrt(state_dim))
self.D = nn.Parameter(torch.ones(1))Test Your Knowledge
Check whether you have mastered this concept with a quick quiz.
Was this lesson helpful?
Your feedback helps us continuously improve the curriculum and interactive visualizations.