Back to Curriculum
Advanced•Modern AI
LLMs
Large Language Models and the GPT architecture.
Interactive Playground
Initializing Interactive Playground...
Research-Level Deep Dive & Equations
Large Language Models do not operate on raw characters or whole words — they operate on tokens, sub-word units derived through data-driven segmentation algorithms. The choice of tokenizer profoundly affects model vocabulary size, sequence length, and downstream performance.
Byte-Pair Encoding (BPE): Originally a data compression algorithm (Gage, 1994), BPE was adapted for NLP by Sennrich et al. (2016). The algorithm starts with a vocabulary of individual characters (or bytes) and iteratively merges the most frequent adjacent pair of tokens into a single new token. After k merge operations, the vocabulary size is |V_base| + k. GPT-2 and GPT-3 use byte-level BPE, starting from 256 byte values rather than Unicode characters, guaranteeing that any input can be tokenized without unknown tokens.
WordPiece: Developed by Schuster & Nakajima (2012) and used in BERT, WordPiece is similar to BPE but selects merges that maximize the likelihood of the training corpus under a unigram language model. The merge criterion is: choose the pair (a, b) that maximizes P(ab) / (P(a) · P(b)), i.e., the mutual information between adjacent tokens.
SentencePiece: Proposed by Kudo & Richardson (2018), SentencePiece treats the input as a raw byte stream and applies either BPE or a unigram language model. The unigram model assigns a probability to each sub-word candidate and finds the segmentation that maximizes the total log-likelihood via the Viterbi algorithm. SentencePiece is language-agnostic and does not require pre-tokenization (e.g., whitespace splitting), making it ideal for multilingual models like T5 and LLaMA.
The tokenizer defines the model's "perception" of language. A vocabulary of ~32K–100K tokens balances compression efficiency (fewer tokens per sentence) against embedding table size. Fertility — the average number of tokens per word — is a key metric: lower fertility means more efficient processing. Languages underrepresented in training data often have higher fertility, leading to performance disparities.
Key Equations
Test Your Knowledge
Check whether you have mastered this concept with a quick quiz.
Was this lesson helpful?
Your feedback helps us continuously improve the curriculum and interactive visualizations.