Transformer
The architecture that made modern AI possible — attention over sequences.
The Transformer (Vaswani et al., 2017 — "Attention Is All You Need") is the neural network architecture underlying virtually every modern LLM. It replaced recurrent networks (RNNs) by processing all tokens in parallel rather than sequentially, enabling training on much larger datasets.
The key innovation is the self-attention mechanism: for every token, the model computes how much each other token is relevant to understanding it. "The bank by the river was steep" — the word "bank" attends to "river" to resolve ambiguity.
A Transformer consists of stacked encoder and/or decoder layers, each containing multi-head self-attention and feed-forward sub-layers. GPT models use decoder-only; BERT uses encoder-only; original machine translation used both.
Scale: GPT-2 (2019) had 1.5B parameters; GPT-4 is estimated at ~1.7T. Every billion parameters requires significant GPU memory at inference time.
In plain terms
An editing team where every word in a document simultaneously reads and responds to every other word — rather than reading left to right one word at a time.
Related concepts
Large Language Model
AI trained on vast text to understand and generate language.
Attention Mechanism
How a model decides which words to focus on when understanding a sentence.
Embeddings
Turning words and ideas into numbers that capture meaning.
Tokens
The tiny chunks of text an AI reads, one at a time.