Attention Mechanism
How a model decides which words to focus on when understanding a sentence.
Attention is the mathematical operation that lets a Transformer weigh the relevance of each token relative to every other token when building its internal representation.
For each token, attention computes three vectors: a Query (what am I looking for?), a Key (what do I represent?), and a Value (what information do I carry?). Dot products between Queries and Keys produce attention scores — how much each token should "attend to" each other.
Multi-head attention runs this process in parallel across multiple "heads," each learning different relationship patterns: one might track syntactic structure, another semantic similarity, another long-range dependencies.
Why it matters for practitioners:
- Long-range dependencies are handled naturally (no vanishing gradient)
- Context window limits arise from the O(n²) cost of computing all pairwise attention scores
- Longer contexts cost quadratically more compute — a 1M-token context requires ~1 trillion attention calculations per layer
In plain terms
When re-reading a legal contract to understand clause 7, your eyes jump back to the definitions section, skip the boilerplate, and focus on the relevant precedent — attention is that selective focus, done in parallel for every word.