Transformers Explained: The Architecture That Changed Everything

En bref : The transformer is the neural network architecture behind every major AI system today — GPT-4, Claude, Gemini, Stable Diffusion, and hundreds more. Introduced in the 2017 paper “Attention Is All You Need,” transformers replaced recurrent neural networks (RNNs) by processing entire sequences in parallel through a self-attention mechanism that lets each element consider its relationship to every other element. This single architectural innovation unlocked the scaling that produced the generative AI revolution.

Eight Pages That Reshaped an Industry

In June 2017, a team of eight researchers at Google published a paper with a title that read like a dare: “Attention Is All You Need.” The paper proposed replacing the dominant neural network architecture for language tasks — recurrent neural networks — with something entirely new. They called it the transformer.

The paper did not invent attention mechanisms. Those had existed for years as add-ons to existing architectures. What it proposed was far more radical: an architecture built entirely from attention, with no recurrence and no convolution. The established wisdom said this should not work. The experimental results said otherwise.

Within two years, transformers had displaced RNNs and LSTMs in natural language processing. Within five years, they had spread to computer vision, protein structure prediction, speech synthesis, and robotics. Every large language model that powers the current AI revolution — GPT-4, Claude, Gemini, LLaMA, Mistral — is a transformer. Understanding how they work is understanding the engine of modern AI.

Why RNNs Had to Go

To understand why transformers mattered, you need to understand what they replaced.

Recurrent neural networks (RNNs) and their improved variant, Long Short-Term Memory networks (LSTMs), processed sequences one element at a time. To understand the word “crashed” in the sentence “The stock market, which had been rising steadily for months despite warnings from economists about overheating, finally crashed,” the RNN had to process every preceding word sequentially, maintaining a hidden state that carried information forward.

This sequential processing created two problems. First, it was slow — each step depended on the previous step, so the computation could not be parallelized across the many cores of a modern GPU. Training on large datasets took impractically long.

Second, information degraded over distance. By the time the RNN reached “crashed,” the information about “stock market” — 20 words earlier — had been compressed through a bottleneck hidden state, diluted by every intermediate word. LSTMs partially addressed this with gating mechanisms, but the fundamental problem remained: long-range dependencies were hard to capture.

Transformers solved both problems simultaneously.

Self-Attention: The Core Innovation

The transformer’s central mechanism — self-attention — allows every element in a sequence to directly attend to every other element, regardless of distance. No sequential processing. No information bottleneck. Direct connections between any two positions.

Here is how it works, step by step.

Queries, Keys, and Values

For each token in the input, the transformer computes three vectors: a query (Q), a key (K), and a value (V). Think of it like a search engine. The query represents “what am I looking for?” The key represents “what do I contain?” The value represents “what information should I pass along if selected?”

The attention score between two tokens is computed by taking the dot product of one token’s query with another token’s key. A high dot product means the two tokens are relevant to each other. These scores are normalized using softmax to create a probability distribution — attention weights — that sum to 1.

The output for each position is a weighted sum of all values, where the weights come from the attention scores. Tokens that are highly relevant to each other exchange the most information. Tokens that are irrelevant to each other exchange almost none.

Multi-Head Attention

A single attention computation captures one type of relationship between tokens. But language has many types of relationships — syntactic, semantic, referential, temporal. The transformer handles this by running multiple attention computations in parallel, each with its own learned Q, K, V weight matrices. These are called attention heads.

A typical transformer layer might have 12 to 128 attention heads. One head might learn to track subject-verb agreement. Another might track pronoun references. Another might track semantic similarity. The outputs of all heads are concatenated and projected through a linear layer to produce the layer’s output.

This parallelism is not just elegant — it is computationally efficient. Because attention computations are matrix multiplications, they map perfectly onto GPU hardware designed for exactly these operations. This is why transformers train faster than RNNs despite processing more information per step.

The Computational Cost

Self-attention has a cost: it scales quadratically with sequence length. Every token attends to every other token, so doubling the sequence length quadruples the computation. For a 1,000-token input, that is 1 million attention computations per layer. For a 100,000-token input, it is 10 billion.

This quadratic scaling is why context windows were initially small (1,024 tokens for GPT-2, 4,096 for GPT-3). Extending context windows to millions of tokens required innovations like FlashAttention (which optimizes memory access patterns), sparse attention (which skips attention computations between distant tokens), and sliding window attention (which limits attention to a local neighborhood plus selected global positions).

Positional Encoding: Teaching Order to a Parallel System

Self-attention is inherently orderless. The attention between tokens depends only on their content, not their position. The sentence “dog bites man” and “man bites dog” would produce identical attention scores without some way to encode position.

Positional encoding solves this by adding position information directly to the token embeddings. The original transformer paper used sinusoidal functions — different frequencies for different positions — so the model could both identify absolute positions and compute relative distances between tokens.

Modern transformers use learned positional encodings or, increasingly, Rotary Position Embedding (RoPE), which encodes relative positions through rotation matrices applied to the query and key vectors. RoPE is particularly effective for extending context lengths beyond the training distribution, which is why it has been adopted by LLaMA, Mistral, and other open-source models.

The Encoder-Decoder Architecture

The original transformer paper described an encoder-decoder architecture, designed for sequence-to-sequence tasks like machine translation.

The encoder processes the input sequence (e.g., a French sentence) through multiple layers of self-attention and feed-forward networks, producing a rich representation of the input.

The decoder generates the output sequence (e.g., the English translation) one token at a time. It uses two types of attention: self-attention over the output generated so far, and cross-attention over the encoder’s representation of the input. The cross-attention mechanism allows each generated token to “look back” at the full input.

A critical detail: the decoder uses masked self-attention, which prevents each position from attending to future positions. When generating the fourth word of a translation, the decoder can only attend to the first three words — not the fifth or sixth. This ensures that generation is auto-regressive (each token depends only on previous tokens) while still leveraging the parallel computation of attention.

Encoder-Only and Decoder-Only Variants

The original encoder-decoder architecture spawned two influential variants.

Encoder-only models (BERT, RoBERTa) use only the encoder stack. Because there is no masked attention, every token can attend to every other token in both directions. This bidirectional attention makes encoder-only models excellent at understanding tasks — classification, named entity recognition, sentiment analysis — but unable to generate text.

Decoder-only models (GPT series, Claude, LLaMA) use only the decoder stack with masked self-attention. These are the models that power generative AI — they generate text one token at a time, each token attending only to previous tokens. Despite having only “half” of the original architecture, decoder-only models have proven remarkably versatile, handling understanding tasks through in-context learning rather than architectural features.

The dominance of decoder-only models in 2024-2026 is one of the most surprising developments in AI. A simpler architecture, scaled massively, outperformed the more complex encoder-decoder design that was theoretically better suited to many tasks.

Feed-Forward Networks and Layer Norms

Attention is the star of the transformer, but two supporting components are essential.

Feed-forward networks (FFNs) follow each attention layer. These are simple two-layer neural networks applied independently to each token position. While attention captures relationships between tokens, FFNs transform individual token representations — adding non-linearity and capacity for storing factual knowledge. Research suggests that FFNs act as key-value memories, storing associations learned during training.

Layer normalization stabilizes training by normalizing the inputs to each sublayer. Without it, training deep transformers (80+ layers) would be numerically unstable, with gradient values either exploding to infinity or vanishing to zero. The placement of layer norms (before or after each sublayer) is a design choice that affects training dynamics — modern practice favors pre-norm (normalizing before each sublayer), which improves training stability for very deep models.

Scaling Laws: Why Bigger Works

The transformer’s most consequential property may be its scalability. Unlike previous architectures that plateaued or became unstable at large scales, transformers demonstrate smooth, predictable improvement as model size, dataset size, and compute increase.

The “scaling laws” documented by OpenAI (Kaplan et al., 2020) and DeepMind (Hoffmann et al., 2022) showed that model performance follows power-law relationships with parameter count and training data. Double the parameters, get a predictable improvement. Double the training data, get a predictable improvement.

This predictability transformed AI research from an art into something closer to engineering. Labs could estimate in advance how much compute and data a model needed to reach a target capability level. The evolution of AI models from GPT-2 to GPT-4 was not a series of lucky breakthroughs but a systematic march along these scaling curves.

Beyond Language: Transformers Everywhere

Transformers have escaped the domain of natural language processing. Vision Transformers (ViT) treat images as sequences of patches and process them with the same attention mechanism used for text. Decision Transformer applies the architecture to reinforcement learning. AlphaFold 2 used attention to predict protein structures, arguably one of the most important scientific breakthroughs of the decade.

The architecture’s generality is its superpower. Any problem that can be cast as a sequence — and almost any problem can — is amenable to transformer processing. This universality, combined with efficiency innovations like mixture-of-experts, suggests that transformers will remain the dominant architecture for years to come.

Whether a fundamentally new architecture will eventually replace the transformer, the way the transformer replaced RNNs, is an open question. State-space models like Mamba offer linear scaling with sequence length, addressing the transformer’s quadratic bottleneck. But so far, no alternative has matched the transformer’s combination of performance, scalability, and generality.

Frequently Asked Questions

Sources & Further Reading

• Attention Is All You Need — Vaswani et al., NeurIPS (2017)
• The Illustrated Transformer — Jay Alammar
• Scaling Laws for Neural Language Models — Kaplan et al., OpenAI (2020)
• An Image Is Worth 16×16 Words: Transformers for Image Recognition (ViT) — Dosovitskiy et al., Google (2020)
• FlashAttention: Fast and Memory-Efficient Exact Attention — Dao et al., Stanford (2022)