How Generative AI Works: From Tokens to Creativity

En bref : Generative AI creates new content — text, images, code, video, music — by learning patterns from massive training datasets and using those patterns to produce novel outputs. The process involves breaking input into tokens, applying attention mechanisms to understand context, and using probabilistic sampling to generate outputs one piece at a time. Different modalities (text, image, video) use different architectures, but all share the same fundamental principle: learning the statistical structure of data well enough to produce new examples that fit the pattern.

The Most Misunderstood Technology of the Decade

Ask someone how generative AI works and you will get one of two answers: “It is just statistics” or “Nobody really knows.” Both are wrong, and the truth is far more interesting.

Generative AI does not copy. It does not retrieve stored answers from a database. It does not search the internet and rephrase what it finds. Instead, it has learned the deep statistical patterns that connect ideas, words, pixels, and sounds — patterns so complex that the outputs feel creative, insightful, and sometimes eerily human.

Understanding how this works — the actual mechanics beneath the magic — is not just academic curiosity. It determines whether you use these tools effectively or waste them on tasks they are fundamentally unsuited for. It explains why AI can write a convincing legal brief but might hallucinate the case citations. It explains why an image generator can create photorealistic portraits of people who do not exist but cannot reliably draw hands with five fingers.

Step 1: Tokenization — Breaking Language Into Pieces

Every generative AI system begins by converting input into a format the model can process. For large language models, that format is tokens.

Tokens are not words — they are word fragments. Modern tokenizers (like byte-pair encoding, or BPE) split text into common subword units. The word “understanding” might become two tokens: “understand” and “ing.” The word “AI” is a single token. An uncommon word like “tokenization” might split into “token,” “iz,” and “ation.”

Why fragments instead of whole words? Efficiency and coverage. A tokenizer with a vocabulary of 50,000 to 100,000 tokens can represent any possible text, including words it has never seen, by combining known fragments. This is how LLMs handle misspellings, neologisms, code, and text in hundreds of languages without needing separate vocabularies for each.

The tokenization step is invisible to users but has practical implications. Models are priced by token count. Context windows are measured in tokens. And because different languages tokenize differently — English is more token-efficient than Arabic or Chinese — the effective context window varies by language.

Step 2: Attention — Understanding Context

Once input is tokenized, the model needs to understand how the tokens relate to each other. This is the job of the attention mechanism, the core innovation of the transformer architecture.

Self-attention works by having each token “look at” every other token in the input and compute a relevance score. When processing the sentence “The programmer fixed the bug in the code that had been causing crashes for weeks,” the attention mechanism connects “crashes” to “bug,” “code,” and “programmer” — even though they are far apart in the sentence.

This happens in parallel across multiple “attention heads,” each learning to focus on different types of relationships. One head might track grammatical dependencies (subject-verb agreement). Another might track semantic relationships (what concepts are related). Another might track positional patterns (what typically follows what).

The multi-head attention mechanism is stacked in layers — modern LLMs have 80 to 120 layers. Each layer refines the model’s understanding of the input, building increasingly abstract representations. The early layers capture syntax and word relationships. The middle layers capture meaning and factual associations. The deep layers capture reasoning patterns and complex inferences.

Step 3: Generation — One Token at a Time

Here is the core insight that surprises most people: generative AI creates text one token at a time, each time predicting the single most appropriate next token given everything that came before.

When you ask an LLM to explain quantum computing, it does not compose the entire response in advance. It predicts the first token (perhaps “Quantum”), then uses that token plus the original prompt to predict the second token (“computing”), then uses everything so far to predict the third, and so on. Each token is generated by running the entire model forward through all its layers.

This auto-regressive process — always predicting the next element based on previous elements — is what makes LLM generation feel fluid and coherent. It is also what creates vulnerability to hallucination: once the model commits to a false claim in token 50, it will generate tokens 51 through 100 that are consistent with that false claim, building an increasingly confident fabrication.

Temperature and Sampling

The model does not predict a single “correct” next token. It computes a probability distribution over its entire vocabulary — perhaps assigning 30% probability to “computing,” 15% to “mechanics,” 10% to “physics,” and fractions of a percent to thousands of other tokens.

The temperature parameter controls how the model samples from this distribution. At temperature 0 (greedy decoding), the model always picks the highest-probability token — producing consistent, predictable, but potentially repetitive output. At temperature 1.0, the model samples proportionally to probabilities — introducing variety and creativity but also increasing the risk of incoherent or irrelevant outputs. Most production systems use temperatures between 0.3 and 0.8.

Other sampling strategies add additional control. Top-k sampling restricts choices to the k most probable tokens. Top-p (nucleus) sampling restricts choices to the smallest set of tokens whose cumulative probability exceeds a threshold p. These techniques prevent the model from making wildly unlikely token choices while preserving diversity.

Understanding temperature explains a common user experience: asking the same question twice and getting different answers. The model is not inconsistent — it is sampling from a probability distribution, and different samples produce different paths through the generation space.

Beyond Text: How Image Generation Works

Text generation is auto-regressive — one token after another. Image generation takes a fundamentally different approach.

Diffusion models (used by DALL-E 3, Midjourney, Stable Diffusion) work by learning to reverse noise. During training, the model is shown clean images that are progressively corrupted with random noise until they become pure static. The model learns to reverse this process — to take a noisy image and predict what the slightly less noisy version looks like.

At generation time, the model starts with pure random noise and iteratively denoises it, guided by the text prompt. Each denoising step moves the image closer to something that matches the description. After 20-50 steps, a coherent image emerges from the noise.

This process explains several quirks of image generation. The iterative nature means you can control the trade-off between quality and speed (more steps = higher quality). The noise-based foundation means outputs are inherently stochastic — the same prompt always produces different images. And the training on complete images (rather than sequential pixels) means the model reasons about global composition, not just local details.

GANs (Generative Adversarial Networks), the previous dominant architecture, used a different approach: two neural networks in competition, one generating images and one trying to distinguish real from generated. GANs produced remarkably realistic images but were notoriously difficult to train and prone to “mode collapse” (generating only a few types of images). Diffusion models largely replaced GANs by 2023 due to their stability and controllability.

Video, Audio, and Multimodal Generation

The same principles extend to other modalities with architectural adaptations.

Video generation (Sora, Veo, Runway) extends diffusion models to the temporal dimension. The model denoises in both space and time, ensuring that each frame is consistent with the frames before and after it. The technical challenge is enormous — a 10-second video at 24 frames per second contains 240 images that must be coherent, temporally consistent, and physically plausible.

Audio generation typically uses transformer architectures similar to text models, but operating on audio tokens — discrete representations of sound learned by audio codecs like EnCodec. The model predicts the next audio token given the previous ones, producing speech, music, or sound effects.

Multimodal models like GPT-4V, Gemini, and Claude can process and generate across multiple modalities — understanding images while generating text, or taking text instructions to produce code. These models use vision-language architectures that align visual and textual representations in a shared embedding space.

The trend is convergence. Early generative AI systems were specialized — a text model, an image model, a code model. Modern systems are increasingly unified, processing any combination of text, image, audio, and video within a single architecture. This mirrors the evolution of AI models from narrow specialists to general-purpose systems.

Code Generation: A Special Case

Code generation deserves separate attention because it reveals something important about how generative AI works.

Code is more constrained than natural language — it must be syntactically valid, logically consistent, and executable. The fact that LLMs can generate working code suggests they learn more than surface patterns; they capture some representation of logic, data structures, and algorithmic thinking.

But code generation also exposes limitations sharply. A model might generate a function that looks correct, passes superficial review, but contains a subtle logic error that only manifests on edge cases. This is the statistical pattern-matching nature of LLMs at work — the code matches the pattern of correct code without being verified through execution.

This is why AI agents that can actually run and test code represent a significant advance. They close the loop between generation and verification, using execution results to refine their outputs — a capability that pure language models lack.

The Emergent Intelligence Question

Perhaps the most fascinating aspect of how generative AI works is what happens at scale. Capabilities that do not exist in smaller models appear spontaneously when models reach certain size thresholds. A model with roughly 10 billion parameters cannot do arithmetic. A model with approximately 100 billion parameters can. A model with roughly 10 billion parameters cannot reason about analogies. A model with approximately 500 billion parameters can.

These “emergent capabilities” were not explicitly programmed. They arise from the statistical patterns in the training data becoming rich enough, at sufficient scale, to support complex behavior. Whether this constitutes genuine understanding or merely very sophisticated pattern-matching is one of the most debated questions in AI research.

What is not debated is the practical impact. Generative AI works well enough to transform how software is written, how research is conducted, how content is created, and how decisions are made. Understanding the mechanism — tokens, attention, sampling, diffusion — helps users work with the technology rather than against it.

Frequently Asked Questions

Sources & Further Reading

• Attention Is All You Need — Vaswani et al., Google Research (2017)
• Denoising Diffusion Probabilistic Models — Ho et al., NeurIPS (2020)
• GPT-4 Technical Report — OpenAI (2023)
• Emergent Abilities of Large Language Models — Wei et al., Google Research (2022)
• The Illustrated Transformer — Jay Alammar