The Synthetic Data Revolution: Training AI on AI-Generated Data

The Internet Has Been Read. Now What?

The scaling laws that powered the large language model revolution rested on a simple assumption: more data, more compute, better models. For a decade, that assumption held. GPT-3 was trained on 300 billion tokens. GPT-4 consumed an estimated 13 trillion tokens (including multi-epoch passes over roughly 5-6 trillion unique tokens). Each generation vacuumed up more of the internet — books, websites, code repositories, academic papers, Reddit threads — and performance improved predictably.

That era is reaching its limits. Epoch AI, a research organization tracking AI inputs, initially estimated that high-quality text data on the internet totaled roughly 9 trillion tokens. Their revised 2024 analysis substantially increased that figure: accounting for multi-epoch training and careful filtering, the effective stock of usable human-generated public text sits at approximately 300 trillion tokens, with a wide confidence interval of 100 to 1,000 trillion. That sounds like a lot — but frontier labs are consuming data at accelerating rates. Epoch AI now estimates that models will fully utilize this stock between 2026 and 2032, depending on training intensity. The remaining untapped sources — private corporate data, paywalled content, non-English text — are either legally encumbered, expensive to license, or insufficient in quality.

The industry’s response has been a dramatic pivot to synthetic data: using AI models themselves to generate the training examples for the next generation of AI models. This is not a niche technique. Over 98% of the alignment data for Nvidia’s Nemotron-4 340B was synthetically generated. Meta’s Llama 3.1 and 3.3 used over 25 million synthetic examples for instruction tuning. Anthropic’s constitutional AI methods involve models generating and evaluating their own training signals. OpenAI’s o1 and o3 reasoning models rely on reinforcement learning and self-play, with o1 reportedly used to generate training data for future models. Gartner predicts that by 2028, 80% of AI training data will be synthetic, up from single digits just a few years ago. The question is no longer whether synthetic data works, but whether it can sustain the scaling trajectory — and what breaks if it cannot.

---

The Techniques: Distillation, Self-Play, and Simulated Worlds

Synthetic data generation encompasses several distinct techniques, each with different strengths and failure modes. Knowledge distillation is the most straightforward: a large, capable model generates training examples that a smaller model learns from. OpenAI’s text-davinci models were used to generate instruction-following examples that trained subsequent models. The “teacher-student” paradigm is now standard practice — virtually every open-source model fine-tuned in 2025 used some form of distilled synthetic data. Meta has gone further by open-sourcing a Synthetic Data Kit, a CLI toolkit for generating reasoning traces and QA pairs to fine-tune Llama models, signaling that synthetic data tooling is becoming commoditized.

Self-play and self-improvement represent a more ambitious approach. Inspired by DeepMind’s AlphaGo, which learned superhuman Go by playing against itself, self-play methods have models generate solutions, evaluate them, and iteratively refine. Google DeepMind’s AlphaProof and AlphaGeometry 2 used synthetic theorem generation to train mathematical reasoning systems that achieved silver-medal performance at the 2024 International Mathematical Olympiad — solving four of six problems and earning 28 points, with AlphaGeometry 2 cracking the geometry problem in just 19 seconds. In language models, techniques like Reinforcement Learning from AI Feedback (RLAIF) have models generate and rank their own outputs, creating a self-improving loop. OpenAI’s o1 and o3 models take this further, using large-scale reinforcement learning to train reasoning capabilities that go well beyond what supervised fine-tuning alone can achieve.

Simulated environments provide synthetic data for embodied AI and robotics. Nvidia’s Omniverse platform generates photorealistic synthetic images and physics simulations for training autonomous vehicles, robotic manipulation, and industrial inspection systems. At GTC 2025, Nvidia announced its Cosmos world foundation models and Isaac GR00T N1, the first open foundation model for humanoid robots. The numbers are striking: Nvidia generated 780,000 synthetic trajectories — equivalent to 6,500 hours of human demonstration data — in just 11 hours, and combining synthetic with real data improved robot performance by 40%. Waymo has pushed simulation even further with its February 2026 Waymo World Model, built on Google DeepMind’s Genie 3, which generates hyper-realistic multi-sensor driving data including both camera and lidar outputs. The system can simulate exceedingly rare events — from tornadoes to unexpected obstacles — that would take decades or be impossible to encounter in real-world driving. Mostly AI, Hazy, and Tonic.ai serve the enterprise market with synthetic versions of sensitive tabular datasets, preserving statistical properties while eliminating privacy concerns — enabling healthcare and financial institutions to share “data” without sharing data.

---

The Risks: Model Collapse and Amplified Bias

The most discussed risk is model collapse, a phenomenon where models trained on synthetic data generated by previous models progressively lose diversity and accuracy. A landmark 2024 paper in Nature by Shumailov et al. (first circulated as a preprint in 2023) demonstrated that iterative training on model-generated data causes the distribution of outputs to narrow over successive generations, eventually converging on a degenerate distribution that loses the tails — the rare but important examples that make models robust. The paper, published as “AI models collapse when trained on recursively generated data,” showed the effect occurs across model architectures including LLMs, variational autoencoders, and Gaussian mixture models.

The intuition is straightforward: a model generating training data will tend to produce “average” examples that reflect its own learned distribution. Rare phenomena, minority perspectives, unusual edge cases, and low-frequency patterns get progressively underrepresented. After several generations of model-on-model training, the resulting system may perform well on common queries but fail catastrophically on unusual ones — precisely the cases where AI reliability matters most. Follow-up research presented at ICLR 2025 confirmed that even small fractions of synthetic data (as little as one in a thousand examples) can trigger collapse if real data is not continuously mixed in.

Bias amplification compounds the problem. If a model has absorbed societal biases from its original training data, synthetic data generated by that model will not merely reproduce those biases but potentially amplify them, since the generation process is unconstrained by the natural diversity of human experience. A medical AI trained on synthetic patient data generated by a biased model could systematically underrepresent symptoms as they manifest in underrepresented populations.

The emerging consensus among researchers is that synthetic data is powerful but must be used in combination with real data, with careful curation, and with explicit diversity-preserving techniques. Several promising solutions have emerged in 2025-2026: data accumulation strategies (mixing real and synthetic data across training generations rather than replacing one with the other), synthetic data verification using external verifiers to rank and filter generated examples, active curation to fill distribution gaps, and watermarking for provenance tracking to prevent unintentional contamination. Scale AI has built a business around human-verified synthetic data, combining AI generation with human quality assurance. Anthropic’s constitutional AI approach uses explicit principles to constrain synthetic data generation, reducing (though not eliminating) the risk of distributional collapse.

---

The Business Models and the Road Ahead

Synthetic data has spawned a rapidly consolidating industry. Scale AI, now valued at approximately $29 billion following Meta’s acquisition of a 49% stake for $14.8 billion in June 2025, provides data labeling and increasingly synthetic data generation services to frontier labs and enterprises — making it one of the most valuable AI infrastructure companies in the world. Nvidia acquired synthetic data startup Gretel AI in March 2025 for over $320 million, folding its privacy-preserving data generation technology and 80 employees into Nvidia’s AI services portfolio. Mostly AI, Hazy, and Tonic.ai (which acquired Fabricate in April 2025 to expand into from-scratch relational data generation) continue serving the enterprise market with synthetic versions of sensitive datasets. These acquisitions signal that synthetic data is no longer a niche capability — it is core infrastructure that the largest AI companies are racing to own.

The economic logic is compelling. Real-world data collection is expensive, slow, legally fraught, and often privacy-constrained. Synthetic data can be generated at marginal cost, customized for specific tasks, and created without the consent and licensing issues that have embroiled AI companies in lawsuits from publishers, artists, and content creators. The copyright landscape continued to shift through 2025 and into 2026: the New York Times lawsuit against OpenAI is progressing through discovery, with a federal judge in January 2026 ordering OpenAI to produce 20 million ChatGPT logs, and summary judgment expected by April 2026. In the UK, the Getty Images case against Stability AI reached judgment in November 2025, with the High Court rejecting the central copyright claim — finding that AI model weights are not “copies” of training images in the legal sense, though the ruling’s reach was limited by the fact that training occurred outside the UK. The growing “robots.txt” arms race between web publishers and AI crawlers continues to increase the relative attractiveness of synthetic alternatives.

Looking forward, the most consequential question is whether synthetic data can support continued scaling. If frontier model performance requires exponentially more data, and natural data is approaching its limits, then synthetic data must either fill the gap or the scaling paradigm must change. Results so far are promising but mixed. Models trained with substantial synthetic components perform comparably on benchmarks, but benchmark performance may not capture the distributional richness that real-world deployment demands. Gartner’s prediction that 80% of AI training data will be synthetic by 2028 suggests the industry is betting heavily on this approach. The next 12-18 months will likely determine whether synthetic data — carefully curated, verified, and anchored in real-world data — is a sustainable foundation for the next generation of AI, or a ceiling that forces fundamentally different approaches to AI improvement.

---

---

Sources & Further Reading

• Will We Run Out of Data? Limits of LLM Scaling — Epoch AI
• Can AI Scaling Continue Through 2030? — Epoch AI
• AI Models Collapse When Trained on Recursively Generated Data — Nature (Shumailov et al., 2024)
• Nemotron-4 340B Technical Report — Nvidia
• Nvidia Acquires Synthetic Data Startup Gretel — TechCrunch
• Isaac GR00T N1: Open Humanoid Robot Foundation Model — Nvidia
• AlphaProof and AlphaGeometry: AI for Mathematics — Google DeepMind
• The Waymo World Model: A New Frontier for Autonomous Driving Simulation — Waymo
• Introducing Llama 3.1 and Synthetic Data Generation — Meta AI
• Constitutional AI: Harmlessness from AI Feedback — Anthropic
• Getty Images v. Stability AI: UK High Court Decision — Mayer Brown
• Gartner Identifies Top Trends in Data Science and Machine Learning — Gartner