AI Data Centers: How They Work and Why They Matter

En bref : The global AI data center market is projected to exceed $150 billion by 2027, but these facilities bear almost no resemblance to the cloud data centers built over the past two decades. AI workloads demand fundamentally different architecture — from networking fabrics that move petabytes per second to liquid cooling systems that extract 100 kilowatts per rack. This article explains what makes AI data centers a distinct engineering category and why building them has become the defining infrastructure challenge of the decade.

In January 2025, Microsoft announced plans to spend $80 billion on AI-capable data centers in a single fiscal year. Two months later, Amazon committed $100 billion. Google, Meta, and Oracle each pledged tens of billions more. By some estimates, the global AI infrastructure race will consume over $300 billion in capital expenditure by the end of 2026 alone.

But this is not simply more of the same. The facilities being built to train and serve AI models are architecturally alien compared to the data centers that powered the first two decades of cloud computing. The racks are denser, the power draws are staggering, the cooling systems are borrowed from industrial manufacturing, and the networking fabrics operate at scales that would have seemed absurd five years ago.

Understanding AI data centers means understanding why traditional approaches broke — and what replaced them.

Why AI Data Centers Are Different

A traditional cloud data center is built around CPUs serving web requests. Each server handles relatively independent tasks: a database query here, a container workload there. The servers communicate, but not constantly. Power consumption per rack runs between 5 and 15 kilowatts, and standard air conditioning keeps temperatures manageable.

AI data centers invert nearly every assumption. A single NVIDIA DGX B200 system packs eight GPUs into one node, drawing over 14 kilowatts alone. A full training cluster might contain thousands of these nodes, all communicating simultaneously during a single training run. Where a traditional data center might draw 20 megawatts total, a purpose-built AI facility routinely requires 100 to 300 megawatts — enough to power a small city.

The difference is not incremental. A rack of NVIDIA GPUs running AI training can draw 40 to 132 kilowatts — with next-generation Blackwell Ultra systems pushing toward 250 kilowatts per rack — compared to 7 kilowatts for a typical cloud server rack. That density gap drives every other design decision: networking, cooling, power distribution, and physical layout.

The Networking Challenge

In traditional cloud computing, east-west traffic between servers matters, but individual requests are relatively small and latency-tolerant. AI training shatters this model. During distributed training, every GPU must synchronize gradients with every other GPU after each forward and backward pass. A cluster of 10,000 GPUs might exchange hundreds of terabytes of data in a single training step.

This is why InfiniBand — a networking technology originally built for high-performance computing — has become the backbone of AI data centers. NVIDIA’s Quantum-2 InfiniBand switches deliver 400 gigabits per second per port with latency under two microseconds, roughly ten times faster than conventional Ethernet for the all-reduce operations that dominate AI training traffic.

Within individual servers, NVIDIA’s NVLink interconnect allows GPUs to communicate at speeds that bypass the PCIe bus entirely. The fourth-generation NVLink in Hopper systems delivers 900 gigabytes per second per GPU, while the fifth-generation NVLink in Blackwell-based systems doubles that to 1.8 terabytes per second. The combination of NVLink inside the node and InfiniBand between nodes creates a two-tier fabric optimized for the specific communication patterns of distributed training.

Ethernet is fighting back. Ultra Ethernet Consortium members — including Broadcom, Cisco, and Microsoft — are developing RoCE v2 (RDMA over Converged Ethernet) variants that approach InfiniBand’s performance at lower cost. For inference workloads, which have less demanding synchronization requirements, Ethernet-based fabrics are already competitive. The AI cloud wars are partly a networking architecture battle fought at the switch level.

The dominant topology for AI clusters is the fat-tree network, which provides multiple redundant paths between any two nodes. A well-designed fat-tree ensures that no single switch failure can partition the cluster, and that bandwidth scales linearly as nodes are added. Building these topologies at scale — connecting 100,000 GPUs with full bisection bandwidth — requires thousands of switches and tens of thousands of optical transceivers, each one a potential point of failure.

Cooling the AI Factory

When every rack draws 40 to 130 kilowatts or more, air cooling simply cannot remove heat fast enough. The physics are unforgiving: air has roughly one-twenty-fifth the thermal conductivity of water and one-thousandth that of direct liquid contact. This is why liquid cooling has become the defining technology of the AI data center era.

Direct-to-chip liquid cooling — where cold plates sit directly on GPU dies, circulating water or specialized coolant — is now standard for high-density AI clusters. NVIDIA’s reference designs for Blackwell-based systems assume liquid cooling. Rear-door heat exchangers, which attach water-cooled radiators to the back of each rack, offer a retrofit path for existing facilities but cannot handle the highest densities.

Immersion cooling, where entire servers are submerged in dielectric fluid, promises even higher thermal extraction rates and is being deployed by companies like GRC and LiquidCool Solutions. However, immersion remains a small fraction of total deployments, partly because servicing submerged hardware is more complex and partly because the supply chain for dielectric fluids is still maturing.

The efficiency metric that matters is Power Usage Effectiveness (PUE) — total facility power divided by IT equipment power. A PUE of 1.0 would mean zero overhead; average air-cooled data centers worldwide typically achieve 1.5 to 1.8, with industry leaders reaching 1.2 to 1.3. Modern liquid-cooled AI facilities are pushing toward 1.08 to 1.15 — Google’s fleet-wide PUE averaged 1.10 in 2025, meaning only 10 to 15 percent of total power goes to cooling, lighting, and other overhead.

Power and Location

The single most constrained resource in AI data center construction is not silicon, not talent, and not capital. It is electrical power.

A 100-megawatt AI data center — a mid-sized facility by current standards — consumes as much electricity as roughly 80,000 American homes. Microsoft’s planned campus in Mount Pleasant, Wisconsin will eventually require around 2 gigawatts across multiple facilities. Meta’s Hyperion facility in Louisiana is designed for over 2 gigawatts, with potential to scale to 5 gigawatts, making it larger than many power plants.

This power hunger is reshaping geography. AI data centers cluster around cheap, abundant electricity: hydroelectric power in Quebec and Scandinavia, natural gas in Texas, and increasingly, dedicated nuclear power agreements like Amazon’s deal with Talen Energy at the Susquehanna plant in Pennsylvania. Some hyperscalers are evaluating small modular reactors (SMRs) as dedicated, on-site power sources for facilities planned in the 2030s.

Grid interconnection has become a bottleneck. In Northern Virginia — the world’s largest data center market, hosting the highest concentration of data center capacity globally — utility Dominion Energy has warned that new large data center connections may face wait times of four to seven years. Similar constraints are emerging in Dublin, Singapore, and Amsterdam, where governments have imposed moratoriums or restrictions on new data center construction.

The Hyperscale Build-Out

The companies building these facilities operate at a scale that dwarfs all prior infrastructure investment in computing history.

Microsoft plans to operate over 300 data centers by end of 2026 and is reportedly building at a pace of one new facility every three days. Google operates 40 cloud regions across six continents and is adding AI-optimized capacity to nearly all of them. Amazon Web Services, the largest cloud infrastructure provider, is expanding capacity across 36 geographic regions. Meta, which operates some of the world’s largest single-site facilities, is building a 2-gigawatt-plus campus called Hyperion in Louisiana — backed by a $27 billion joint venture with Blue Owl Capital — that would be the largest data center ever constructed, with potential to scale to 5 gigawatts.

Behind the hyperscalers, a second tier of GPU cloud providers — CoreWeave, Lambda, Crusoe Energy — is building smaller but highly specialized AI-focused facilities, often co-located with renewable or stranded energy sources.

What Comes Next

Three trends are reshaping the next generation of AI data centers. First, modular and prefabricated construction is accelerating deployment timelines. Microsoft’s “datacenter in a box” approach uses factory-built modules that arrive on-site ready to connect, cutting construction time from 18 months to as little as six.

Second, inference scaling is creating demand for a different kind of facility. Training clusters optimize for maximum GPU-to-GPU bandwidth. Inference clusters optimize for latency and throughput per dollar, often using different hardware (NVIDIA’s L40S, Google’s TPU v5e, or custom ASICs) and less exotic networking.

Third, edge AI data centers — smaller facilities placed closer to users — are emerging to serve latency-sensitive applications like autonomous vehicles, real-time translation, and industrial robotics. These facilities trade scale for proximity, typically running 1 to 10 megawatts rather than hundreds.

The AI revolution runs on silicon, but it lives in these buildings. Understanding how AI data centers are designed, powered, cooled, and connected is essential for anyone making infrastructure decisions in the decade ahead.

Frequently Asked Questions

Sources & Further Reading

• NVIDIA — DGX B200 Datasheet
• NVIDIA — NVLink & NVSwitch: Fastest HPC Data Center Platform
• IEA — Data Centres and Data Transmission Networks Energy Use
• Google — Data Center Efficiency: Power Usage Effectiveness
• TechCrunch — Microsoft to Spend $80 Billion in FY25 on Data Centers for AI
• Meta — Hyperion AI Data Center in Louisiana
• TechCrunch — Amazon Doubles Down on AI with a Massive $100B Spending Plan for 2025