The Bare Metal Comeback: Why AI Workloads Are Leaving Virtual Machines

For a decade, the orthodoxy in enterprise IT was clear: virtualize everything. Abstract the hardware, share the resources, bill by the minute. The hypervisor was the unsung hero of the cloud era, letting AWS, Azure, and Google carve one physical machine into dozens of neat, portable instances. It was efficient, flexible, and enormously profitable for hyperscalers. Then AI arrived — and the orthodoxy shattered.

Training a large language model, running a diffusion pipeline, or serving real-time inference at scale has exposed a fundamental truth that virtualization enthusiasts spent a decade papering over: hardware abstraction has costs, and for GPU-intensive workloads those costs are not theoretical. They show up in benchmark scores, training times, and cloud bills. The result is one of the more counterintuitive trends in 2026 infrastructure: bare metal is back, and it is being driven not by old-school sysadmins who distrust hypervisors, but by AI engineers who have run the numbers.

What Virtualization Actually Costs You on GPUs

When you rent a GPU virtual machine from a major cloud provider, you are not getting direct access to the GPU. A hypervisor layer sits between your workload and the silicon. For CPU compute, this overhead is largely invisible — virtualization has become extraordinarily efficient at abstracting processor cycles. For GPUs, the picture is more complicated.

GPU virtualization technologies such as NVIDIA’s vGPU and MIG (Multi-Instance GPU) partition the physical card so multiple tenants can share it. This works well for inference workloads with predictable, moderate loads. But for training — where you need consistent, sustained throughput across thousands of CUDA cores, fast NVLink interconnects between GPUs, and deterministic memory bandwidth — any virtualization layer introduces jitter, latency variability, and throughput reduction. Independent benchmarks have repeatedly shown 10–25% performance degradation for large-scale training workloads running on virtualized GPU instances compared to equivalent bare metal configurations.

Beyond raw performance, there is the interconnect problem. Modern AI training relies on high-speed GPU-to-GPU communication via NVLink (within a node) and InfiniBand or RoCE (across nodes). These interconnects are latency-sensitive at the microsecond level. Virtualization layers and shared network fabrics introduce unpredictable latency spikes that can stall gradient synchronization across a training cluster, forcing idle GPU cycles across hundreds of cards simultaneously — a ruinously expensive inefficiency at scale.

The Rise of Specialized Bare Metal GPU Clouds

The performance gap created a market, and the market created a new class of cloud provider. CoreWeave, founded in 2017 as a cryptocurrency miner and pivoted to GPU cloud in 2019, became one of the most prominent examples. By 2024 the company had raised over $7.5 billion in funding and secured contracts with Microsoft, Cohere, and IBM to provide bare metal NVIDIA H100 and H200 clusters at scale. Its pitch is simple: dedicated hardware, no hypervisor, full NVLink and InfiniBand performance, billed by the GPU-hour.

Lambda Labs took a similar approach, building a GPU cloud aimed specifically at AI researchers and ML engineers who want raw performance without the overhead of AWS’s general-purpose abstraction layers. By early 2025, Lambda was operating clusters of NVIDIA H100 SXM5 nodes connected via 3.2 Tbps InfiniBand fabric — configurations that would be impractical to offer in a virtualized multi-tenant environment.

In Europe, Hetzner and OVHcloud have expanded bare metal GPU offerings to serve mid-market AI startups and research institutions priced out of hyperscaler rates. OVHcloud’s bare metal AI lineup, built around NVIDIA A100 and H100 cards, became particularly popular with French and German research labs seeking GDPR-compliant infrastructure with full hardware isolation. Equinix Metal (now rebranded and integrated into Equinix’s broader platform) offers bare metal as a connectivity play — collocating dedicated compute next to its global interconnection fabric, letting companies run AI workloads on owned hardware while maintaining cloud-speed network access.

Hyperscalers Fight Back — Partially

AWS, Azure, and Google have not ignored the bare metal signal. All three now offer dedicated instance types that provide near-bare-metal performance by disabling most hypervisor overhead. AWS’s “bare metal” EC2 instances (the `.metal` suffix types) give customers direct hardware access for specific use cases. Google’s A3 instances, built on NVIDIA H100 GPUs connected via Google’s own NVLink-equivalent fabric, are positioned explicitly for large-scale AI training.

But hyperscalers face a structural tension. Their entire pricing and resource-utilization model is built on multi-tenancy and abstraction. Offering true bare metal at scale undermines the efficiency that makes their margins work. As a result, hyperscaler bare metal offerings tend to be more expensive per GPU-hour than specialist providers, carry longer minimum commitments, and provide less scheduling flexibility — you cannot easily burst a 512-GPU bare metal job for 6 hours on AWS the way you might scale a CPU workload.

The hyperscalers’ answer has been to invest in custom silicon that sidesteps the GPU virtualization problem entirely. AWS Trainium and Inferentia, Google’s TPUs, and Microsoft’s Maia chips are all designed to be run as dedicated accelerators, not shared resources. Training on TPU pods or Trainium clusters gives performance comparable to bare metal NVIDIA configurations without the same virtualization penalties — though it requires porting workloads away from CUDA, which remains the dominant programming model.

The Economics: When Does Bare Metal Win?

Not every AI workload belongs on bare metal. The calculus depends on utilization rate, workload duration, and sensitivity to performance variability.

For inference serving — where a model is already trained and you need to respond to API calls — shared GPU VMs or even CPU inference (for smaller models) often make economic sense. Demand is variable, over-provisioning a bare metal cluster for inference means paying for idle GPU cycles, and the latency overhead of a hypervisor layer is negligible at the API-call level.

The case for bare metal becomes compelling when: training runs last days or weeks; when you are running distributed training across multiple nodes where interconnect performance is critical; when you are doing hyperparameter sweeps or continuous training pipelines that run at consistently high GPU utilization; or when regulatory or data-isolation requirements mandate single-tenant hardware.

A rough rule of thumb circulating in infrastructure circles: if your GPU utilization will average above 70% for sustained periods, bare metal nearly always wins on total cost of ownership. Below that threshold, the flexibility and elasticity of shared VMs — especially for burst workloads — often justify the performance trade-off.

What This Means for Infrastructure Strategy in 2026

The bare metal comeback is reshaping how AI teams think about their compute stack. The era of defaulting to whatever GPU instance AWS lists first is ending. Infrastructure engineers at AI-native companies are increasingly doing bespoke procurement — negotiating long-term bare metal reservations with specialist providers like CoreWeave or Lambda, using hyperscaler VMs for development and experimentation, and reserving owned or collocated hardware for production training runs.

The result is a more heterogeneous cloud landscape. Workloads are routed based on their specific needs: cheap spot VMs for experimentation, bare metal clusters for training, serverless inference APIs for low-volume serving. Multi-cloud and hybrid strategies, once a solution in search of a problem, now have genuine technical justification in the AI infrastructure domain.

For the hyperscalers, the challenge is existential in a narrow but significant sense: the highest-value, highest-margin AI customers are increasingly the ones most likely to route their largest compute spend away from general-purpose cloud and toward specialized bare metal or custom silicon. The companies that built the cloud era on the premise that abstraction was always better are now learning, at enormous cost, that physics sometimes disagrees.

Sources & Further Reading

• CoreWeave Raises $7.5B to Build Out AI Infrastructure — TechCrunch
• Bare Metal vs. Virtual Machines for AI Workloads — The New Stack
• NVIDIA H100 NVLink and InfiniBand Architecture — NVIDIA Developer Blog
• OVHcloud Expands Bare Metal AI GPU Lineup for European Research — OVHcloud Blog
• Lambda Labs GPU Cloud Benchmarks: H100 SXM5 Cluster Performance — Lambda Labs Blog
• Google A3 Instances and the Case for Hyperscaler GPU Optimization — InfoQ