NVIDIA’s Groq Deal: How the Vera Rubin Platform Reshapes AI Inference

The GTC Announcement That Rewrote the AI Hardware Map

On March 16, 2026, Jensen Huang took the GTC stage in San Jose and unveiled a series of announcements that sent shockwaves through every data center operator, cloud provider, and AI startup on the planet. The headliner was the Groq 3 Language Processing Unit — NVIDIA’s first chip to emerge from its $20 billion licensing and talent deal with Groq, announced on Christmas Eve 2025 and representing the largest deal in NVIDIA’s history.

The transaction was structured as a non-exclusive licensing agreement rather than a traditional acquisition. NVIDIA licensed Groq’s inference technology IP and hired approximately 90 percent of Groq’s employees, including founder Jonathan Ross and President Sunny Madra. Groq continues to operate as an independent company under new CEO Simon Edwards, though its GroqCloud inference service was not part of the transaction.

The move was vintage Jensen: audacious, vertically integrating, and designed to close the one gap in NVIDIA’s armor that competitors had been quietly exploiting. NVIDIA has dominated AI training for a decade, but inference — running trained models to produce answers, generate images, and power AI agents — represents a different engineering challenge where Groq’s deterministic, SRAM-first architecture had been outperforming GPU-based solutions on latency and power efficiency.

Why Inference Is the New Battleground

Training a frontier model is extraordinarily expensive — hundreds of millions of dollars for a single run. But training happens once. Inference happens billions of times per day. Every ChatGPT query, every AI-generated search result, every autonomous agent making a decision — all of it is inference. Industry estimates place inference at 60-70% of total AI compute spending, and that share is accelerating as the world shifts from building models to deploying them at scale.

The fundamental problem is that GPUs, while excellent at training through massively parallel matrix multiplications, are architecturally over-provisioned for many inference workloads. A single user query does not need 80GB of HBM3e bandwidth. It needs fast, deterministic token generation with predictable latency. This mismatch is why inference-specialized chips from Groq, Cerebras, and others had been gaining traction with companies frustrated by GPU inference costs.

Groq’s core insight was to eliminate the memory bottleneck. Traditional AI accelerators shuttle data between compute units and external DRAM or HBM, creating latency and consuming enormous power. Groq’s LPU architecture places massive amounts of SRAM directly on-die, keeping entire model layers in ultra-fast on-chip memory. The result: deterministic execution, predictable latency, and radically better energy efficiency for inference workloads.

The LP30: Groq’s Crown Jewel Gets NVIDIA Resources

The centerpiece of the Vera Rubin platform’s inference capability is the LP30, the next-generation LPU chip that Groq had been developing and that now benefits from NVIDIA’s manufacturing relationships and virtually unlimited R&D budget. The LP30 is manufactured by Samsung on the 4nm SF4X process, with NVIDIA planning to ship in Q3 2026.

The LP30’s specifications represent a generational leap:

• 512MB of SRAM per die — Half a gigabyte of the fastest memory available, directly on the chip. For context, NVIDIA’s Blackwell B200 has roughly 64MB of L2 cache. The LP30 has eight times that in raw on-chip memory, eliminating the need to access external memory for most inference operations.

• 150TB/s of on-chip bandwidth — The internal data movement rate within the LP30 is nearly 7 times greater than the Rubin GPU’s HBM4 bandwidth of 22TB/s per GPU. Data is always where the compute needs it.

• 1.23 FP8 PFLOPS of compute — Per chip, with 98 billion transistors driving inference performance.

• Full LPX rack: 256 LPUs, 128GB aggregate SRAM — A complete Vera Rubin LPX rack packs 256 LP30 chips delivering 40 PB/s of aggregate bandwidth. This is sufficient to hold large model layers entirely in on-chip memory with minimal inter-chip communication overhead.

• 35x throughput per megawatt versus Blackwell NVL72 — This is the number that will reshape data center economics. At a time when AI data centers are constrained by power availability, a 35x improvement in inference throughput per unit of electricity is transformational, not incremental.

Vera Rubin: The Heterogeneous Compute Vision

The Vera Rubin platform is not simply GPUs plus LPUs in the same rack. It represents NVIDIA’s most comprehensive system to date: seven chips, five rack-scale systems, and one AI supercomputer designed for the full AI lifecycle.

The seven chips include the Vera CPU, Rubin GPU (336 billion transistors, HBM4 with 22TB/s bandwidth per GPU), NVLink 6, ConnectX-9 SuperNIC, BlueField-4 DPU, Spectrum-6 switch, and the Groq 3 LPU. The Rubin GPU alone delivers 50 PFLOPS of NVFP4 inference — a 5x improvement over Blackwell GB200 — while the NVL72 rack is rated at 3.6 exaFLOPS.

The heterogeneous decode architecture is orchestrated by NVIDIA Dynamo, which classifies incoming requests and routes them to optimal hardware. Prefill and attention computations go to Rubin GPUs. Latency-sensitive decode operations — the token-by-token generation that powers chatbots and agents — route to the LP30 LPUs. Developers write code using the existing CUDA ecosystem; the runtime handles routing transparently.

The economic implications are significant. The NVL72 rack delivers 10x higher inference throughput per watt at one-tenth the cost per token versus the prior Blackwell platform. Products will be available from partners in the second half of 2026.

1,500 Tokens Per Second: The Agent Speed Threshold

One number from the keynote deserves special attention: 1,500 tokens per second for agent workloads. NVIDIA VP Ian Buck stated that the combination of Rubin GPUs and Groq racks “moves us from a world where 100 tokens per second is a reasonable throughput to one of 1,500 TPS or more for AI agent intercommunication.”

This target is not about chatbot speed — 50 tokens per second already feels instantaneous to a human reader. The 1,500 tok/s target is designed for AI agents consuming other AI agents’ output. In agentic workflows where an orchestrator dispatches tasks to specialist agents, collects responses, reasons over them, and dispatches further tasks, the speed of each individual inference call compounds across the entire chain.

At 100 tokens per second, a multi-step agent chain handling a customer inquiry might take 15-25 seconds. At 1,500 tokens per second, the same chain completes in under 3 seconds. For time-sensitive applications — financial trading, real-time fraud detection, autonomous systems — that difference determines viability.

Jensen explicitly positioned the Vera Rubin LP30 as “the inference engine for the agentic era,” arguing that AI agent proliferation will drive inference demand 10-100x beyond current levels.

The $1 Trillion Order Pipeline

In perhaps the most audacious claim of the keynote, Jensen Huang revealed that NVIDIA sees $1 trillion in purchase orders for Blackwell and Vera Rubin through 2027. This effectively doubles a prior outlook of roughly $500 billion in demand through 2026.

These are commitments and intent — multi-year purchase agreements, staged deliveries, and capacity reservations from hyperscalers, AI labs, and sovereign buyers — rather than booked revenue. For context, NVIDIA reported $215.9 billion in fiscal year 2026 revenue, with data center revenue comprising over 91% of total sales.

The pipeline rests on several drivers: inference revenue overtaking training as enterprises scale AI deployments, over 40 countries pursuing sovereign AI infrastructure with NVIDIA as default vendor, and the expected surge in agentic AI compute demand.

Space-1: AI Inference Goes Orbital

In the most visually dramatic portion of the keynote, Jensen unveiled Space-1, a Vera Rubin module engineered for orbital data centers. The premise: as AI inference demand exceeds terrestrial power and cooling constraints, deploying compute in orbit offers near-unlimited solar power and natural vacuum cooling.

NVIDIA launched partnerships with Aetherflux, Axiom Space, Kepler Communications, Planet Labs, Sophia Space, and Starcloud to develop space-based AI infrastructure. The Space-1 module delivers up to 25x more AI compute for space-based inferencing compared to the H100 GPU.

Jensen noted a key engineering challenge: “In space, there’s no convection, there’s just radiation.” Cooling remains an active R&D problem. The project is in early engineering rather than active construction, but the symbolic message was clear — NVIDIA’s ambition for AI compute has no terrestrial ceiling.

What This Means for Competitors

The Groq deal eliminates NVIDIA’s most credible inference-specialized competitor while simultaneously strengthening its inference offering.

AMD loses its best argument. AMD had positioned its MI300X and upcoming MI400 GPUs as inference-competitive alternatives. With NVIDIA now offering purpose-built inference silicon alongside GPUs, AMD must compete on two fronts simultaneously.

Cerebras faces intensified pressure. Its wafer-scale engine had gained traction with sovereign AI projects and research institutions, but NVIDIA’s LP30 backed by NVIDIA’s sales force and CUDA ecosystem narrows Cerebras’s differentiation.

Cloud providers must recalculate. AWS, Google Cloud, and Microsoft Azure have been developing custom inference silicon (Inferentia, TPU, Maia). The Vera Rubin platform may reduce the urgency of those custom chip programs — or accelerate them as cloud providers seek to avoid total NVIDIA dependence.

The Integration Risk

Deals of this magnitude carry execution risk. NVIDIA must retain Groq’s engineering talent — roughly 90% of employees joined, but the LPU architecture lives in the expertise of a few hundred engineers whose startup culture differs markedly from NVIDIA’s 30,000-person organization. The NVIDIA Dynamo orchestration layer must deliver seamless heterogeneous routing; if developers need manual GPU-vs-LPU decisions, adoption will lag. And a $20 billion deal by the world’s most valuable semiconductor company will attract antitrust scrutiny across the US, EU, and Asian markets.

The Groq deal was structured to keep the “fiction of competition alive,” as one analyst put it — the non-exclusive licensing agreement technically allows Groq to license its IP elsewhere. Whether that amounts to meaningful competition remains to be seen.

Sources & Further Reading

• NVIDIA Vera Rubin Opens Agentic AI Frontier — NVIDIA Newsroom
• Nvidia GTC 2026: CEO Jensen Huang sees $1 trillion in orders — CNBC
• Inside NVIDIA Groq 3 LPX: Low-Latency Inference Accelerator — NVIDIA Developer Blog
• How Nvidia’s $20 billion Groq 3 LPU deal reshapes the Vera Rubin Platform — Tom’s Hardware
• Nvidia buying AI chip startup Groq’s assets for about $20 billion — CNBC
• NVIDIA Launches Space Computing, Rocketing AI Into Orbit — NVIDIA Newsroom
• NVIDIA FY2026 Financial Results — NVIDIA Newsroom
• A closer look at Nvidia’s Groq-powered LPX rack systems — The Register