2026-07-22 · 5 min · hardware · gpu · inference · systems · explainer
An agentic workload isn't a single prompt and response — it's sustained inference across many reasoning steps that plan, call tools, verify, and revise over long contexts. That execution pattern stresses a GPU differently than a chatbot does: it wants low per-step latency, high decode throughput, efficient long-context attention, large KV-cache capacity, and the ability to spread a model across tightly coupled GPUs. NVIDIA's Rubin GPU is a bet that the right response is to co-design for exactly that pattern — and the headline claim is up to 10× more agentic throughput per unit of energy than Blackwell.
That "10×" is a vendor number on an internal 2-trillion-parameter MoE workload, so read it as a design target rather than an independent benchmark. What's more interesting than the single figure is how it's assembled — because it isn't one trick, it's a checklist of agentic bottlenecks each with its own answer.
The chip
Physically, Rubin is two reticle-limited compute dies fused into one package over a high-speed inter-die link (NV-HBI): 336 billion transistors, 224 SMs, 896 Tensor Cores, a third-generation Transformer Engine that flexes precision across formats for up to 50 petaflops of NVFP4, up to 288 GB of HBM4 at 22 TB/s, and NVLink 6 at 3,600 GB/s of scale-up bandwidth. Those are the raw numbers; the architecture is about turning them into sustained utilization.
One checklist, many bottlenecks
Here's the spine of the whole design. Each agentic-inference bottleneck gets a specific Rubin feature — click through them:
Intermediate attention scores are compressed to a structured 2:4 sparse form (nonzeros + metadata), so softmax and the second attention GEMM do less work — and exponential throughput rises to 4× BF16 / 2× FP32 vs Blackwell, so softmax keeps pace.
Two are worth dwelling on. Long-context attention is where agentic runs spend their time, and Rubin attacks it from two sides at once: it compresses the intermediate attention scores into a structured 2:4 sparse form so softmax and the second attention GEMM operate on fewer values, and it raises exponential throughput so softmax — which becomes the bottleneck once the matrix math speeds up — keeps pace.
The other is MoE decode: as expert counts climb, just locating and moving expert weights becomes the cost. Blackwell tracks one memory descriptor per expert; Rubin keeps a single shared descriptor and overrides the pointer and stride inline in the TMA instruction at runtime — less metadata bookkeeping, more GPU time on actual matmuls.
Generation over generation
The concrete comparatives NVIDIA gives are memory bandwidth and softmax (exponential) throughput. On both, the Blackwell-Ultra-to-Rubin step is the large one:
Memory is the quiet star here. Decode — the token-by-token generation phase — is fundamentally memory-subsystem bound, and agentic workloads spend more of their runtime there (long contexts, big KV caches, interactive generation). HBM4 doubles the interface width of HBM3e for 2.8× the bandwidth of Blackwell, while 288 GB of capacity keeps trillion-parameter models and their KV state resident instead of spilling to slower memory. Capacity and bandwidth do different jobs — one holds the context, the other feeds the cores — and decode needs both.
The data center as one unit of compute
The last move is to stop thinking about a GPU and start thinking about the AI factory as a fixed power budget. Rubin's efficiency story is rack-scale: Intelligent Power Smoothing uses on-rack energy storage to absorb the sharp power swings of AI workloads (about −10% average draw and −20% on 50 ms peaks), and DSX MaxLPS turns that reclaimed headroom into more GPUs — up to 40% more in the same megawatt envelope.
Same megawatts in the door. Intelligent Power Smoothing trims ~10% off average draw and ~20% off 50 ms peaks, and DSX MaxLPS turns that reclaimed headroom into up to 40% more GPUs — the AI-factory version of performance per watt.
All of this lands in the Vera Rubin NVL72 rack — third-gen MGX, cable-free trays, 45°C liquid cooling, hot-swappable NVLink switch trays — designed so compute, networking, cooling, and power behave as one execution domain.
The take
Rubin is a useful lens on where inference hardware is going: not chasing a bigger peak-FLOPs poster number, but co-designing every layer around the execution pattern of agents — sparse long-context attention, distributed MoE decode, tighter kernel handoffs, fused scale-up communication, and power treated as the real budget. It's the opposite end of the spectrum from running a model on a single DGX Spark, and the same underlying question — how do you keep expensive compute actually busy? — answered at rack scale. Keep the caveat in mind: the marquee numbers (10× throughput/watt, 40% more GPUs) are NVIDIA's own, on NVIDIA's workloads, for a platform still rolling out. The architecture is real and specific; the multipliers are the vendor's to prove.
Source: Inside NVIDIA Rubin GPU Architecture (NVIDIA, 21 July 2026). Performance figures are NVIDIA's, several on an internal 2T-MoE workload; the figures are NVIDIA's, the interactives are mine.