# NVIDIA Rubin: co-designing a GPU for the shape of agentic inference

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/nvidia-rubin
> date: 2026-07-22
> tags: hardware, gpu, inference, systems, explainer
An agentic workload isn't a single prompt and response — it's sustained inference across many reasoning
steps that plan, call tools, verify, and revise over long contexts. That execution pattern stresses a
GPU differently than a chatbot does: it wants low *per-step* latency, high decode throughput, efficient
long-context attention, large KV-cache capacity, and the ability to spread a model across tightly
coupled GPUs. [NVIDIA's Rubin GPU](https://developer.nvidia.com/blog/inside-nvidia-rubin-gpu-architecture-powering-the-era-of-agentic-ai/)
is a bet that the right response is to co-design for exactly that pattern — and the headline claim is
**up to 10× more agentic throughput per unit of energy than Blackwell.**

<Figure
  src="/articles/nvidia-rubin/fig1.png"
  alt="Pareto-frontier chart of agent throughput versus interactivity for Hopper, Blackwell, and Rubin systems; the Rubin NVL72 + Vera frontier sits far above the others, roughly 10× more agents and about 2× more tool calls."
  caption="Pareto frontiers of throughput vs interactivity — a ~10× generational uplift on NVIDIA's internal 2T-MoE agentic workload (NVIDIA, Figure 1)."
/>

That "10×" is a vendor number on an internal 2-trillion-parameter MoE workload, so read it as a design
target rather than an independent benchmark. What's more interesting than the single figure is *how*
it's assembled — because it isn't one trick, it's a checklist of agentic bottlenecks each with its own
answer.

## The chip

<Figure
  src="/articles/nvidia-rubin/fig2.png"
  alt="Annotated die diagram of the NVIDIA Rubin GPU: two compute dies joined by NV-HBI, graphics processor clusters, HBM4 controllers, a central L2 cache, NVLink, and PCIe Gen 6."
  caption="The Rubin GPU: two reticle-limited dies unified over the NV-HBI inter-die link (NVIDIA, Figure 2)."
/>

Physically, Rubin is **two reticle-limited compute dies** fused into one package over a high-speed
inter-die link (NV-HBI): **336 billion transistors, 224 SMs, 896 Tensor Cores**, a third-generation
Transformer Engine that flexes precision across formats for **up to 50 petaflops of NVFP4**, up to
**288 GB of HBM4 at 22 TB/s**, and NVLink 6 at **3,600 GB/s** of scale-up bandwidth. Those are the raw
numbers; the architecture is about turning them into *sustained* utilization.

## One checklist, many bottlenecks

Here's the spine of the whole design. Each agentic-inference bottleneck gets a specific Rubin feature —
click through them:

<BottleneckMap />

Two are worth dwelling on. **Long-context attention** is where agentic runs spend their time, and Rubin
attacks it from two sides at once: it compresses the intermediate attention scores into a structured
**2:4 sparse** form so softmax and the second attention GEMM operate on fewer values, and it raises
exponential throughput so softmax — which becomes the bottleneck once the matrix math speeds up — keeps
pace.

<Figure
  src="/articles/nvidia-rubin/fig3.png"
  alt="Diagram showing a dense activation matrix converted into a 2:4 sparse matrix plus metadata, applied to the attention and MLP activation stages of a transformer block."
  caption="Activations compressed to 2:4 sparse (values + metadata), cutting work in softmax and the second attention GEMM without changing the block's interface (NVIDIA, Figure 5)."
/>

The other is **MoE decode**: as expert counts climb, just locating and moving expert weights becomes
the cost. Blackwell tracks one memory descriptor per expert; Rubin keeps a single shared descriptor and
overrides the pointer and stride inline in the TMA instruction at runtime — less metadata bookkeeping,
more GPU time on actual matmuls.

## Generation over generation

The concrete comparatives NVIDIA gives are memory bandwidth and softmax (exponential) throughput. On
both, the Blackwell-Ultra-to-Rubin step is the large one:

<GenSpecs />

Memory is the quiet star here. Decode — the token-by-token generation phase — is fundamentally
**memory-subsystem bound**, and agentic workloads spend more of their runtime there (long contexts,
big KV caches, interactive generation). HBM4 doubles the interface width of HBM3e for **2.8× the
bandwidth** of Blackwell, while 288 GB of capacity keeps trillion-parameter models and their KV state
resident instead of spilling to slower memory. Capacity and bandwidth do different jobs — one holds the
context, the other feeds the cores — and decode needs both.

## The data center as one unit of compute

The last move is to stop thinking about a GPU and start thinking about the **AI factory** as a fixed
power budget. Rubin's efficiency story is rack-scale: Intelligent Power Smoothing uses on-rack energy
storage to absorb the sharp power swings of AI workloads (about −10% average draw and −20% on 50 ms
peaks), and **DSX MaxLPS** turns that reclaimed headroom into more GPUs — up to **40% more** in the
same megawatt envelope.

<PowerFactory />

<Figure
  src="/articles/nvidia-rubin/fig5.png"
  alt="Grid comparison of AI-factory capacity: without DSX MaxLPS much of the power budget is stranded and unused; with DSX MaxLPS up to 40% more GPU slots fit in the same budget."
  caption="For a fixed power budget, DSX MaxLPS recovers stranded capacity to provision up to 40% more GPUs (NVIDIA, Figure 10)."
/>

All of this lands in the Vera Rubin NVL72 rack — third-gen MGX, cable-free trays, 45°C liquid cooling,
hot-swappable NVLink switch trays — designed so compute, networking, cooling, and power behave as one
execution domain.

## The take

Rubin is a useful lens on where inference hardware is going: not chasing a bigger peak-FLOPs poster
number, but **co-designing every layer around the execution pattern of agents** — sparse long-context
attention, distributed MoE decode, tighter kernel handoffs, fused scale-up communication, and power
treated as the real budget. It's the opposite end of the spectrum from running a model on a single
[DGX Spark](/articles/dgx-spark-batching), and the same underlying question — *how do you keep
expensive compute actually busy?* — answered at rack scale. Keep the caveat in mind: the marquee
numbers (10× throughput/watt, 40% more GPUs) are NVIDIA's own, on NVIDIA's workloads, for a platform
still rolling out. The architecture is real and specific; the multipliers are the vendor's to prove.

---

*Source: [Inside NVIDIA Rubin GPU Architecture](https://developer.nvidia.com/blog/inside-nvidia-rubin-gpu-architecture-powering-the-era-of-agentic-ai/)
(NVIDIA, 21 July 2026). Performance figures are NVIDIA's, several on an internal 2T-MoE workload; the
figures are NVIDIA's, the interactives are mine.*