2026-07-10 · 7 min · inference-optimization · llm · systems · explainer · moe · quantization
A DGX Spark is not a datacenter. It's a small GB10 Grace-Blackwell box with 128 GB of unified memory — the kind of thing that sits on a desk. And yet one of them, running vLLM, will hold a conversation with dozens of people at the same time on a 35B-class model. The surprising part isn't the model or the silicon. It's a scheduling trick called continuous batching, and it's the single most important reason a small box can feel like a shared server.
The instinct is to picture the users taking turns — one prompt finishes, the next begins. That's not what happens. At every decode step the server bundles all the currently-active conversations into one forward pass through the GPU, and appends exactly one new token to each. Sixty-four people, one pass. Drag the concurrency and step the clock to see it move:
+ 4 more streams in the same batch (only 12 drawn). All 16 share one fused pass per step.
Every step, the server sweeps up all active streams and runs one fused forward pass — appending a single token to each stream's own KV cache. Because a lone decode barely uses the GPU, sharing the pass across many streams multiplies aggregatethroughput while each user's own rate drifts down. Streams that finish free their KV blocks; queued ones slot in — the batch churns without the GPU ever waiting.
Why decoding one token is a waste of a GPU
To see why batching helps so much, you have to look at what generating a single token actually costs. LLM inference splits into two phases — the prefill and decode — and it's the decode phase, one token at a time, that dominates a chat session.
Decoding one token for one user means: load the model's weights out of memory, multiply the single current token's vector through them, read that user's entire KV cache to do attention, and emit one token. The arithmetic is tiny — one token — but you had to stream all the weights through the chip to do it. That makes single-stream decode memory-bandwidth-bound: the GPU's compute units sit almost idle, waiting on memory. You paid to load the weights and barely used them.
Continuous batching is the fix that falls out of that observation. If you're loading the weights anyway, run more tokens through them on the same load. Gather N users' current tokens, stack them, and do one fused matmul against the weights you already fetched. The weight-load cost — the expensive part — is now amortized over N streams instead of one. Aggregate throughput climbs steeply, and keeps climbing until you run out of either compute or KV-cache memory.
The two curves are the whole trade. Pack in more users and aggregatethroughput climbs toward a ceiling — that's the win. But each user's own stream slows, because they now share every forward pass. The dots are committed spark-bench runs (concurrency 1/8/16); the 64-user point is a reported run past the committed sweep.
That's the trade in two curves. Aggregate tokens/second rises and saturates; per-user tokens/second falls the whole way. Both are true at once, and conflating them is the most common way these numbers get oversold.
"Continuous," not just "batched"
Plain static batching — wait for N requests, run them together, wait for all N to finish — would be useless for chat, because requests arrive at random times and finish at wildly different lengths. One long generation would stall everyone.
vLLM's batching is iteration-level (also called in-flight batching): the batch is re-formed every single decode step. A request whose prompt just arrived joins the batch on the next step; a request that just emitted its stop token leaves it and frees its memory immediately. The GPU never blocks waiting for the slowest member — that's the "continuous" part, and it's what the join/leave churn in the first diagram is showing. Streams flow through a batch that is constantly being rebuilt.
The enabling piece underneath is PagedAttention. Each user's KV cache is stored not as one contiguous slab but as a list of fixed-size blocks (the paged rows in the diagram), allocated on demand from a shared pool — exactly like virtual memory pages. Without it, fitting 64 independent, different-length caches into one memory space would fragment badly and you'd waste most of it. With it, 64 caches pack tightly, and a finished request's blocks return to the pool for whoever's next. KV-cache memory — not FLOPs — is what ultimately caps how many users fit.
The numbers, and which ones I trust
Here's where honesty matters. The story above is mechanism, and it's solid. The specific figures need sorting into what the public spark-bench results actually contain versus what's a reported run.
The model is Qwen3.6-35B — specifically an A3B mixture-of-experts: ~35B total parameters but only ~3B active per token. That's half of why it's fast (you only compute a fraction of the network each step) and why it fits comfortably (the weights are also quantized — the committed throughput runs use vLLM with NVFP4, a 4-bit format, and an FP8 variant). A 35B model serving this briskly on a 128 GB box is a quantized MoE, not a dense fp16 35B — and that distinction is load-bearing, not a footnote.
The committed spark_bench.csv sweeps concurrency 1 → 16 and shows the batching curve directly. On
the NVFP4 build, aggregate decode throughput roughly triples from a single user to sixteen:
…while each individual user's stream slows by roughly the same factor — you're trading personal latency for collective capacity:
The take
Continuous batching is the quiet reason "local inference" and "serving other people" stopped being mutually exclusive. The mechanism is honest and general: decode is memory-bound, a lone stream wastes the GPU, so amortize the weight-load across as many live streams as KV-cache memory will hold, rebuilding the batch every step so nobody waits on anybody. On a DGX Spark that turns a desk-sized box into a small shared server — genuinely dozens of concurrent users on a quantized 35B MoE.
What it is not is a speedup for the person on the other end. Each user's tokens come at a human reading pace, and that pace gets slightly worse as the room fills up. The DGX Spark headline — one small box, many users — is real and impressive; it's just a statement about aggregate capacity and clever scheduling, not about raw single-stream speed. Hold both halves of that at once and the number stops being a magic trick and becomes what it actually is: good systems engineering.
Mechanism (continuous / in-flight batching, PagedAttention) is standard
vLLM. Throughput and latency figures are read from the committed
spark-bench results/spark_bench.csv (Qwen3.6-35B-A3B, vLLM
NVFP4/FP8, concurrency 1–16); the 64-user / ~700 tok/s / ~38 W figures are a reported run and are labelled
as such above. The interactive diagrams illustrate the mechanism; their tok/s curve is a saturating fit
pinned to the committed points and the reported endpoint.