Nemotron 3.5 Bolt E4B: Gemma 4 E4B, retrained for talking robots, in a 4GB file that weighs 6.18 GB

On 2026-10-23 NVIDIA posted: "Available now: NVIDIA Nemotron 3.5 Bolt E4B 4GB. Our newest small open model is built for local assistants, interactive characters and robots that can carry on a conversation and use tools to get things done." The clip shows a Reachy Mini robot chatting and moving, "with Bolt handling conversation and NVIDIA Cosmos handling vision. Both models run locally on Jetson Orin Nano 8GB."

Unsloth reposted it with three claims of its own: the model "runs locally on 6GB RAM", it was "Trained on Gemma 4", and "it supports vision".

Each is half right. I read the primary sources: NVIDIA's blog post, the 18-page technical report, both model cards, and the files themselves. I read the safetensors and GGUF headers by HTTP range request (a few MB instead of 22 GB), compared byte ranges of individual tensors against Google's Gemma-4-E4B-it, and rendered NVIDIA's chat template to see the exact tool-call wire format. Then I ran one quant on a CPU, with llama.cpp, to check the memory claims and to see what the model actually says.

Numbers are labelled measured (I counted or read it from a file), reported (NVIDIA or Unsloth says so; I did not re-run it) or reasoned (my arithmetic on the other two).

expand

What I ran, and what I still could not

I ran Unsloth's UD-Q4_K_XL GGUF (4,758,125,760 bytes, SHA-256 checked against the Hub) with its mmproj-Q8_0 vision projector. I used llama.cpp built from source at commit 7894e8b, CPU only. The machine is a shared cloud VM with 4 vCPUs (Intel Xeon @ 2.10GHz) and no GPU. I used 2 threads at the lowest scheduling priority, while other jobs kept the load average between 6 and 12. Every speed below is therefore a floor, not a benchmark of the model. The results are in "What it does on a CPU" below.

What I still could not do:

• Run NVIDIA's own 4GB file. It is 6.18 GB, and the run had to stay under a 6.5 GB scratch budget alongside the build. The Unsloth quant has the same architecture with different quantization choices, listed below.
• Use a GPU or a Jetson. NVIDIA's VRAM, latency and DFlash numbers stay reported.
• Run any benchmark. Four prompts, one tool exchange, one long-context needle and one image make a smoke test, not an evaluation.
• Test a hard 6 GB memory limit. I measured what the process holds, not what the kernel could squeeze it into.

"Trained on Gemma 4" means: the same model, with new text weights

Distilled from Gemma 4 into a new architecture, fine-tuned from it, or trained on its outputs? The files answer this.

• Same architecture. config.json names Gemma4ForConditionalGeneration. Diffed against Google's config, exactly one field changes: max_position_embeddings goes from 131,072 to 65,536 (measured). Bolt is sold as a 64K-context model, and that is the reason.
• Same tokenizer. tokenizer.json has the same SHA-256 as Google's: cc8d3a0c… (measured, Hub LFS hashes).
• Same tensors, new values. Google's checkpoint has 2,130 tensors and Bolt's main file has 2,076. The other 54 are the key, value and key-norm weights of layers 24 to 41, moved into a side file called model-frozen-shared-kv.safetensors (measured). I come back to why below. I range-read the first megabyte of sampled tensors from both repos. All 8 sampled vision-tower tensors and all 8 audio-tower tensors are byte-identical to Google's. All 8 sampled text-layer tensors differ, and so do both embedding tables (measured).

The report fills in the rest. Bolt is "further post-trained from Gemma4-E4B-it with multi-environment RLVR and RLHF" (reported), on NeMo-RL and NeMo-Gym, with 256 prompts per batch and 8 rollouts each. The environments cover tool calling, instruction following and hierarchy, structured outputs and a generative reward model.

There was deliberately no SFT stage. NVIDIA tried SFT on its own post-training data first. Tool calling improved, but "SFT on data from diverse teachers resulted in a regression on conversational qualities and verbosity" (reported). So: a Gemma 4 E4B, retrained with RL, text side only.

Two choices fit the use case. Every prompt was trained with thinking on and off, at a "deliberate 2 : 1" non-reasoning to reasoning ratio. And because RL made answers longer, NVIDIA added length penalties and cycled them tighter and looser "as the training progressed" (reported).

What "E4B" means: per-layer embeddings

Gemma's card defines it: "The 'E' in E2B and E4B stands for 'effective' parameters." The small Gemmas use per-layer embeddings (PLE). An ordinary model gives each token one embedding vector at the bottom of the stack. PLE also gives each token a small extra vector for every layer, fetched from a separate table and fed into that layer.

Here are the shapes, from Bolt's config.json and safetensors header (measured):

• 42 decoder layers, width 2,560, vocabulary 262,144.
• hidden_size_per_layer_input is 256. Each token has one 256-wide vector per layer, so 42 × 256 = 10,752 numbers per token.
• The table embed_tokens_per_layer is therefore 262,144 × 10,752: 2,818,572,288 parameters. That is 35% of the checkpoint.
• In each layer, a gate (2,560 → 256) on the hidden state multiplies the token's per-layer vector. A projection (256 → 2,560) then adds the result to the residual stream. per_layer_model_projection (2,560 → 10,752) mixes a projection of the main embedding into the per-layer input.

Why this is cheap at run time: the table is only ever indexed. A forward pass reads one 10,752-number row per token, never the whole 2.8B-parameter table. It can sit in slow memory, or on flash.

Summing the BF16 header (measured):

Remove the PLE table and the two encoders and 4.64B remain. Remove the token embedding too and 3.97B remain (reasoned). Google's card calls E4B "4.5B effective (8B with embeddings)", and Bolt's card says "8B (4B effective)". Both are rounding the same thing: the parameters that do arithmetic for every token. The 4 in "E4B" is not a quantization level, and it is not the "4GB".

The "4GB" file is 6.18 GB, and both numbers are honest

NVIDIA's GGUF repo is called NVIDIA-Nemotron-3.5-Bolt-E4B-4GB. Its one model file, Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf, is 6,180,446,208 bytes (measured). I parsed its header (measured):

• 720 tensors and 7.52B parameters. Text only: there are no vision or audio tensors, and NVIDIA's repo ships no mmproj.
• per_layer_token_embd is stored at Q8_0: 2,994.7 MB, or 48% of the file.
• token_embd is also Q8_0: 713 MB. It doubles as the LM head, because the embeddings are tied.
• The 42 decoder blocks are Q4_K, with Q6_K for ffn_down and attn_v: 2.40 GB in all.
• The header records an importance matrix of 510 chunks. The report says it came from about 1M tokens of RL rollouts.

So the parts that compute come to about 3.17 GB, and the part that is only looked up is another 3.0 GB (reasoned). The lookup part is what llama.cpp moves aside. In llama.cpp's src/llama-arch.cpp, LLM_TENSOR_PER_LAYER_TOKEN_EMBD is registered as LLM_TENSOR_LAYER_INPUT with op GGML_OP_GET_ROWS. In src/llama-model.cpp, input-layer tensors go to dev_input = { cpu_dev, … } (measured, read from llama.cpp master on 2026-10-24). With every layer offloaded to a GPU, the table stays in host memory and only the rows the batch needs are gathered.

That reconciles the claims:

• "4GB" is VRAM. NVIDIA reports 3.5 GB of VRAM on an RTX 5070 for a 3K-token prompt, and 4.3 GB for 32K cached plus 2K new tokens (reported, llama.cpp CUDA, Q4_K_M). The 3.17 GB of compute weights plus the cache below fit those numbers, with a few hundred MB left for compute buffers (reasoned).
• "6GB RAM" is about the file. Unsloth's own docs say "E4B's ~4GB figure is VRAM, not total system RAM or download size" and list the Q4_K_M download at 6.18GB. On a CPU-only machine with llama.cpp's default mmap, every block weight is read on every token. The PLE table is only read one row per distinct token. Before running it, I estimated a working set near 3.3 GB at a few thousand tokens. The measurement below gives 3.4 GB for the working set. The process's RSS, though, is much larger than that, because llama.cpp pre-reads the whole file at load and, by default, keeps a second, repacked copy of the weights.

The widget below adds up the same numbers. Switch between GPU offload and CPU-only, drag the context, and turn off KV sharing to see what the 18 shared layers save.

Why the cache stays small: 18 layers borrow their keys

The 54 tensors moved to the side file are the KV projections of layers 24 to 41. num_kv_shared_layers is 18, so the last 18 layers do not compute keys and values. They reuse the cache of the last earlier layer of the same attention type. The weights exist in Google's checkpoint, but they are never used, and NVIDIA marked them "frozen" and set them apart. Only 24 layers own a cache.

Of those 24, every sixth is full attention: layers 5, 11, 17 and 23. Their KV uses 2 heads of dimension 512, so 4,096 bytes per token per layer in F16. The other 20 are sliding-window layers with a 512-token window and 2 heads of dimension 256. Their cache stops growing at 512 tokens, at about 21 MB in total. At the full 65,536-token context that comes to 1.09 GB of KV. Without sharing (7 full and 35 sliding layers) it would be 1.92 GB, about 43% more (reasoned from config.json).

The tool-call format is not JSON

Bolt keeps Gemma 4's chat format and adds two things to the template. I rendered NVIDIA's chat_template.jinja with a sandboxed Jinja for one tool turn, with conversational_mode on and thinking off (measured: this is the template's output, not the model's):

<bos><|turn>system
You are Jib, a desk robot.
 
You are having a spoken conversation with the user. Respond as briefly as possible while still addressing their questions fully.<|tool>declaration:set_light{description:<|"|>Turn a room light on or off<|"|>,parameters:{properties:{on:{type:<|"|>BOOLEAN<|"|>},room:{type:<|"|>STRING<|"|>}},required:[<|"|>room<|"|>,<|"|>on<|"|>],type:<|"|>OBJECT<|"|>}}<tool|><turn|>
<|turn>user
Kitchen lights off, please.<turn|>
<|turn>model
<|tool_call>call:set_light{on:false,room:<|"|>kitchen<|"|>}<tool_call|><|tool_response>response:set_light{value:<|"|>{"ok": true}<|"|>}<tool_response|>

Three things to notice:

• A call is call:name{key:value}. Strings are wrapped in a dedicated <|"|> token instead of quote characters. A JSON parser will not read it. Use a server's Gemma 4 parser: vLLM's --tool-call-parser gemma4, or llama.cpp's --jinja.
• Conversational mode is only a system prompt. It appends one fixed sentence, "Respond as briefly as possible…", to your system prompt, or creates one with it. No weights switch. NVIDIA chose the sentence with its internal Brevity Benchmark.
• Text goes before the tool calls. The one deliberate change to Gemma's template keeps an assistant's text ahead of its tool calls. The robot can say "turning it off" and then call the tool. With thinking on, the template inserts <|think|> into the system turn. With thinking off, the prompt matches Gemma's.

What it does on a CPU

Everything in this section is measured by me on the machine described above, using Unsloth's UD-Q4_K_XL file. The model outputs are quoted verbatim and trimmed only where marked. They were generated on a CPU VM at the card's recommended sampling (temperature 1.0, top-p 0.95) with seed 7. They are single samples, not curated ones.

What is in the file

Unsloth's layout differs from NVIDIA's. I read it with llama.cpp's gguf-py. The per-layer table per_layer_token_embd is IQ4_NL at 1.585 GB, where NVIDIA uses Q8_0. token_embd is Q4_K at 0.377 GB, and everything else comes to 2.779 GB. The parts that compute therefore total 3.156 GB, almost the same as in NVIDIA's file. Unsloth saved its 1.4 GB on the lookup table.

Speed

llama-bench with 2 threads gave 30.77 ± 12.21 tokens/s for a 512-token prompt and 2.46 ± 0.37 tokens/s for generation. The wide error bars come from the other tenants on the machine. Through llama-server, short prompts were processed at 10 to 22 tokens/s and generation ran at 1.5 to 3.1 tokens/s. A 15,520-token prompt took 539.2 s to process (28.79 tokens/s), so a 16K-context question costs about nine minutes before the first word. On two shared vCPUs you can talk to it, slowly.

Memory: three different numbers

RSS comes from /proc/<pid>/status. "Actually touched" is the Referenced total from /proc/<pid>/smaps, after clearing the bits with /proc/<pid>/clear_refs before the run.

What the table shows:

• The loader reads the whole file. File-backed RSS is 4.76 GB, the entire file, in every configuration. The PLE table is paged in at load even though inference barely reads it. These are clean page-cache pages that the kernel can drop under pressure, but they count toward RSS.
• The working set matches the lookup argument. With repacking off, three chats touched 3.14 GB of the mapped file. That is the 3.156 GB of compute weights and next to nothing of the 1.585 GB table. During the 16K run with defaults, only 0.06 GB of the file was touched; the compute weights were being read from the repacked copy. The working set is about 3.4 GB at short context, close to my 3.3 GB estimate (reasoned from the table).
• Repacking doubles the compute weights. By default llama.cpp's CPU backend copies the weights into a layout its kernels prefer: 3.4 GB of anonymous memory on top of the mapped file. That is what pushes default RSS to 8.2 GB, and 8.5 GB at 16K.

So "runs on 6GB RAM" holds for the working set, and for RSS with --no-repack (5.03 GB). It does not hold for llama.cpp's default RSS. A phone OS that kills an app by its resident size will see the larger number. --no-repack did not slow generation here: the same three chats ran at 2.3 to 3.1 tokens/s without it and 1.5 to 2.0 with it, but the load noise on this VM is larger than any difference repacking makes.

Three chats

With conversational_mode on and thinking off, system prompt "You are Jib, a small desk robot that talks with people in an office.", I asked: "Hey Jib, I have a meeting in ten minutes and I haven't eaten. Quick idea?"

Grab a protein bar. Quick energy.

That is nine tokens, which is the brevity training doing its job. The same prompt without repacking (a different run, same seed) gave "Grab a protein bar or some nuts. Quick energy."

With thinking off, I asked about the topic of this article: "In two sentences: why does a GPU need less memory for a model whose embedding table stays on the CPU?"

The GPU does not need to store the entire embedding table because it only needs memory for the active parts of the model and intermediate computations. The CPU handles the larger embedding table, transferring only the necessary lookups to the GPU as needed for forward and backward passes.

Right in outline; "backward passes" is wrong for inference.

With thinking on, I asked: "A shop sells pens at 3 for $2. How much do 15 pens cost? Answer with the number only." The reasoning (trimmed):

2.  **Find how many groups of 3 are in 15 pens:** $15 / 3 = 5$ groups.
3.  **Calculate the total cost:** 5 groups * $2/group = $10.
 
The answer should be the number only.

The answer: 10. That took 116 tokens in total, reasoning included.

One tool call

I sent the set_light tool from the template section through llama-server's OpenAI endpoint (--jinja), with conversational mode on and thinking off. The system prompt was "You are Jib, a desk robot that controls the office lights." and the user said "Kitchen lights off, please." The raw text the model generated, read from /completion on the same rendered prompt:

<|tool_call>call:set_light{on:false,room:<|"|>Kitchen<|"|>}<tool_call|>

That is exactly the Gemma 4 format, followed by end-of-turn. llama.cpp parsed it into a standard OpenAI tool call:

[{"type": "function", "function": {"name": "set_light", "arguments": "{\"on\":false,\"room\":\"Kitchen\"}"}, "id": "HV2BrvJuN00EiOvSAeD4WGVeSwQ9Cjwj"}]

The call parses. The arguments are valid JSON, both required keys are present, and on is a real boolean. The model capitalised "Kitchen", which a strict enum would reject.

The second turn surprised me. After I sent back the tool result {"ok": true}, the model ended its turn with no text in 3 of 4 samples (seeds 7, 2 and 3). Seed 1 said Done.. For a voice robot, silence after an action may be fine. For a chat UI, plan to say "done" yourself.

One image

With Unsloth's Q8_0 projector, I sent NASA's public-domain Apollo 17 "Blue Marble" photograph (AS17-148-22727, 1280 × 1280) with thinking off and the prompt "What is in this image? Two sentences." The image and prompt came to 749 prompt tokens, processed in 65.6 s. The reply:

This image is a view of the Earth from space, centered over Africa and parts of Eurasia. It shows landmasses like the Sahara Desert and the African continent surrounded by oceans and white clouds.

That is correct: the photograph shows Africa, Arabia and Madagascar under cloud. The retrained language model still reads Google's unchanged vision encoder well enough for a caption. The report's MMMU-Pro drop is about harder questions than this. With the projector loaded and repacking off, peak RSS was 6.13 GB.

Vision: in the checkpoint, not in the training

Is Unsloth's "it supports vision" true? It depends on which file you download.

• The BF16 checkpoint contains Gemma's 0.169B vision tower and 0.309B audio tower, byte-identical in every tensor I sampled (measured). The card still lists the inputs as "Text" and its vLLM command passes --limit-mm-per-prompt '{"image":0,"video":0,"audio":0}'.
• NVIDIA's 4GB GGUF has no vision tensors and no projector (measured).
• Unsloth's repo adds mmproj files: F16 at 0.99 GB and Q8_0 at 0.56 GB (measured, sizes). Images work by bolting Google's unchanged encoder onto NVIDIA's retrained language model, and in my one test the caption was right.

NVIDIA measured what that costs. "Although the model was trained text-only", the report says, audio got better and vision got worse (reported, Table 8, reasoning mode): VoiceBench 80.30 against Gemma's 78.20, but MMMU-Pro Vision 42.02 against 47.63, and OmniDocBench 1.5 error 0.42 against 0.21. NVIDIA blames "misalignment between the vision encoder and the language model introduced by text-only training". It tried retraining the encoder alone and "observed no meaningful improvements". The Reachy demo does not use Bolt's eyes at all: Cosmos 3 Edge does the seeing.

Benchmarks: a clear win on instruction and persona, not on agents

NVIDIA picked 21 benchmarks in five groups and ran them all in its own NeMo Gym harness, with Gemma 4 E4B, Qwen3.5-4B and its own Nemotron 3 Nano 4B as baselines. Here are the BF16 rows that move the most (reported, card and report Table 3):

My arithmetic on the full 21-row table (reasoned):

• Averages. Bolt averages 63.85 against 58.10 for Gemma 4 E4B, 57.75 for Qwen3.5-4B and 44.77 for Nano 4B. That matches the 63.9% and 58.1% in NVIDIA's chart and its "~40% higher" claim over Nano: the ratio is 1.43.
• Against its parent. Bolt is ahead by more than 2 points on 13 rows. Eight rows sit within 2 points either way, the only loss being EQ-Bench. The report sizes repeats "to limit score variation to 1–2 percentage points", so those 8 are best read as ties.
• RoleBench is graded against Gemma. It is a pairwise win rate "against a fixed Gemma4-E4B-it reference". Gemma's 49.95 is a coin flip against itself, and Bolt's 67.05 means the judge preferred Bolt two times in three.
• Against Qwen3.5-4B. Bolt loses 5 rows: BFCL-v3, ToolSandbox, Tau2-Bench by 20 points, RULER by 8 and LongBench v2 by 9. The two rows that look most like an agent finishing a multi-turn task (Tau2, ToolSandbox) belong to Qwen.
• Quantization is nearly free here. Bolt's Q4_K_M rows average 63.58 against 63.85 in BF16. Some rows go up: RelyToolBench +6.13, ToolSandbox +5.36. That is noise or harness variance, not 4-bit magic.

A thread in the replies credited Bolt with "MultiChallenge: 56.03%" and "Instruction retention: 76.8%". Neither number is on the card or in the report, so I cannot check them.

expand

That chart averages only MultiChallenge, RoleMRC, BFCL-v3, IHEval, SysBench and RelyToolBench, so Bolt scores 70.2 against Gemma's 63.2. It leaves out Tau2-Bench and the long-context rows, where Qwen leads. The second chart is the one I find more useful. It shows the length penalties working:

expand

Bolt with thinking off (57.3% at 210 tokens) is about level with Gemma 4 E4B with thinking on (58.1% at 782 tokens). The "up to ~80% fewer tokens" claim is 672 against Qwen's 3,604, which is 81% fewer (reasoned). For a voice agent, those are the tokens the user sits through.

Speed numbers, all NVIDIA's

On a Jetson Orin Nano 8GB, NVIDIA reports 4.2 GB of VRAM, a 5,785 ms time to first token and 15.5 tokens/s for a 3K-token prompt. With 32K cached and 2K new tokens it reports 4.9 GB, 6,259 ms and 13.0 tokens/s (reported, llama.cpp CUDA, Q4_K_M). Nearly six seconds before the first token is long for a robot that is supposed to chat. The report does not say how the Reachy demo hides it.

The release also ships a five-layer, 360M-parameter DFlash drafter (see DFlash2). NVIDIA reports 1.21× the throughput of Gemma's own MTP drafter on an RTX 5070 (reported).

Licence: OpenMDW, over an Apache base

Bolt's weights are under the OpenMDW License 1.1, not the NVIDIA Open Model License. OpenMDW is permissive: keep the notices, lose your rights if you sue over patents or copyright, and there are no conditions on outputs. Gemma 4, which Bolt was "post-trained on", is Apache 2.0. Its vision and audio towers ship unchanged, so if you redistribute Unsloth's mmproj you are redistributing Apache-2.0 weights. That is my reading, not legal advice.

Where it fits

If you were going to run Gemma 4 E4B for an assistant, character or robot, Bolt is a drop-in swap. It has the same architecture, the same runtime support and the same memory. In NVIDIA's own harness it follows system prompts and personas much better and says less. If your agent mostly runs long multi-step tool tasks, the card's Tau2 and ToolSandbox rows say Qwen3.5-4B is still the better 4B. If you need the model to see, keep Gemma 4 E4B, or give the eyes to a separate model, as the Reachy demo does.

For background, see the parent's KV-cache design in Gemma 4, NVIDIA's large sibling in Nemotron 3.5 Super VL, and the voice-agent plumbing that already runs on Reachy Mini in speech-to-speech. On a CPU it works, slowly. It answers briefly, calls tools in clean Gemma format, finds a needle at 15K tokens and captions a photo. Pass --no-repack if resident memory is your budget: the lookup table costs almost nothing to run, but the default loader still holds every byte of it.