Qwenputer: a RISC-V computer compiled into Qwen3.5-4B, and what its memory actually is

mdjsonmcpcopy↗ai

2026-10-30 · 21 min · qwen · linear-attention · hybrid-attention · compilers · agents

Why read this

Notable· top 60%

Checkpoint headers, 12 bit-identical weight comparisons, and the delta-rule RAM rebuilt on transformers' own recurrence. The model itself was not run.

• Original analysis
• Widely used
• A new technique

·LLM architecture·Runs on a consumer GPU·Apache-2.0·Research teardown

How this was scored

Is it new?

2 of 3: A real new idea, method or capability

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

1 of 3: API-only, gated or restrictive licence

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

1 of 3: Relevant for months

Does it affect many?

2 of 3: A widely used model, tool or lab release

Only here?

2 of 3: A teardown or measurement few others did

Score 62 of 100, ranked 329 of 666 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

@divagr1925 posted that Qwen3.5-4B now has "a full RISC-V CPU inside its weights, using the model's own Gated DeltaNet layers as RAM", running C and MicroPython inside its forward passes with no external runtime. On the DeepMind arithmetic sets the post reports a base score of 55 to 90%, 95 to 100% for base Qwen with a Python tool, and 97.5 to 100% for Qwenputer. The write-up calls it "not a plugin, a tool or a sandbox".

That is a strong claim, and most of it can be checked without a GPU. The checkpoint's tensor headers are public, any tensor can be sampled by range request, the compiler is open, and the memory trick is a few lines of linear algebra that transformers already implements. This article does that. It checks the layout of the checkpoint, tests whether Qwen's weights are really untouched, rebuilds the memory on the reference recurrence from transformers, and runs a toy computer on top of it. I did not run Qwenputer itself: it wants a 24 GB GPU, and I had a CPU box. A later pass ran the repo's emulator, firmware and tests on that box; see "What running it showed".

What is claimed

The repository's own summary, read at commit f07a0b2 (30 October), is more careful than the post (all reported):

• A 32-bit RV32IM core, with the Zba extension and a bignum coprocessor, compiled into 11 extra layers in front of Qwen3.5-4B's 32. It passes the official riscv-tests suites, its programs match qemu, and about 18 million instructions have run free-running without an error.
• Its memory is 256 KB of code ROM, 128 KB of read-only data and 32 KB of RAM, all held in the state of Gated DeltaNet heads. The circuit has about 1M nonzero weights and costs 1.2 to 1.4 tokens per instruction.
• Qwen reaches it through its ordinary tool-call syntax. A calc firmware does bignum arithmetic (+ - * // % ** gcd, and exact fractions in v2) and answers inside the forward pass. Qwen's weights are not edited: "Qwen's original weights stay exactly as they are", with the computer added in extra dimensions and layers.
• The checkpoint is 18.8 GB, roughly double the base model, "although most of it is zero padding", and a small driver called RvChat is needed because plain generate() does not run it.

The repo's gate table is the cleanest account of what the machine buys. On 40 problems per set, with Qwen's token budget at 4,096 (reported, repo docs/rv32.md; calc v1 in every row except the last, which is the v2 re-run for Qwenputer and the host-answered tool, and base Qwen's score is from the v1 run):

So the machine matches a Python tool, it does not beat one, and the repo says why the misses happen: "rv1's misses are Qwen's, not the machine's". Every value the machine returned was exact, and the failures were a dropped sign, a re-subtraction after a correct result, and loops of 44 calls on an answer that was already right. The headline claim is therefore about where the arithmetic runs. The gain over base Qwen comes from having a calculator at all.

What the checkpoint is

Every number in this section is measured: I read the tensor headers of both checkpoints through HTTP range requests, and sampled rows of the tensors I wanted to look inside.

• 43 layers in model.layers: 0 to 10 are the compiled CPU, 11 to 42 are Qwen's 32. The pattern from layer 11 is Qwen's own: 24 Gated DeltaNet layers and 8 full-attention layers, one full layer after every three (layers 14, 18, ... 42).
• Hidden width 3,968, against Qwen's 2,560. The MLP width is unchanged at 9,216.
• 478,656 embedding rows, 230,336 more than Qwen's 248,320. The repo counts 230,324 added chip tokens; the 12-row difference is presumably padding, which I did not check.
• 9,357,602,496 parameters: 1.90B in the embedding table, 1.93B in the 11 chip layers, 5.53B in Qwen's layers. The base checkpoint is 4.66B, and about 0.45B of that is a vision tower that Qwenputer drops.
• The eleven chip layers are ordinary Gated DeltaNet layers in shape. A chip layer has the same 175,013,824 weights as a Qwen linear-attention layer (in_proj_qkv is 8,192 by 3,968, the MLP is 9,216 by 3,968), because they are the same module with different numbers in it.

Is Qwen's own weight block untouched? I compared sampled rows of the Qwenputer tensors with the matching base tensors (Qwenputer layer i + 11 against base layer i). I compared 24 random rows each of the MLP up_proj and down_proj of four layers, the in_proj_qkv of a linear layer, the q_proj and k_proj of a full-attention layer, and 48 rows of the embedding table, always over the 2,560 shared dimensions. All twelve comparisons were bit-identical. That supports "Qwen's original weights stay exactly as they are". It is a sample, not a proof: a few dozen rows per tensor out of billions of weights, and I did not touch the layers between those I sampled.

Is the padding really zero? In the sampled Qwen layers, the nonzero entries were exactly the first 2,560 columns: 64.5% of each matrix (2,560/3,968). The rest is zero, which is the "mostly zero padding" the post describes. Counting the Qwen layers as their base values plus zeros, the unique content of the file is roughly the 4.2B-parameter base text model, the embedding rows, and about a million chip weights, in a file that is 9.36B parameters wide.

What does a compiled layer look like? Very different. In rows sampled from chip layers 0, 5 and 10, at most 0.09% of the entries were nonzero, and in one tensor (in_proj_qkv of layer 0) none of the 48 rows I sampled held a nonzero value. The values that are there are large: the biggest magnitudes were 2³⁰ (1.07 × 10⁹) and 2⁴⁷ (1.41 × 10¹⁴). The design note explains why. The compiler needs every number the circuit touches to be exact in bf16, so it keeps values to at most eight significant bits and uses huge power-of-two scales to push other effects away (reported). The sampled rows are consistent with a layer built that way; they do not verify the author's count of 1,040,595 nonzero weights, which at a density of 0.054% is too sparse to estimate from 48 rows.

The embedding table shows a second thing. The design note says every token carries a "ballast" of large constants so that normalisation layers divide by an exactly known power of two. In the sampled rows, Qwen's own token rows had about 93 nonzero entries beyond the 2,560 they need (2,653 on average, with a largest magnitude of 2⁴⁰), and the appended chip-token rows were 2.4% nonzero, about 94 entries each. So the ballast is in the table, on both kinds of token (measured, 64 rows per region).

The memory trick, on the real recurrence

Qwen3.5 uses Gated DeltaNet in three of every four layers. Each head keeps a state matrix $S$ with one row per key dimension, and each token updates it with the delta rule. The reference code in transformers (torch_recurrent_gated_delta_rule, read from the installed package) does, for each token:

The first step is the gate: $e^{g}$ is a number between 0 and 1 that fades the whole memory. The second nudges the state along the key direction toward the new value $v$, and $\beta$ says how far. The third reads the state with a query. Now take a one-hot key $k = e_a$, set the gate to 0 in the log domain (so nothing fades), and set $\beta = 1$. The second step becomes $S_a \leftarrow v$: it overwrites row $a$ and nothing else. A one-hot query $q = e_a$ then reads back exactly row $a$. That is a RAM with addresses, and its size does not depend on how long the conversation is. The idea that the delta rule is an associative memory is older: Schlag, Irie and Schmidhuber showed it in 2021 (Linear Transformers Are Secretly Fast Weight Programmers). What Qwenputer adds is making it exact and addressable inside a production model, and the next sections check how exact.

How exact, on the reference code

I ran the reference function itself, in fp32 on CPU, with one head of 128 key rows and four byte-valued columns (a 32-bit word). Each token either writes (a one-hot key, $\beta = 1$) or reads ($\beta = 0$, so nothing changes).

A unit one-hot vector is not exactly one inside this code. The function L2-normalises keys and queries as x * rsqrt(sum(x²) + 1e-6), so a unit key becomes 0.9999995, writes only approximately overwrite, and a byte read comes back up to about 2.4 × 10^-4 off. For byte values that is invisible after rounding to the nearest integer, and it is the reason the design keeps its values to eight bits. Scaling the key up before the norm makes the 1e-6 vanish, and the remaining 1.5 × 10^-5 is the rounding of dividing by $\sqrt{128}$ in the query scale. This is the same mechanism the design note describes when it gives every token a large ballast so the normalisations see exactly known magnitudes (reasoned: I did not trace this through the compiled model, I showed it on the reference function). The practical reading is that exact RAM is not a property of the delta rule alone. It is the delta rule plus a lot of care about magnitudes.

What breaks it

Everything the compiler has to hold fixed. All rows below are measured on the same reference function, one head, 128 key rows.

Stock Qwen3.5 sits in the middle of that table. Its gate and write strength are learned, and the model has no reason to hold either at 1, so I expect (reasoned) a smooth, fading, blended memory: good for language, wrong for addresses. Qwenputer's circuit overrides both on the positions it owns, which is the point of its "mute" mechanism, where columns on a marker dimension drive Qwen's own gate and write-strength inputs to $-1024$ on chip tokens so that Qwen's state passes through a trace unchanged (reported, design note). The random-key row is the usual associative memory: with random keys the read error grows with the number of items and passes 100% once items exceed the 128 rows. One-hot keys have no crosstalk at all.

Why bother with the state at all

Full attention keeps a KV cache that grows with the conversation. A Gated DeltaNet head keeps a fixed matrix. From Qwen3.5-4B's config (measured: 24 linear layers, 8 full layers, 32 value heads of 128 by 128, 4 KV heads of width 256):

The state is a constant 48 MiB, and the cache overtakes it at 1,536 tokens. That is the post's "fixed-size as the chat grows" (reasoned from the config; the compiled core's 32 KB of RAM is a small corner of the same states).

From RAM to a computer

A memory is not a CPU. The compiler's job is to turn a circuit description into weights, and the repo's design note (reported) says how: a long list of small steps (read this register, add these two numbers, jump if zero) becomes pieces of transformer layers. Attention heads do the lookups, SwiGLU units do the arithmetic, and the Gated DeltaNet heads hold the state. The arithmetic uses thresholds: silu(K x) with $K = 128$ is an exact ReLU at every value the circuit produces, and a 32-bit add is two MLP sublayers. Eleven layers are enough because depth is set by the longest dependency chain in the design, not by the number of instructions.

The memory map is a good picture of how it fits. The code ROM is 256 KB of pre-decoded 4-byte words (one word per instruction, with the destination and source register slots and the immediate already unpacked), held in the key heads of two sublayers. The 32 KB of RAM is 16 key heads by 2 value heads by 128 rows of 32-bit words, again in two sublayers. The register file is two key heads (the two read ports) with 32 rows each. A write is gated: a key is the address from the previous token and a store flag, and $\beta$ is 1 only on a store.

The program counter is the thing that makes a computer. Each instruction is a few generated tokens: one naming the memory row to write (ADDR), one carrying the value (VAL), and a STEP token to end the instruction. Those tokens are the CPU's execution trace, so while Qwen decodes them the CPU is running. At 1.2 to 1.4 tokens per instruction, the repo reports about 3,200 trace tokens a second on a single L4 with its fused kernel (reported).

I could not run their model in the first version, and the repo's own core only later (see "What running it showed"). What I could show then is the memory half of the idea, on the same recurrence. I built a toy register machine: eight 32-bit registers and a small data memory live in rows of one head's state, with each word stored as four byte columns. Every register read or write goes through the reference recurrence as a token. The ALU and the program counter are plain Python, so this is the RAM alone, not the compiled arithmetic. I checked every run against a pure-Python interpreter.

My toy spends 2.3 tokens an instruction because it reads each operand with its own token. The compiled core presumably folds the reads into the attention that produces the next token, which would explain the repo's 1.2 to 1.4; I did not check that. The two numbers are not comparable, and I do not claim anything about the core's speed from mine. The point of the exercise is narrower. Twenty thousand instructions of read-modify-write traffic through the real Qwen3.5 recurrence never lost a bit, once the keys carry enough magnitude.

How Qwen uses it

The circuit and the language model share positions. After a <tool_call> tag, a bridge captures the call's tokens into a text buffer that the core reads as memory at address 0x8000 (16 bytes per token, up to 80 tokens per call). </tool_call> starts a run. Each byte the firmware prints is echoed as the matching Qwen token, so Qwen reads the answer as an ordinary tool response, and a halt is echoed as a newline. The circuit owns exactly those positions, and Qwen predicts everywhere else (reported, design note). The calc firmware is C compiled with -DBRIDGE; it parks between calls, evaluates the first parameter, and prints a <tool_response> with values of up to 62 digits.

Two details of the repo are worth keeping in view:

• The merged model's firmware is calc. The post says the CPU "runs C and MicroPython in its forward passes". The design note, in its list of what is not done, says MicroPython "has run on the standalone core and executor (P2), not yet inside the merged model; it needs a bridge build (P4)". The two statements can both hold, since the core can run those programs while the shipped Qwen bridge drives calc. The later run (below) confirms the design note's wording at the newest commit.
• Qwen's text is almost unchanged, not identical. The mean KL divergence between the merge and base Qwen over 16 prompts is 6.1 × 10^-4, against a floor of 5.1 × 10^-4 between base Qwen's own two attention kernels, and greedy text matched on 10 of 16 prompts (reported, rv/textcheck.py). That is "within noise", as the post says; it is also not bit-identical.

Limits

All reported by the author or the repo, and consistent with what the headers show:

• The checkpoint is 18.7 GB in bf16 (9.36B parameters), and about 55% of the entries are zero by construction, because Qwen's layers are padded to a wider hidden size. Building it from the base model needs about 31 GB of RAM.
• It needs the RvChat driver to load the firmware and keep trace tokens out of Qwen's layers, so it is not a drop-in generate() model.
• The 80-token capture window for a call was not raised, because a wider window would overflow bf16 thermometer weights.
• Text decoding is still Qwen's modules at about 68 ms a token on an L4, with the reference linear-attention path in place of the optimised kernels, which are not installed.
• 0.8B was skipped: it has one value head per key head, and the core needs two.

What running it showed

The first version of this article read the repo and did not run its code. That was a limit of the sandbox I had then, not a choice, and I have since run the code (commit bec1a0c, 31 October, with the owner's permission; the scripts and logs are in public/articles/qwenputer/verify/). bec1a0c differs from the f07a0b2 I first read in the README only, so everything quoted from the design notes still stands. Everything below is measured by me.

• The MicroPython claim is unchanged. At bec1a0c, experimental/riscv/docs/rv32.md still says MicroPython "has run on the standalone core and executor (P2), not yet inside the merged model; it needs a bridge build (P4)". The merged model's bridge firmware is calc-bridge.elf. The post's "runs C and MicroPython" holds for the standalone core only.
• The repo's emulator agrees with my toy's semantics, with one correction to a comparison. I assembled my three toy programs as real RV32 and ran them on the repo's own emulator (ref.Machine, the generator of the trace the circuit is trained on). The registers match mine: 5,050; 102,334,155 and 165,580,141; and 83,810,205. The instruction counts are mine plus the setup the assembler adds: 305, 206 and 20,375, against my 303, 204 and 20,371, because a constant above 2,047 takes two real instructions (lui and addi) and the program ends with an exit call.
• The 2.3 tokens per instruction was my toy's, and the real trace costs 1.0 to 1.3. The repo's trace for the same three programs is 306, 266 and 27,164 tokens: 1.00, 1.29 and 1.33 tokens per instruction. The 1.2 to 1.4 in the repo is therefore not an artefact of its mix of programs; it holds on mine too. My toy's 2.33 on the multiplication is the price of reading each operand with its own token, as I guessed, and now I know the real trace does not pay it. The earlier text said I had not checked this, and the comparison is now made.
• The firmware calc runs, and it is exact. On the repo's reference machine, 12345*6789 prints 83810205 in 403 instructions and 952 trace tokens (2.36 per instruction: a short calculation is mostly fixed work per digit, and the bignum steps cost more tokens than the C mix). 2**100 gives 1,267,650,600,228,229,401,496,703,205,376 in 1,845 instructions. The repo's own calc --cases 300 compared 300 random problems of ten kinds against Python with 0 mismatches (7 were too large for the digit file and were refused). So the 1.2 to 1.4 is a property of C-compiled programs, and calc's own traces run higher on small inputs.
• The test suites pass where I ran them. test_rv_ref.py and test_rv_bridge.py (the emulator against qemu output, and the bridge): 131 passed, 12 skipped. test_q35_numerics.py and test_q35_mute.py (the primitives run in the real transformers Qwen3.5 modules, in bf16 and fp32, and the check that text tokens are bit-identical and chip tokens leave Qwen's state alone): 7 passed in 28 s, with bf16 required to be bit-exact.
• The key-scale result reproduces. I re-ran my script from scratch: the output file is byte-identical to the first run (a unit key reads back 2.4 × 10^-4 off, a key scaled to 2¹⁰ and 2²⁰ 1.5 × 10^-5, zero wrong after rounding at every scale; and the 12,345 × 6,789 program runs 20,371 instructions in 47,539 tokens, matching Python).
• The chip layers inside stock Qwen3.5 modules. The repo's test_rv_hf.py loads the compiled 11 layers into transformers' own Qwen3_5ForCausalLM (bf16, reference kernels) and checks the residual stream against its IR evaluator at every position. My first attempt grew to 7 GB of resident memory, more than the shared box allows, so I stopped it and queued it behind the heavy-job lock; at the time of writing it has not finished, so this slice is not verified by me. Still not run: the 9.4B model (it needs a 24 GB GPU and a driver, and the box has 15 GB of memory shared with other jobs), the benchmark tables, the 230,000-position replay, and the complete test_rv_core.py and test_rv_fuzz.py runs, which build the full circuit and walk long traces; I stopped them after about 100 tests of that file passed with none failing, because they were taking the CPU for most of an hour; the count is approximate, from the progress dots. The benchmark numbers remain reported.

What it shows

The useful part of Qwenputer is not that a language model can emulate a CPU; people have shown language models executing programs for a while. It is that Qwen3.5's hybrid architecture contains a memory that, with its gate and write strength pinned and its keys one-hot, is exact, addressable and fixed-size, and that the model's own layers can be rewired to use it without touching Qwen's weights (the headers and my 12 sampled comparisons agree). The price is also visible in the file: a checkpoint twice as wide, most of it zeros, and a driver that has to keep the two halves apart. Whether this is a route to useful in-model tools, or a very good argument about what linear attention can store, is the open question. What would settle it is a model that learns to use such a memory, not one that has it compiled in. The author's follow-up on hybrid models hiding reasoning in their RNN state asks a related question from the other side.

Reproducing it

The scripts and raw outputs are committed next to the article, in public/articles/qwenputer/: toy.py (the recurrence experiments and the register machine, which imports torch_recurrent_gated_delta_rule from transformers 5.20) with toy_results.json, trace_export.py with trace.json (the trace the widget steps through), density.py, exact.py and emb.py with their outputs (the row sampling and the comparison with base Qwen), and headers.json (every tensor's shape in both checkpoints). The 31 October re-runs against the repo (its tests, calc, and repo_crosscheck.py) are in public/articles/qwenputer/verify/, with RUNLOG.md listing every command. They need torch, transformers and requests, and about 15 minutes on a CPU, most of it waiting on range requests.

Sources

• Divyansh Agrawal, Qwenputer: a fully functioning, deterministic RISC-V CPU inside Qwen3.5, 22 October 2026, and the thread.
• The code, MIT licensed: divagr18/qwenputer, read at commit f07a0b2. The experimental/riscv README, docs/rv32.md and docs/q35.md are the design documents I quote.
• The model, Apache-2.0: divagr1925/Qwen3.5-4B-Qwenputer, compared with Qwen/Qwen3.5-4B.
• Schlag, Irie and Schmidhuber, Linear Transformers Are Secretly Fast Weight Programmers, ICML 2021.