# Oruk's looped speech model: 2.04B parameters folded into two blocks that run 52 times

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/oruk-looped-speech
> date: 2026-10-11
> tags: speech, asr, looped-transformers, distillation, lora, on-device

Nathan Roll, who runs [Oruk Labs](https://oruk.ai), posted a 20-second animation on October 10 with a claim that made me stop scrolling: "we made a speech model 13× smaller by making it repeat itself. 52 layers → 2 shared blocks. 2.04B parameters → 153M. The encoder loops 24 times. The decoder loops 28. Smaller weights, roughly the same compute."

I wanted to know three things. Which model has 52 layers and 2.04B parameters? What does it mean to "loop" a speech model, and how do you get there from a normal one? And who wins when you keep the compute and throw away 92% of the weights?

The source is thin, and I'll say that up front. There is the post, two follow-ups from Roll in the same thread, his replies to about twenty people, and the video. There are no weights, no paper, no blog post on oruk.ai (its research feed stops at October 1), and no accuracy numbers. Roll says so himself: "Research in progress. Accuracy results will follow." So this piece spends most of its words on the mechanism, which is well understood, and on arithmetic I can check, and ends with a list of what nobody outside Oruk can know yet.

## The teacher is Qwen3-ASR-1.7B, and its 2.04B hides a duplicate

The second post in the thread names it: "We distilled Qwen3-ASR-1.7B into two shared blocks using about 12,000 hours of English speech." The third explains the odd number: "The teacher's name says 1.7B; its full model has 2.04B parameters."

The Hugging Face page for [Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B) says 2.35B, so I read both safetensors headers to see who was right. The checkpoint holds 2,349,217,408 numbers. One tensor, `thinker.lm_head.weight`, is 151,936 × 2,048 and is the same shape as the token embedding; `config.json` says `tie_word_embeddings: true`. I pulled three rows of each by byte range (rows 0, 1,000 and 151,000) and they are byte-identical. The checkpoint stores the output head twice. Drop the copy and you get 2,038,052,480 unique parameters. Roll's 2.04B is right, and the Hub's 2.35B is right about the file.

| Part | Layers | Parameters |
|---|---|---|
| Audio front end: three stride-2 convolutions, projection, two output projections | | 15,168,128 |
| AuT audio encoder, width 1,024, FFN 4,096, 16 heads | 24 | 302,309,376 (12,596,224 each) |
| Qwen3-1.7B decoder, width 2,048, FFN 6,144, 16 query and 8 KV heads | 28 | 1,409,408,000 (50,336,000 each) |
| Token embedding, tied to the output head | | 311,164,928 |
| Final norm | | 2,048 |
| **Unique total** | **52** | **2,038,052,480** |

So the 52 layers are 24 encoder layers plus 28 language-model layers, and 2.04B over 153M is 13.3×, which is the "13×" in the post. Against the Hub's number it would read 15.4×. Oruk picked the honest denominator.

<ModelCard repo="Qwen/Qwen3-ASR-1.7B" note="The teacher. 2,349,217,408 numbers in bf16 across two shards, of which 2,038,052,480 are unique: lm_head is a byte-identical copy of the tied token embedding (measured from the headers). 24 AuT encoder layers plus 28 Qwen3-1.7B layers. Apache-2.0." />

<Figure
  src="https://ai.thesatyajit.com/articles/oruk-looped-speech/fig3.jpg"
  alt="Two diagrams. Left, the AuT model: an FBank input at 100 Hz goes through three downsampling Conv2d layers and 32 self-attention layers in the AuT encoder to a 12.5 Hz hidden sequence, which an 8-layer AuT decoder with cross-attention and self-attention turns into text. Right, Qwen3-ASR: a pretrained AuT encoder turns audio into embeddings, which sit in the input of the Qwen3 LM beside Qwen tokenizer embeddings of context entities and the prompt, and the LM writes 'Language Chinese asr_text' followed by the transcript."
  caption="Qwen3-ASR: the AuT encoder (pretrained on its own as an attention encoder-decoder, left) is cut at its 12.5 Hz hidden states and fed to a Qwen3 language model as tokens (right). The figure draws AuT with 32 encoder layers; the released 1.7B's config has 24 (Qwen3-ASR technical report, Figure 2; image from the Qwen3-ASR README)."
/>

The figure matters for one reason. Qwen3-ASR has no cross-attention. The [technical report](https://arxiv.org/abs/2601.21337) describes the encoder as conducting "8 times downsampling" to "a 12.5Hz token rate", and those audio tokens go into the language model's input sequence like any other tokens. The teacher's "decoder", the half Oruk folds into 28 passes, is a 28-layer chat-model stack doing next-token prediction over audio embeddings and text. Thirty seconds of audio is 375 audio tokens in front of the transcript.

For scale, the report puts the teacher at 1.63 and 3.38 WER on LibriSpeech test-clean and test-other, and 8.45 on GigaSpeech. The student has to be measured against those.

## What looping is

A transformer layer is a function from a sequence of vectors to a sequence of vectors of the same shape. That shape-preservation is what makes stacking possible, and it also means nothing stops you from feeding a layer its own output. A normal 28-layer decoder computes

$$
h_{28} = f_{28}(f_{27}(\cdots f_1(h_0)))
$$

with 28 different sets of weights. A looped decoder computes the same composition with one function:

$$
h_{28} = f(f(\cdots f(h_0)))
$$

Parameters drop by 28×. FLOPs do not drop at all, because each token still goes through 28 layer applications. The post's whole trade fits in its last line: "Smaller weights, roughly the same compute." The [looped Transformer explainer](/architectures/looped-transformer) on this site works through the cost model in detail; the one-line version is that stored parameters are paid once and FLOPs are paid per pass.

The idea is old. The [Universal Transformer](https://arxiv.org/abs/1807.03819) (Dehghani et al., 2018) looped the whole encoder and decoder, added an embedding of the step number to the input so the shared weights could tell which pass they were on, and let each position decide when to stop. [ALBERT](https://arxiv.org/abs/1909.11942) (Lan et al., 2019) made it a compression tool: share every layer of BERT and ALBERT-large has 18M parameters to BERT-large's 334M. ALBERT's own ablation is the most useful table on the subject. With all layers shared, ALBERT-base drops from 81.6 to 80.1 average on its downstream suite while going from 89M to 12M parameters. Sharing only attention costs nothing (81.7 at 64M). Sharing only the feed-forward layers costs most of the drop (80.2 at 38M). The FFN is where layers differ from each other, and it is also where most of the parameters live.

ALBERT also taught the lesson Roll's replies keep returning to. ALBERT-xxlarge has 235M parameters to BERT-large's 334M, a 4,096-wide hidden state, and its own table puts it at about 3 times slower to train through the data. Fewer parameters is a statement about storage.

## Converting a trained model into a looped one

Oruk did not train a looped model from scratch. They started from a trained 52-layer teacher and distilled. The closest published recipe is Google DeepMind's [Relaxed Recursive Transformers](https://arxiv.org/abs/2410.20672) (Bae et al., 2024), and the video suggests Oruk did something close to it.

<Figure
  src="https://ai.thesatyajit.com/articles/oruk-looped-speech/fig4.png"
  alt="Three stacks. A vanilla transformer with distinct layers 1 to N; a recursive transformer with one block of layers 1 to K looped N over K times; a relaxed recursive transformer with the same looped block, where every layer has a column of small hourglass-shaped LoRA modules numbered 1 to N over K, one per loop."
  caption="From a vanilla stack to a recursive one, and the relaxed version that gives every pass of every shared layer its own low-rank adapter (Relaxed Recursive Transformers paper, Figure 1)."
/>

The paper makes three moves. First, it initialises the shared block from the trained model's own layers instead of from random weights. For a block of K layers looped N/K times, "stepwise" picks K layers spread evenly through the stack, "average" averages the layers that will map onto each slot, and "lower" just keeps the first K. Then it uptrains briefly. Second, it relaxes the tie: each pass gets its own LoRA delta on every linear map, so pass $i$ computes with $W + B_i A_i$, where $W$ is shared and $B_i A_i$ is a rank-$r$ correction. The deltas are initialised from a truncated SVD of the difference between the original layer and the shared one, so pass $i$ starts as a low-rank approximation of the layer it replaced. Third, it distills from the full model. The result they lead with: a recursive Gemma 1B built from Gemma 2B scores up to 13.5 points higher on few-shot tasks than a non-recursive Gemma 1B trained from scratch.

<Figure
  src="https://ai.thesatyajit.com/articles/oruk-looped-speech/fig5.png"
  alt="Left, a six-layer model. Middle, three ways to initialise a two-pass looped block of three layers: stepwise uses layers 1, 3 and 6; average uses the means of layers 1 and 4, 2 and 5, 3 and 6; lower uses layers 1, 2 and 3. Right, the relaxed version with LoRA modules per pass, where truncated SVD of layer 1 minus the averaged layer 1-and-4 gives the B and A matrices."
  caption="Initialising a looped block from a trained model, and the per-pass LoRA deltas taken from an SVD of what sharing threw away (Relaxed Recursive Transformers paper, Figure 2)."
/>

Speech has a precedent too. Google's [Sharing Low Rank Conformer Weights](https://arxiv.org/abs/2303.08343) (2023) repeated and shared Conformer blocks, then low-rank-decomposed the shared parts, and got a 5M-parameter model to 2.84 and 2.94 WER on LibriSpeech dev-clean and test-clean, for always-on recognition on low-memory neural processors. The paper's phrase for it is "retain the full model in-memory while increasing the number of virtual transformations applied to the input." Oruk's pitch is the same idea at thirty times the size.

## What the video shows

The animation is mostly a counter rolling down and two rings filling. One frame is worth more than the text.

<Figure
  src="https://ai.thesatyajit.com/articles/oruk-looped-speech/fig2.jpg"
  alt="Two frames side by side. Left: the label PARAMETERS over 2,040M, with two thick translucent stacks of many thin sheets, one blue and one pink. Right: 153M in a blue-to-orange gradient, with two thin flat tiles, one blue and one orange, each with a wave drawn on it."
  caption="The opening of the announcement: two stacks of layers collapse into two tiles (Oruk Labs' announcement video on X)."
/>

<Figure
  src="https://ai.thesatyajit.com/articles/oruk-looped-speech/fig1.jpg"
  alt="Two concentric-line rings side by side. The blue left ring is labelled 24 over 24; the orange-purple right ring is labelled 28. Each ring has a small card at its top showing a boxed W, a plus sign, and a tall thin bar next to a short wide bar. Caption text: two shared blocks, 24 + 28 passes."
  caption="Each ring is one shared block; every completed lap is a pass. The card on each ring reads W plus a tall thin matrix and a short wide one (Oruk Labs' announcement video on X)."
/>

The card on each ring is the standard LoRA picture: a square $W$ plus a tall $d \times r$ matrix and a short $r \times d$ one. Nothing in the text says per-pass adapters, but that drawing is how everyone draws $W + BA$. My reading is that each pass gets its own low-rank delta, as in the relaxed recursive recipe. I can't confirm it.

Two replies add detail. Asked whether it is a looped transformer, Roll answered "looped *conformer*", so at least the encoder block has a Conformer's convolution module, which Qwen3-ASR's AuT encoder does not. And the open problem he names is "the arbitrary number of loop problem, which we're still figuring out how to positionally embed (like RoPE but for loop#)". It also fits the adapter reading. Per-pass LoRA ties the model to exactly 24 and 28 passes, because pass 29 has no adapter. A loop-index embedding, which is what the Universal Transformer used, would let you run a different number of passes at inference: fewer for an easy clip, more for a hard one.

## What 153M forces on the student

Here the arithmetic gets interesting. If the student's shared blocks have the teacher's widths, the encoder block is 12.6M and the decoder block is 50.3M. Add the 15.2M front end and you are at 78M before a single embedding row. Qwen's vocabulary at width 2,048 is 311M on its own. It does not fit in 153M, at any rank of adapter.

So something else changed, and there are only two ways out:

- Make the decoder narrower. With Qwen's 151,936-token vocabulary, the embedding alone passes 153M at a width of about 1,000, so the real width would have to be well under that. But at half the width, each decoder pass costs a quarter of the FLOPs, and "roughly the same compute" stops being true.
- Shrink the vocabulary. Keep width 2,048 and the arithmetic leaves room for at most about 36,600 tokens with no adapters at all. The student was distilled on English only, where a 32k-token vocabulary is ordinary. That route keeps the compute claim intact.

I think the second is more likely, because it is the only one consistent with both of Roll's numbers. But a Conformer encoder block is larger than AuT's, which squeezes the budget further, and I'm guessing at shapes Oruk hasn't published. The widget below lets you try combinations. With width 2,048, a 32,768-token vocabulary and rank-8 adapters on every pass, it lands at 157.5M, close enough to show the shape is plausible and far from proof that it is the one.

<LoopBudget />

## Where the memory saving pays, and where it doesn't

Roll is careful about this, more careful than most of the replies. "The benefit is a smaller model to store. Repeating the blocks still costs compute, so this isn't a speed claim." Asked whether the student was worse, he said 2-3%, and to the follow-up ("same inference compute but … more than 2% worse?") he answered "cynically yes", pointing at the 13× smaller memory footprint.

Footprint is real and easy to see. The teacher's unique weights are 4.08 GB in bf16 and 1.02 GB even at 4 bits. The student at 8 bits is about 153 MB. Someone asked whether it would run on a Raspberry Pi Zero 2 W, which has 512 MB of RAM; Roll said yes. The teacher cannot. A dictation app that keeps the model resident leaves the rest of RAM to the app you are dictating into, which another reply pointed out.

Bandwidth is subtler, and it is where I disagree with the most-liked reading of the post. Decoding one token in a memory-bound regime means streaming every weight that token touches out of DRAM. For the teacher's decoder that is 28 layers plus the output head, 1.72 billion parameters, or 1.72 GB per token at 8 bits. A phone with a 64-bit LPDDR5X-8533 bus peaks at about 68 GB/s, so the ceiling is roughly 40 tokens a second.

A looped decoder streams the same block 28 times per token. Unless that block stays on chip between passes, it is read from DRAM 28 times, and the traffic is the same 28 layer-reads the teacher pays. Looping saves bandwidth only if the shared block fits in on-chip memory: SRAM, a system-level cache, an NPU's local buffer. A teacher-width decoder block is 50.3 MB at 8 bits and 25.2 MB at 4 bits. Whether that fits depends on the chip, and Oruk hasn't said which chip. One reply came at the hardware from another side: NPUs "have lots of MACs but sucks at quants", so a model that is small without aggressive quantization suits them. Roll answered "EXACTLY". If the block does sit on chip, the per-token DRAM traffic collapses to the output head and the adapters, and the decode ceiling stops being memory and becomes arithmetic. It changes which hardware can run the model, and I think it is the most interesting thing about the project. It is also conditional on hardware Oruk hasn't named.

Two costs do not shrink at all. The KV cache: each of the 28 decoder passes attends over earlier tokens with keys and values computed from that pass's input, so the student caches 28 layers' worth per token, exactly like the teacher. (Mixture-of-Recursions, [arXiv 2507.10524](https://arxiv.org/abs/2507.10524), proposes reusing the first pass's keys and values to cut this; nothing says Oruk does.) And latency: the passes are sequential, so a decoded token waits for 28 layer applications, as it did before.

The encoder is a different regime entirely. It runs once per clip over all frames in parallel, which makes it compute-bound: about 227 GFLOPs of weight arithmetic for 30 seconds of audio at teacher width, whether those FLOPs come from 24 distinct layers or one layer 24 times. Looping saves the encoder nothing but storage.

## What "2-3% worse" could mean

Roll's estimate, mid-training, is that the student "will be 2-3% worse than the teacher." On LibriSpeech test-clean, where the teacher is at 1.63, a relative 2-3% is 1.66 to 1.68 WER, a rounding error. Two to three absolute points would be 3.63 to 4.63, a different class of model. On test-other the same readings give 3.45 to 3.48, or 5.38 to 6.38. I'd bet on relative, because ASR people usually talk about WER that way and because a 13× compression that loses only a relative 3% would be the headline. He also said the model currently ranks "~23 on openasr leaderboard". I couldn't check that: there's no Oruk checkpoint I could find, and the leaderboard only lists submitted models.

## What we don't know yet

Everything that would let you judge the result. No word error rate on any named test set. No architecture: block widths, vocabulary, whether the decoder is still a Qwen3 stack or a smaller decoder, whether the encoder is a FastConformer like the Parakeet model behind Oruk's open [Orukeet](https://huggingface.co/oruk/orukeet). No adapter rank, if there are adapters. No distillation recipe beyond "about 12,000 hours of English speech". Roll answered "Very similar!" to someone asking about a diffusion loss, which I can't interpret without more. No latency on any device. No release decision; one reply asked whether it will be open source and got no answer in the thread.

It's also worth knowing what the comparison is. A 153M English model distilled from a 52-language teacher is not doing the teacher's job, and a fair check would score it against a 150M-class English recognizer trained directly, not only against Qwen3-ASR-1.7B. I'd want that number before believing the looping, rather than the English-only narrowing, is what earns the result.

Looping is having a moment on this site: [SMELT](/articles/looped-transformers-matched-compute) shows it still wins at matched FLOPs, [IFM's ablations](/articles/looped-models-done-right) pull apart what makes looped models work, [LOOM](/articles/loom-looped-moe) gets past two loops, and [virtual logic depth](/articles/virtual-logic-depth) argues looping buys reasoning rather than knowledge. Those are all language models chasing quality per parameter. Oruk is the first I've seen take a trained speech model and fold it for footprint, and their last public experiment, [a fruit-fly speech classifier](/articles/fly-connectome-reservoir), shipped with the control that undid its headline. If this one ships with the same honesty, it will be worth reading. For now it is a well-chosen idea with no numbers.

## How I checked

I read the post, Roll's two follow-ups and the first page of replies through the fxtwitter mirror, and pulled frames from the 1080p video with ffmpeg; the counter values and the W-plus-low-rank card come from those frames. For the teacher I read `config.json` and the headers of both safetensors shards of `Qwen/Qwen3-ASR-1.7B` by HTTP range request, summed tensor shapes by group, and compared rows 0, 1,000 and 151,000 of `embed_tokens` and `lm_head` byte for byte. WER and architecture quotes are from the Qwen3-ASR technical report; ALBERT, Relaxed Recursive Transformers, the shared-Conformer paper and Mixture-of-Recursions numbers are from their arXiv pages. The 153M budget, the bandwidth ceilings and the encoder FLOPs are my arithmetic from the counted shapes, at two FLOPs per weight per token, and the widget does the same sums. I did not run any model, and Oruk's student is not public, so nothing about it is measured here.