Qwen4-27B: what a merged kernel PR does and does not leak

On 25 October a post with 83,000 views said that "Qwen4-27B architecture is being reverse-engineered from a merged kernel PR", and listed what the PR supposedly leaks: INT4 on one or two cards, MTP with four draft tokens, FP8 at TP2, four residual streams, QSA sparse attention. The post carries two images. One is the Qwen 4 logo. The other is a screenshot of the PR.

I read the screenshot, then went looking for the code behind it. Three things come out.

• The PR is real and merged, but into a downstream branch, not into vllm-project/vllm-xpu-kernels.
• Its visible text supports three of the post's five claims directly, one by inference, and says nothing about a fifth.
• The architecture it points at is already public, in vLLM, under the name Qwen4Exp. It is Qwen3.8-Flash-Next's. A dense 27B built on it would be new. A release is not.

Labels used throughout: measured means I read it in a file or ran it. Reasoned is my arithmetic on those. Unconfirmed means no public source says. I did not find a reported figure for the 27B at all, and that is a finding too.

What the screenshot says

expand

The title is "Enable Qwen3.8-flash-next 8 cards fp8 & int4 and optimize qwen4-27b". It merged 10 commits touching 56 files into analytics-zoo:downstream/v0.31.0, from a branch dated 2026-10-19. Those are measured from the image. My translation of the scope paragraph, with the model names as written:

SYCL and SYCL-TLA kernels for downstream v0.31.0: Qwen3.8-Flash-Next at TP4 INT4 and TP8 INT4/FP8, and the Qwen4-27B dense model at TP2 FP8 and TP1/TP2 INT4. Mainly covers M from 1 to 8, MTP draft=4 with batch 1, and large prefill.

So the post's list is a fair reading of the scope sentence. But look at the proportions. Most of the PR is for Flash-Next, the 125B mixture of experts with 512 experts, at four and eight cards. The 27B gets one clause. The post's headline, "you might not need a MoE cluster", is about the smaller half. Several replies said the same: a 27B is dense, so the sentence is a non sequitur.

Three more things the image shows:

• The kernels target Intel XPU through SYCL. One reply noticed that "1-2 cards" therefore means Intel GPUs. The text lists coverage, decode at M from 1 to 8 and draft 4, and gives no speed or INT4 quality number. (Measured: I read the whole visible page.)
• The model integration (the model class, metadata, scheduler, graphs) lives in a companion, "vLLM#51", of the same downstream. This PR only supplies the native kernels.
• The image is cut off partway down the page. There are 56 files and I see a table of eight rows. Anything below that I cannot see.

What I could not do. I could not open the PR. analytics-zoo is the name of Intel's older BigDL-era organisation, and the five public repositories a search returned for it hold none of this; the repository carrying downstream/v0.31.0 is not readable without credentials from my session. I cannot verify the author's affiliation either. The claim "Intel's vLLM XPU kernels merged this" is therefore unconfirmed as worded. What I can verify is below: the public repositories that sit next to it.

What the public code shows

Two public repositories carry related, readable evidence.

vllm-project/vllm-xpu-kernels (checked at daf6240, 26 October). Commit 8b95ea6, "Fix Qwen4Exp TP=2 kernel coverage and DP padding corruption (#635)", from Intel's hongbo.zhao, 14 October. It adds one row to the paged-decode table: query group 16, head size 256, page size 64. The comment says Qwen3.8-Flash-Next ("Qwen4Exp") at TP2 has a per-rank GQA ratio of 12, which falls in the 16 bucket. (Measured.) There is no Qwen4-27B anywhere in that repository, and no QSA kernel. Its public mHC kernels (csrc/xpu/mhc) were written for DeepSeek-V4; they carry HC = 4 and 20 Sinkhorn repeats.

One oddity, measured from the config: Flash-Next has 24 query heads over 2 KV heads, so its GQA ratio is 12 at TP1 as well, not "at most 8" as the commit message says. I cannot explain that gap. It does not change the fix.

vllm-project/vllm (6674eed, 26 October). It contains a package vllm/models/qwen4_exp/: hyper-connection ops, a QSA attention with its indexer and selection kernels, an MTP module, per-layer embeddings, and test files for each. It was added on 31 August by "Support Qwen3.8-Flash-Next (#53896)". Two facts matter here:

• Its __init__.py raises NotImplementedError("Qwen4Exp currently supports CUDA and ROCm only") on XPU. So XPU support for this architecture does not exist upstream. A downstream with its own model class (the PR's "vLLM#51") is exactly what you would expect. (Measured.)
• The architecture name is Qwen4ExpForConditionalGeneration, with model_type qwen4_exp. Alibaba's own released checkpoint of Flash-Next uses it. "Qwen4" in these file names is the architecture family, not a model release.

That second point is the one the post's headline hides. The PR title says qwen4-27b, a name from someone's branch. Public code says Qwen4Exp, and attaches it to a model already on the Hub.

The shapes, compared with what exists

I fetched config.json for both released models. The table sets those against the PR's text.

The three "reasoned" entries that carry the most weight come from the same trick. Two numbers in the PR text only make sense for specific shapes.

1. "Q12/KV2" at TP2. It is a per-rank shape, so the model has 24 query and 4 KV heads in total. Qwen3.8-27B has exactly that. Flash-Next at TP2 would be Q12/KV1, and that is the row the public commit added. The PR text then says the old "Q3 and Q6" paths are kept; 24 heads at TP8 and TP4 give 3 and 6 per rank, which are Flash-Next's TP8 and TP4 shapes. (Reasoned.)
2. "BA48/96". In a Gated DeltaNet layer the b and a projections are as wide as the number of value heads, so the pair is 96 at TP1 and 48 at TP2 if there are 48. Both released models have 48. (Reasoned.)
3. "H5120". The visible text names "H2560/H5120" for the hyper-connection and gate kernels. 2,560 is Flash-Next. 5,120 is the 27B's hidden size. (Measured as text; the identification is reasoned.)

Taken together: the dense model in the PR has the released Qwen3.8-27B's skeleton (hidden 5,120, 24 query heads over 4 KV heads, 48 GDN value heads) with Flash-Next's two new organs added, hyper-connections and QSA. The released 27B has neither (its config has no hc_count and no indexer_* fields). So "carried over" covers the body, and "new" covers the residual path and the attention selector.

One more leak the post misses: the PR's MoE rows say E512/K2560/N160 or 320. Those are Flash-Next's 512 experts and hidden 2,560, with the 640-wide expert split across 4 or 2 cards. It is a MoE kernel note, not 27B information. (Reasoned.)

The mechanism, briefly

Hyper-connections. An ordinary transformer has one residual stream. Qwen's Gated Residual keeps four, each H wide, so the residual is 4H per token: 20,480 numbers at H = 5,120. Before each sublayer a gated mix reads the four lanes down to one H-wide input. After it, a combine writes the sublayer's output back into all four, scaled per lane by 2 sigmoid(inj / 4), which is exactly 1 when the learned injection is 0. I ported vLLM's reference for these ops to numpy and ran them on CPU. The site's Flash-Next article explains why Qwen deleted the lane-mixing matrix, and xHC and GLM-5.3-Flash show the alternative that keeps it.

QSA. Sparse attention picks which past tokens to read, but the picker can cost more than the reading. QSA compresses keys into blocks of 4, scores a query against each block with relu(q . k) summed over the index heads, keeps the top 2,048 / 4 = 512 blocks, expands them back to 2,048 tokens and appends up to 3 tail tokens. The attention then reads 2,051 positions however long the context. Compare MiniMax's MSA, which also scores blocks of past KV but picks them per GQA group.

MTP. One draft layer, run recursively. vLLM's config for this family sets n_predict to 1, so "draft = 4" means that layer is replayed four times, and the verifier checks 5 positions per step. The config also has an index_share_for_mtp_iteration flag, which lets the draft iterations reuse the QSA selection instead of redoing it.

Running the references on CPU

I ported test_hc_ops.py and the selection and expansion references in test_qsa_reference.py to numpy (my port, not vLLM's code run) and ran them at tiny scale on a shared 4-core VM. These are measured, and they show the mechanism, not the model.

grouped rmsnorm per-stream rms: [1. 1. 1. 1.]
mix out shape (3, 64) vs residual width 256 -> ratio 4.0
injection=0 weight [1. 1. 1. 1.]
unit-injection: |out-res - blk| max 2.3841858e-07
injection logits [-8. -2.  2.  8.] -> write weights [0.238 0.755 1.245 1.762] (range 0..2)
selected blocks [1, 5, 10, 11]
expanded token idx [4, 5, 6, 7, 20, 21, 22, 23, 40, 41, 42, 43, 44, 45, 46, 47] width 16
needle token 21 selected: True  in block 5
dense attn mass on selected tokens: 0.414 | cos(sparse,dense)= 0.8649 | read 16 of 48

What the run shows. The mix is a plain 4:1 read. A zero injection writes the block output unchanged into every lane, and the weights range over 0 to 2, so a lane can mute or double what a layer writes. In the QSA toy (ratio 4, a budget of 4 blocks, 48 tokens, one needle planted in token 21) the compressed scores find the needle's block; the sparse read keeps 41% of the dense attention mass from a third of the positions. It is random data, so 0.865 is a statement about this toy, not about quality.

At the leaked constants the same code reads 2,051 tokens, which is 25.0% of an 8,192 context, 3.1% of 65,536, and 0.78% of 262,144. The scan to choose them covers 65,536 compressed blocks at the long end. (Reasoned from the config's budget and ratio.)

What the shapes cost to serve

Every number below is reasoned, from Qwen3.8-27B's config, since nothing public gives the real one.

• Parameters. The decoder's linear layers hold 24.35 billion and the embedding and LM head 1.27 billion each, 26.89 billion in all. The released checkpoint is 55,586,114,863 bytes including a vision tower and an MTP head; 26.89 billion at 2 bytes is 53.79 GB, so that is consistent. Hyper-connections add little: the gates are low-rank.
• INT4 weights. At 4.125 bits a weight (4 bits plus a 16-bit scale per group of 128), the decoder is 12.56 GB. With the embedding and head kept in BF16 the file is 17.64 GB. The plain "27B at half a byte" of 13.5 GB that replies quoted leaves those out. It fits a 24 GB card and does not fit a 12 GB one without squeezing the vocabulary tables, which this site's 12 GB build does.
• KV cache. Only the full-attention layers keep one: 16 layers of 4 KV heads, width 256, K and V, 2 bytes: 65,536 bytes a token (64 KiB), half that in FP8. At 262,144 tokens, 16 GiB in BF16 and 8 GiB in FP8. The 48 GDN layers hold a fixed 151 MB per sequence (FP32), and QSA's compressed index keys add 1 KiB a token, 268 MB at 262,144. Hyper-connections add no cache: the lanes exist only inside a layer.
• Decode ceiling. Decoding at batch 1 is bandwidth bound. Per step the card streams the decoder and the LM head: 15.10 GB at INT4. At 456 GB/s, the published figure for Intel's Arc B-series (I did not re-check it), that is 30.2 tokens a second, and 60.4 on two cards at TP2 before any all-reduce cost. With MTP verifying draft 4, the 3.3 tokens a step reported for Flash-Next would make about 100 and 199. Those are ceilings, not predictions: no acceptance rate, speed or quality number for the 27B is public, which is the reply's point.
• Why QSA matters here. Dense attention at 262,144 would add 17.2 GB of KV reads a step and cut the ceiling from 30 to 14 tokens a second. QSA reads 2,051 tokens plus the index, 0.40 GB, and holds it near 29.4.

For the other formats: FP8 weights are 29.4 GB, too big for one 24 GB card, so the PR's FP8 target is two cards at TP2, with a ceiling of 33.9 tokens a second before communication. BF16 is 53.8 GB, with a ceiling of 8.9 on a single card that could hold it.

Post versus evidence

What a careful reader should take from this

A kernel PR is weak evidence about a model's release, and several replies said so. The useful reading is narrower. People at an inference vendor have working kernels, in the first week of a downstream branch, for a dense model with Flash-Next's residual path and sparse attention at the released 27B's width. That is a real engineering signal, and the strongest I could extract is the Q12/KV2 shape. It says nothing about weights, benchmarks, a name or a date. Qwen4 is not released, and Alibaba has not confirmed any of this.

What I could not check

• The PR itself. I read a screenshot, and only its top.
• Whether "Qwen4-27B" is a final name or one engineer's branch label.
• The dense model's layer count, lane count, vocabulary and QSA constants.
• Any speed, acceptance rate or INT4 accuracy for it.
• Intel's Arc bandwidth figure, which I took as published.