WeVisOne-VL-4B: boxes as tokens, mattes from a 9.6M decoder

mdjsonmcpcopy↗ai

2026-10-31 · 18 min · vision-language-models · object-detection · segmentation · image-matting · grounding · tencent

Why read this

Essential· top 10%

Weights counted against Qwen3-VL-4B, decoder tensors read, the engine's second pass found, vision tower timed on CPU. Full model not run.

• Analysis found nowhere else
• Interactive explanations
• Runs on a consumer GPU

·Vision & multimodal·Apache-2.0·Practitioner model

How this was scored

Is it new?

2 of 3: A real new idea, method or capability

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

2 of 3: Open code or weights with real limits

Will I understand it?

3 of 3: Mechanism carried by interactives built from real code or data

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

1 of 3: Relevant for months

Does it affect many?

2 of 3: A widely used model, tool or lab release

Only here?

3 of 3: The only place this analysis exists

Score 76 of 100, ranked 56 of 666 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

The post that sent me here says WeVisOne-VL-4B is "one model for points, boxes, masks, and alpha mattes" with "a lightweight 9.6M pixel decoder" that works "without relying on SAM". That is a specific architecture claim, and most of it can be checked from files: a header, a config, a tokenizer, a 38 MB checkpoint and 2,300 lines of Python.

So I did that. I read the paper (arXiv 2610.31329, CC BY 4.0), the model card, the released Tencent/WeVisOne-VL repository at commit 9a6bcd8, the safetensors header, the tokenizer and the decoder checkpoint. I compared tensors against both Qwen3-VL-4B releases. I ran the vision tower on CPU. I did not run the full model, and I did not run Tencent's inference package: it is downloaded code, and the 4.8-billion-parameter model does not fit the memory this machine allows.

Every claim carries a label. Measured means I counted or timed it myself from the files. Read means I read it in the released code or the paper. Reported means Tencent says it and I did not reproduce it. Reasoned means I derived it from the numbers.

What changed from Qwen3-VL-4B

Qwen3-VL-4B-Instruct has 4,437,815,808 values. WeVisOne-VL-4B has 4,830,581,248 (measured from the two headers). The gap is 392,765,440 values, and it has an exact explanation.

Qwen3-VL-4B ties its output head to its embedding table, so the two share one 151,936 x 2,560 matrix. WeVisOne-VL-4B has a vocabulary of 152,680 rows and unties them: a separate lm_head.weight of 152,680 x 2,560 sits beside the embedding. Untying costs one extra matrix of 388,956,160 values, and the 744 new rows cost 744 x 2,560 x 2 = 3,809,280 more. The two add to 392,765,440 (measured, arithmetic checked against both headers). Every transformer and vision tensor has the same shape as in the base: 36 text layers of 100,930,816 values each, hidden size 2,560, a 24-layer vision tower of hidden size 1,024.

It does not follow that nothing else changed. I streamed six tensors from the WeVisOne file and from both Qwen3-VL-4B releases by range request and compared them (measured). Every one differs from both bases, including a vision-tower tensor, which fits the paper's second training stage unfreezing the vision encoder. On five of the six tensors the WeVisOne tensor is closer to Instruct than to Thinking: relative L2 distances of 0.016 to 0.074 against 0.021 to 0.093. The sixth, the final norm weight, is a tie. That is consistent with Instruct as the starting point. It is not proof, and the paper only says "Qwen3-VL".

The decoder is not in the safetensors file at all. It ships as mask_decoder.pt, a PyTorch pickle. Loading an unknown pickle is how code gets executed, so I did not load it. I read its opcode stream instead: the only three names it imports are the tensor rebuild helper, FloatStorage and OrderedDict (measured). With a reader that accepts only those three names and builds inert records, the tensor table reads out: 296 tensors and 9,609,764 values in fp32, matching the repository's checkpoint_contract exactly.

The interface is tokens

The model has no detection head. Boxes, points and "nothing here" are text the language model writes. The tokenizer shows how: ids 151,669 to 152,669 are 1,001 new tokens, <0> to <1000>, one per normalised coordinate bin, and ids 152,670 to 152,678 are nine tag tokens: <ref>, </ref>, <box>, </box>, <point>, </point>, <polygon>, </polygon> and <OBJ> (measured from tokenizer.json). That is 1,010 new tokens in all. "None", which the model writes when the thing you asked for is absent, is an ordinary existing token, id 4,064.

expand

The released prompts are short fixed strings. Detection is "Locate all the instances that match the following description:" followed by the labels; pointing is "point to:"; matting adds "and provide the alpha mattes:" (read, prompts.py). The answer in the matting case is the one in the figure: for each instance, four coordinates and then <OBJ>.

Two consequences follow from this being an ordinary token stream.

Coordinates are quantised to a thousandth of the image side (read). On a 1,920-pixel-wide photo a step is 1.92 pixels, and a box can be off by half of that from rounding alone. On a 4,096-pixel-wide image the step is 4.1 pixels. The visual-prompt helper in the released code builds reference boxes with int(value * 1000), which truncates rather than rounds, so those boxes are biased toward the top left by half a bin on average (read; a small effect, and one a user will never see).

The answer length scales with what is in the picture. A box costs five tokens, a segmentation instance six, a point three, plus four tokens of tags (reasoned from the format above). A crowded scene with 500 boxes is about 2,500 tokens of decoding, which the paper handles in training by capping examples at 50 boxes in the first stage and 500 in the second.

The decoder: 9.6 million values that read the vision tower

Paper Figure 4 draws the decoder. The released config and checkpoint say how big each piece is.

expand

From the config and the tensor table (measured, mask_decoder.pt):

• Visual feature adapter, 1,794,432 values. Reads the ViT's features before its 2 x 2 merge, at four layers: 5, 11, 17 and 23 (vision_feature_layers in the config). The paper says "intermediate" layers; the config includes the last one.
• Query builder, 578,880 values. Turns an <OBJ> hidden state (2,560 wide) and its box (Fourier-encoded, 32 frequencies) into a query of width 192.
• Query decoder, 3,560,832 values. Six deformable blocks, 8 heads, feed-forward width 768. The largest part, 37% of the decoder.
• Pixel fusion, 3,184,896 values. Merges the four levels top to bottom, plus a two-layer CNN on the original photo resized to a fixed 1,024 x 1,024 canvas.
• Dynamic mask head, 49,601 values. An MLP makes a per-instance kernel that is multiplied against the pixel features.
• Alpha cascade (highres_refine), 441,123 values. Three stages at a quarter, a half and the full decoder canvas, with 48 channels each. This is the part that makes the matte "full resolution", and it is 4.6% of the decoder.

"Full resolution" needs a footnote. The cascade's last stage is the decoder's own 1,024 x 1,024 canvas, set by cnn_image_size, and the engine then resizes the sigmoid of those logits bilinearly to the photo's height and width (read, engine.py). On a photo larger than 1,024 pixels the matte is therefore upsampled from the decoder canvas, not computed at the photo's resolution. The model card's "stride 1" is stride 1 of that canvas.

Masks cost a second pass over the whole model

This is the part the paper does not draw. In the paper the decoder takes "the token's hidden state, its predicted box, and multi-level features from the Qwen3-VL vision encoder". In the released engine, those are gathered by running the model a second time.

predict first calls generate_text. If the answer contains <OBJ>, it calls decode_masks, which builds a new conversation with the finished answer as the assistant turn, feeds it through the model with no KV cache, captures the last hidden state at each <OBJ> position and the pre-merge vision features, and only then runs the decoder (read, engine.py lines 435 to 518). So a mask request pays for the vision tower twice and for a full forward pass over the image tokens, the prompt and the whole answer a second time.

I am not saying this is wasteful by design: a teacher-forced pass is the simplest way to get hidden states at positions the model itself chose, and it keeps the inference package short. But it is how this release is wired, not something the method requires. The decoder could read each <OBJ> state during generation.

What a request costs

The cost of a request has four terms: the vision tower, the language model's prefill over the image tokens, one decode step per answer token, and the pixel decoder per mask.

Image tokens. The vision tower cuts the image into 16 x 16 patches and the 2 x 2 merge leaves one language-model token per 32 x 32 pixels, so a square image of side S costs (S/32)^2 tokens. The engine's defaults allow 16 to 2,025 tokens, about 2.07 million pixels (read, engine.py); the paper trains at 16 to 512 tokens first and 1,960 later (reported).

KV cache. 36 layers, 8 KV heads of 128 dimensions, K and V, two bytes each: 147,456 bytes, 144 KiB per token (measured from the config). A 1,960-token image holds about 276 MiB of cache before the answer.

The vision tower, measured. I streamed the 315 vision tensors, 415,347,712 values (188 seconds, range requests), and ran Qwen's native vision-tower code on CPU in bf16 on synthetic images of shapes and circles. The machine was shared and heavily loaded, so the seconds are a ceiling, not a benchmark; read the FLOPs.

PyTorch's FLOP counter does not count the CPU attention kernel. I found that because my formula came out higher than the counter by exactly 4 x N^2 x 1,024 x 24 at every size (60.4, 305.9 and 966.8 GFLOPs), which is the attention matrix products in all 24 layers. Counted plus attention equals the formula to within 0.01% at all three sizes, so the part of the formula I could not otherwise check, the projections, the MLPs, the patch embedding and the four mergers, is right. The attention term grows with the square of the patch count: at 3,136 patches it is a third of the tower's compute.

The shapes are also what the decoder reads. At 896 pixels the language model sees 784 image tokens, but each of the four hooked layers (5, 11, 17 and 23) gives 3,136 x 1,024 features, four times denser, and the three DeepStack maps that go into the language model are 784 x 2,560 each (measured). The decoder's visual memory is therefore built from a grid four times finer than the one the language model reads.

The pixel decoder, reasoned. From the convolution shapes in the checkpoint, one instance costs about 16.8 GFLOPs at a quarter of the canvas, 70.0 at a half and 197.0 at full canvas, plus 3.6 for the semantic layer: 287.5 GFLOPs per instance (reasoned, 2 x out x in x k^2 x pixels). The language model's prefill is about 7.27 GFLOPs per token (2 x 3,633,509,376). So one mask instance costs the decoder about as much as prefilling 40 tokens, and 100 instances cost about 29 TFLOPs, more than the vision tower and the language model's prefill combined at moderate image sizes.

That is the practical reading of "9.6 million parameters": the parameter count is 0.2% of the model, the compute per instance is that of a small network at high resolution, and it is paid once per mask. For a handful of objects it is noise. For a crowded scene it is the bill. The engine batches 16 instances at a time (read), and the 48-channel maps at 1,024 x 1,024 are 201 MB per instance per tensor in fp32 (reasoned), so memory, not arithmetic, is probably what sets the batch size.

Decode. Every answer token reads the language model's weights once. In bf16 that is about 8.05 GB per token, so a bandwidth-bound decode on a 1 TB/s card is about 8 ms a token, and a 600-token answer (100 matting instances) is about 5 seconds (reasoned; I did not time a decode).

How it was trained

All of this is reported from the paper; I could not check it.

The claim "without relying on SAM" is true of inference, and only of inference. The paper's data engine is built from SAM-family and other off-the-shelf models, and the decoder learns to imitate them.

expand

Boxes come from SAM 3 and a DINO-style in-house detector, merged and then checked by Qwen3-VL. Each retained instance gets a binary mask from SAM 2. Matte labels come from MatAnyone2, with BiRefNet candidates accepted only if they pass overlap tests, and a trained "matting annotator" (a frozen SAM2.1-Base+ with a new alpha refinement head) re-annotates the whole corpus (all read from the paper's sections 2.2 to 2.6). So the 9.6M decoder is distilled from a stack that includes SAM 2 and SAM 3. What changed is that you no longer have to run them at inference. A decoder trained on those labels cannot be better than the labels, and the paper's manual spot-checks are of a subset only.

• Data engine summary. Concept discovery, targeted image acquisition, box and referring annotation, masks, matting, hard negatives, and a loop from training errors back to data.
• Stage 1, task alignment. 90 million examples, 13,000 steps, vision encoder and projector frozen, 16 to 512 visual tokens, at most 50 boxes per category to avoid repetition collapse.
• Stage 2, resolution and density. About 40 million examples, 9,000 steps, everything unfrozen, up to 1,960 tokens and up to 500 boxes per answer.
• Stage 3, tokens and pixels. The decoder warms up on 250 million segmentation samples for 100,000 steps; its weights and the <OBJ> embedding are then attached to the stage-2 checkpoint and trained 9,000 steps alternating text and pixel losses 1:1; a final 100,000 steps on 130 million matting samples train the decoder alone with the language model frozen.
• Reinforcement learning. GRPO on 178,000 examples picked for high reward variance, with a reward built from Hungarian-matched box IoU, cubed, that penalises misses and duplicates. For segmentation and matting, RL uses only box rewards: no mask or matte is scored. The reward for an absent target is 1 only if the model outputs nothing.

The last point matters for reading the results. The matte quality comes from supervised decoder training; RL improves where the boxes land.

What it reports

Every number below is from the paper's tables; none is mine. I chose the comparisons that show the pattern and the ones that complicate it.

The post's headline figures are all there: 71.3 is Dense200's mean F1, 89.6 is PixMo-Count's test accuracy, 71.2 is ReasonSeg's validation gIoU and 96.4 is the rejection rate on WeRef-Bench. Counting each dataset once however many tasks use it (RefCOCOg appears in both referring detection and segmentation, and the four detection sets are reused for pointing), I count 31 distinct benchmarks across the paper's Tables 2 to 8, consistent with "over 30".

Four things to keep in mind when reading them.

1. WeRef-Bench is the authors' own. It has 2,880 English and Chinese queries (reported), written by the group, and it is where the 96.4% comes from. The public absent-target benchmarks agree on direction (RefCOCOg-Rej 67.4 against 22.5 for the best other VLM), but not on size.
2. The metric is F1, not AP. Generative models have no confidence scores, so the paper follows Rex-Omni in reporting F1 at an IoU threshold (read). The strict IoU 0.95 numbers are small for everyone: 19.5 for WeVisOne on COCO, 3.9 on VisDrone.
3. The 4B is not uniformly better than the 2B. On box-prompted matting the 2B has the lower error on two of three datasets in mean absolute difference (6.6 against 7.3 on P3M-500-NP, 5.3 against 6.1 on PPM-100; lower is better). On Ref-ZOM's cumulative IoU the 2B is higher too, 73.4 against 72.2. The 4B's clearest margins are in crowded counting and in rejecting absent targets.
4. The matting gaps depend on what is being compared. On the authors' own WeRefMat set with category prompts, SAM2-Matting with the SAM 3 variant scores 70.8 mean absolute difference against 16.4 for the 4B (both multiplied by 1,000; lower is better). That baseline has to find the object from a category word, which is not what it was built for. On the public box-prompted sets the gaps are far smaller, and on two of three the 2B beats the 4B. I would trust the box-prompted comparison more (my judgment).

For the site's other work on promptable segmentation, see SAM 3 as a segmentation skill and SAM 3.1's object multiplex; for another recent open document and text reader built on a different design, Unlimited-OCR-2.

What the paper itself says it cannot do

The limitations section is short and plain (reported). The model was trained on a narrow perception mix, so "the limited amount of general-purpose training data may compromise the general capabilities relative to the base model". It produces structured output "without chain-of-thought", which limits harder reasoning. It is single-turn. And benchmark prompts are tidy while "real-world user instructions are often vague and hesitant". The model card adds that the pixel decoder rejects video grids and that fine alpha boundaries are sensitive to resolution, occlusion and prompt phrasing.

What I did not do

• I did not run the full model. At 4.83 billion parameters it needs about 5 GB even in 8-bit, above the memory this session allows, and its pixel decoder is run through Tencent's inference package, which is downloaded code.
• I did not run the decoder. Its shapes and parameter counts are measured; its FLOPs are reasoned and untimed.
• I did not reproduce any benchmark. Every accuracy figure is Tencent's.
• I compared six tensors, not the whole checkpoint, so "closer to Instruct" is a sample.

What would settle the open questions is easy to name: run the released package on a 24 GB GPU, time a request with 1 instance and with 100, and see whether the second pass and the decoder cost what the arithmetic above says.

Sources

• The scripts and raw outputs behind the measured numbers are in public/articles/wevisone-vl-4b/code/ in this site's repository: read_decoder_table.py (reads the decoder checkpoint's tensor table without executing it), cascade_flops.py, vit_run.py with its results, the six-tensor comparison and the decoder's tensor shapes.

• Paper: WeVisOne-VL: Unified Visual Perception from Objects to Pixels, arXiv 2610.31329v2, CC BY 4.0. Figures 3 and 4 are reproduced here with attribution.

• Model card, config.json, tokenizer.json, model.safetensors header and mask_decoder.pt: tencent/WeVisOne-VL-4B.

• Code: Tencent/WeVisOne-VL, commit 9a6bcd8.

• Base models for the comparison: Qwen3-VL-4B-Instruct and Qwen3-VL-4B-Thinking.

• The launch post: ModelScope on X.