LensVLM-9B: the 10.1× is how hard it squeezes, not how much it wins by

mdjsonmcpcopy↗ai

2026-09-26 · 17 min · vision-language-models · long-context · retrieval · multimodal · ocr · open-weights · explainer

Apple put LensVLM-9B on Hugging Face on 21 September, and the code on GitHub a day later. The reposts describe it as "a Qwen3.5-9B finetune that turns long documents into small page images to save tokens, then pulls up the full text of only the pages relevant to your question", and add that it "beats RAG-style baselines up to 10.1x".

The first is a good summary. The second reads like a margin. It is not one. 10.1× is a compression ratio, and at that ratio the margin over the best retriever is zero.

Here is what I read: the model card, config.json, processor_config.json, NOTICE and LICENSE; the safetensors header over HTTP range requests, plus the headers of both Qwen3.5 checkpoints it could have come from; the paper, arXiv 2605.07019 (v1, 7 May 2026); and Apple's code at commit 10709a7. I did not run the model. Measured below means I computed it from a file I read. Reported means the paper's number, not re-run. Reasoned means my inference from those.

Text as pictures, and why that compresses

The premise is older than this paper. PIXEL trained a language model on rendered text in 2022. DeepSeek-OCR framed it as "compressing long contexts via optical 2D mapping". It reported 97% OCR precision below 10× compression and about 60% at 20×. DeepSeek-OCR is the backbone Jina-OCR-v1 inherits. Glyph rendered long inputs for a VLM and ran a genetic search over rendering settings to hold 3-4× compression.

The trick works because an image encoder charges for area, not for words. Qwen3.5's encoder cuts an image into 16-pixel patches and merges them 2×2, so a page of $H \times W$ pixels costs

visual tokens, whatever is printed on it. Print smaller and the cost per word falls. The paper's input compression rate is the ratio of the document's $N$ text tokens to the visual tokens of its $K$ pages:

There are three presets. Apple's rendering_config.py carries the same numbers as the paper's Table 7 (measured). Black DejaVu Sans on white:

The paper's worked example: a 10,000-token document becomes 25 pages × 72 = 1,800 visual tokens at 5× (5.6×), 19 × 48 = 912 at 10× (11.0×), and 27 × 24 = 648 at 15× (15.4×). I redid the arithmetic and it holds. The page count barely moves, 19 to 27. The compression comes from each page getting cheaper, 72 tokens down to 24.

The cost is legibility.

expand

The paper says it plainly: "Past a certain compression rate, the pixels simply do not contain the bits." Averaged over seven benchmarks, Qwen3.5-9B-Base answering from the compressed pages scores 31.3% at 5×, against 72.4% from the full text. Glyph, which was trained for exactly this job, gets 47.2% (reported).

The escape hatch: read_page

LensVLM stops asking the model to read the thumbnails. It asks the model to find the page, then hands it that page's text. The paper calls the tool Expand. The released code calls it read_page.

expand

The loop, as lensvlm/prompts.py and lensvlm/evaluate.py run it (measured):

1. The renderer word-wraps the document and cuts it into pages. Each page gets its number drawn in the bottom-right corner in a 10-pixel font, so the index stays legible when the body text is not.
2. The user turn is every page image, then the question. The system prompt describes one tool and says: "Do not read the entire document — only read pages that are likely relevant."
3. The model thinks, then emits a call. The harness appends the page's exact source text as a <tool_response>.
4. One page per call, up to six turns, then an answer. Greedy decoding, 2,048 tokens a turn.

<tool_call>{"name": "read_page", "arguments": {"page": 10}}</tool_call>

The README's demo shows the whole behaviour in one trace: 15 page thumbnails, a thought that page 10 mentions an "American actress, singer", one read_page call, and the answer "Chief of Protocol of the United States". Trajectories average 1.3 calls. One call reaches 60.8% accuracy and a second adds 6.1 points (reported).

Why should a model that cannot read a page be able to pick it? Picking one of $K$ pages is a much smaller target than producing an answer string. The paper frames that with the data-processing inequality, then admits it is not enough. Untrained, the base model picks a gold page 25.8% of the time at 5× and 18.0% at 10×. Selection has to be learned. After training it reaches 76.8%, 71.0% and 52.1% at the three presets (reported, Table 13).

The recipe, all reported:

• Data. NQ, HotpotQA, MuSiQue and HELMET, padded with distractor passages to 3K-32K tokens and rendered at all three presets: 143,145 samples. The base model answers each one straight from the images, and the 105,202 it gets wrong (73.5%) become the training set.
• Traces. Qwen3.5-397B writes a tool-use trajectory for each hard sample. It is handed the question, the evidence pages uncompressed, and the gold answer. It cannot fail at reading, because it is never asked to read. Its job is to write a plausible search.
• SFT, then RL. LLaMA-Factory at a learning rate of 5e-6 for 2 epochs on 80% of the hard set. Then DAPO on 15K samples, 16 rollouts per prompt, with the vision encoder and projector frozen throughout. The reward is $R = 0.7c + 0.3cu$, where $c$ is answer correctness from an LLM judge and $u$ says the tool was used. The tool bonus only pays on a correct answer.

SFT does most of the work, 31.3% to 63.7% at 5×. RL adds 5.2 points. The attention analysis shows what changed. After training, the model gives 4-6 points less attention to distractor pages and 15-16 points more to the text read_page returned.

The token arithmetic

Every call puts one page of text back into the context, 405, 540 or 378 tokens. So the paper also reports an effective compression rate, which counts everything the reader actually processes:

$T_{\text{reader}}$ is the page tokens plus whatever read_page returned. The effective rate is always below the input rate, and further below it for short documents, where a call is a bigger share.

Take the 10,000-token document at the 15× preset. The pages cost 648 tokens. One read_page call adds 378 more, so effective compression falls from 15.4× to 9.7×. Two calls take it to 7.1×. That is how a "15×" preset turns into the paper's 10.1× (reasoned). The widget also sizes a top-k retriever the way the paper does, reading $\lceil K / C_{\text{in}} \rceil$ page-sized chunks. At every preset that retriever reads fewer tokens than LensVLM does.

The checkbox covers a trap in the processor config. Qwen's stock preprocessor_config.json sets a floor of 65,536 pixels. transformers' smart_resize rounds each side to a multiple of 32 and scales up any image under the floor. A 15× page is 128 × 190. Under the stock floor it becomes 224 × 320 and costs 70 tokens instead of 24, so the preset compresses 5.4× a page instead of 15.75×. The 10× page goes from 48 tokens to 70 the same way (reasoned from the resize rule). Apple's processor_config.json sets min_pixels to 1, and its code passes min_pixels=1 to vLLM as well. Eleven quantizations and re-uploads appeared on the Hub within three days. I have not checked what each one's image preprocessing does, and that is the first thing I would check.

What "10.1×" measures

Here is the abstract: LensVLM "outperforms retrieval-based, text- and visual-compression baselines up to 10.1× effective compression across seven text QA benchmarks". 10.1× is the effective compression LensVLM reaches at the 15× preset. "Up to" is the edge of what LensVLM was tested at. It is not an accuracy or speed multiplier. At that setting LensVLM averages 52.1%, twenty points below the full text.

The averages, reported from Table 11, each with the ECR it was measured at:

Three readings.

Against compression baselines the claim holds with room to spare. Glyph, LLMLingua-2 and plain page reading trail by 19 to 41 points.

Against text retrieval it holds at 5× and 10×, then runs out. LensVLM leads by 3.8 and 4.2 points at the first two presets, then ties at 15×. And the tie is not at equal budgets: the retrievers ran at 11.9×, so LensVLM read about 18% more tokens. Reasoned: if I interpolate linearly between each retriever's 8.3× and 11.9× points, a retriever given LensVLM's 10.1× budget lands between 53.3 (BM25) and 55.0 (Jina-v4). That is ahead of 52.1. Interpolating between two points is a guess, but it is the only comparison at matched budget the paper's numbers allow.

The headline figure leaves out the strong retrievers. Figure 1 plots ColPali as the only retriever, and ColPali is the weakest of the six at every preset. The text retrievers are in Tables 6 and 11.

expand

The average also hides where the margin comes from. Pick a benchmark:

MuSiQue carries it. That benchmark needs two to four hops, and LensVLM beats the best retriever there by 10.6, 14.2 and 7.3 points across the presets. That is the mechanism working as designed. A top-k retriever has to fit every hop's page into $k$ slots before it reads anything. LensVLM reads page 2, learns a name, then goes looking for page 5. On single-evidence tasks the picture turns over. The best retriever beats LensVLM on RULER at every preset, by 2.9, 4.9 and 1.8 points. It wins Qasper at every preset too. At 15× on HELMET it is 69.0 against 57.2.

Then there is the split between benchmarks. NQ, HotpotQA, MuSiQue and HELMET are in-domain: LensVLM trained on their training splits, and the retrievers are zero-shot on all seven. On the three out-of-domain benchmarks alone, LensVLM leads the best retriever by 2.0 points at 5× (58.8 to 56.8) and 1.9 at 10× (50.1 to 48.2). At 15× it trails BM25, 41.0 to 41.6 (measured: my averages of Table 11's columns). The paper does state that retriever ECR leaves out indexing, and that embedding retrievers need a GPU pass over the whole corpus. But BM25 indexes on a CPU too. And LensVLM pushes every page of every document through its vision encoder on every query.

The files

Parameters (measured). The header sums to 9,409,813,744, which matches the Hub's figure:

That is the same count this site measured for Qwen-Image-2.1's prompt rewriter, another Qwen3.5 9B fine-tune. Both Qwen3.5-9B checkpoints on the Hub hold 9,653,104,368. The difference is 243,290,624 parameters of mtp.* tensors, the multi-token-prediction head. LensVLM does not ship it, although its config.json still says mtp_num_hidden_layers: 1. So there is no MTP speculative decoding from this checkpoint (reasoned). The 48 tensors Qwen stores in FP32 are BF16 here. The architecture is unchanged: 32 layers, 24 of them Gated DeltaNet linear attention and 8 full attention; hidden size 4096; 4 KV heads of dimension 256.

Which base (measured). The card's base_model and the NOTICE say Qwen/Qwen3.5-9B. The paper says Qwen3.5-9B-Base. The paper also says the vision encoder and projector were frozen through SFT and RL, so those tensors should be byte-identical to whichever checkpoint training started from. I range-read them from all three repositories and hashed them. The 216 norms and biases of the 27 ViT blocks are identical in all three, so they cannot settle it. Seven vision tensors do differ between the two Qwen checkpoints: the six of the projector and the position embedding. All seven of LensVLM's match Qwen3.5-9B-Base by SHA-256, and none match Qwen3.5-9B. The language model agrees: in all six of its tensors I sampled that training had changed, LensVLM sits closer to the Base checkpoint. For the final norm the relative distance is 3.5e-4 from Base and 5.3e-4 from the instruct model. What does come from the instruct model is chat_template.jinja, which is byte-identical to Qwen3.5-9B's. The Base repo ships no template. That is probably why the metadata names the instruct model. The licence and architecture are the same either way. But the thinking and tool-call habits were taught by Apple's traces, on a base model.

Code (measured). Commit 10709a7 has the renderer, the read_page loop on vLLM, an LLM-judge evaluator and data preparation for NQ, HotpotQA and MuSiQue. It has no training code, no synthetic traces, no baseline implementations, and nothing that computes ECR. You can re-run LensVLM's accuracy on three of the seven benchmarks. You cannot re-run the comparison the headline claim rests on.

Licence (measured). The Apple Machine Learning Research Model License grants use "exclusively for Research Purposes". It defines those as non-commercial and excludes "use in any commercial product or service". Fine-tunes and quantizations inherit that restriction. The NOTICE says putting these terms on an Apache-2.0 derivative is "permitted by Section 4". Open weights, then, but not open source.

What it costs

A token of context costs 32 KiB of KV cache on this backbone: 8 full-attention layers × 4 KV heads × 256 dimensions × key and value × 2 bytes (measured from config.json). The linear-attention layers keep a fixed-size state instead. The paper measured peak KV at the 15× preset, on vLLM across eight B200s (reported):

I recomputed the MiB from the token counts at 32 KiB each, and they match. As ratios, those runs are 4.7× and 6.3× smaller contexts, not 10.1×. Of the 2,288 tokens, 480 are page images (20 × 24). The rest is the system prompt, the question, the tool calls, the returned text and the model's own reasoning. ECR leaves the reasoning out. The KV cache does not.

Latency goes the other way. With the typical single call, LensVLM runs two sequential turns: about 17 s, against about 8 s for one turn over the full text, at batch size 1 on a B200 (reported). The paper is candid about it: "LensVLM targets accuracy recovery under lossy visual compression, not inference speed".

Where it breaks

• Length. Every evaluation document is under 32K text tokens. RULER's 128K contexts were cut down to fit, and Apple's load_model defaults to a 32,768-token window. Nothing here shows how it behaves at 200K, where compression would matter most.
• Selection at 15×. 52.1% of trajectories open at least one gold page. The rest answer without ever opening one.
• Scale. The same recipe at 2B and 4B reaches 45.5% and 56.3%, in the table where the 9B scores 68.9%. Only the 9B is released.
• Code. Zero-shot on RepoQA and CodeQueries it scores 38.1, 26.1 and 20.0 at the three presets, against 65.6 from the text.
• Native documents. On MMLongBench-Doc the paper trains separate OCR-tool and zoom-tool variants. Neither is on the Hub. At 5× the base model reading compressed pages (51.2) beats both of them (45.8 and 50.5).
• Evidence the release did not produce. The contamination-free PubMed test in Table 5 post-trains a fresh model on that corpus. It says nothing directly about the checkpoint you download.
• The judge. Accuracy is scored by Qwen3.5-397B. GPT-oss-120B agrees on 91.7% of answers (Cohen's κ 0.81), which is reassuring, but it is still a model grading a model.

Where it fits

The paper calls read_page a learned retriever, one where the VLM is "indexer, selector, and reader in a single end-to-end model". I think that is the right frame, and it says when to use it. It helps with multi-hop questions over a document of a few tens of thousands of tokens, asked once, with no index and no wish to build one. There it beats the retrievers by the widest margins in the paper. It is the wrong tool for finding a single fact buried in a long document. On RULER, a ranking function from the 1990s beats it at every preset, with fewer tokens and an index built on a CPU. The paper's own suggestion is the sensible hybrid: "a cheap retriever could prune grossly irrelevant chunks before rendering."

The most portable result is not the checkpoint. Appendix 9 runs the same select-then-expand loop by prompting alone: show the thumbnails, ask for a page number, return that page's text. Sonnet 4.6 gains 4.8 to 9.0 points over reading the compressed pages directly (reported). The pattern needs no weights and no licence. What Apple trained is a 9B model that picks the page reliably enough to make it pay.