Unlimited-OCR-2: the 512-token budget that reads 416

mdjsonmcpcopy↗ai

2026-10-27 · 18 min · ocr · document-parsing · vision-language-models · sparse-attention · long-context · baidu

Why read this

Notable· top 60%

SR-SWA read from the shipped code and rebuilt in NumPy: a 128+12x32 budget that reads 416-448 keys in base mode, a header count, and where 'constant' stops.

• Original analysis
• Runs on a consumer GPU
• A new technique

·Vision & multimodal·MIT·Research model

How this was scored

Is it new?

2 of 3: A real new idea, method or capability

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

2 of 3: Open code or weights with real limits

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

1 of 3: Relevant for months

Does it affect many?

1 of 3: A specialist community

Only here?

2 of 3: A teardown or measurement few others did

Score 63 of 100, ranked 270 of 621 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

Baidu's launch post for Unlimited-OCR-2 says the new attention "condenses text-visual attention interactions into a fixed-size set of working references," and promises "nearly constant per-step decoding latency and memory footprint as documents grow longer." The model card says the set is "e.g., 512 tokens."

That is a claim about a number of keys, and the number can be counted. I read the technical report (a PDF in the model repo), config.json, dash_attention.py, modeling_deepseekv2.py and modeling_unlimitedocr.py, and pulled the safetensors headers by range request. Then I wrote the read-set rule as a small NumPy program and counted what a decode query touches at 1 to 300 pages. I did not run the model and I did not OCR anything.

Every claim below carries a label. Measured means I counted or timed it myself from files. Reported means Baidu says it and I did not reproduce it. Reasoned means I derived it from the code or the numbers.

What it is, and what changed from version 1

Unlimited-OCR is a document parser built on DeepSeek-OCR's recipe: a vision encoder compresses each page into 273 tokens, and a small mixture-of-experts language model writes out the page as text. Version 1 (June 2026) fed all pages of a document into one request and bounded the text history with a sliding window, which the report calls R-SWA, for Reference Sliding Window Attention. The text it had already written fell out of the window. Every visual token of every page stayed attendable.

Version 2 replaces that with SR-SWA, Sparse Reference Sliding Window Attention. The visual tokens a query can read are now limited too. The report describes two steps (reported, section 3.3):

1. Page Gate. Candidate blocks are only those that hold at least one image token of the page the decoder is currently writing.
2. Token Sieve. Each candidate block is summarised into one key, scored against the query, run through an entmax (a sparse softmax), and the top-k blocks are kept. Their log-probabilities, centred on their mean, are added to the attention logits as a bias, so the loss trains the router with no distillation target.

Add the local window and you have the read set. The report's Figure 1 shows the idea against v1, and Figure 3 the pipeline.

expand

expand

The parameters, counted

I read the two safetensors headers (256,864 and 90,160 bytes of JSON) with HTTP range requests and multiplied out the shapes. The total is 3,343,978,497 stored values (measured). Of those, 257 are an int64 position_ids buffer in the CLIP tower; the rest are bf16. The tensors come to 6,687,958,536 bytes, or 6.69 GB, a little below the roughly 6.9 GB I had expected from the hub listing.

The "A0.5B" part is a decoder statement, so the right question is what one decoded token touches. Counting the text decoder only, with 6 of 64 routed experts, both shared experts, attention, the layer-0 dense MLP, the router and lm_head, I get 578,064,128 values (measured count, reasoned selection). The report writes "3B-MOE-A570M", which is 1.4% under my count and consistent with it. The embedding table is a lookup, and the SAM and CLIP encoders run on the page image, not per generated token.

Three things in the table are worth a sentence. The routed experts are 72% of everything stored and about 39% of what a token touches. The two 165M tables around the decoder, embeddings and lm_head, are 10% of the checkpoint. And the router that this article is about, dash_router, is 3,936,768 values, 0.12% of the model. A second module called indexer of nearly the same size ships in the checkpoint. On the decode path I read, it supplies positional coordinates and page ids; I could not find its weights multiplied into a score.

The shipped config says 512

config.json carries the knobs under sparse_attention_config (read):

The arithmetic is 128 + 12 x 32 = 512, the "e.g., 512 tokens" on the card. The sparse_topk_blocks: 16 entry is the older total budget of 16 blocks; remote_blocks = topk_blocks - local_blocks, which is the same 12. The training code anneals the hard top-k from 64 to 12 over 1,000 steps (DASH_K_START, DASH_K_END, DASH_K_ANNEAL_STEPS default values), a detail the report does not mention.

The router, as written in _block_scores_headcls and its inference twin _block_scores_infer (read), works in four steps. A learned vector head_cls, initialised to zero so it starts as mean pooling, attends the 128-d index keys inside each 32-token block to make one summary key per block. The query's 128-d index vector is dotted with each summary and divided by the square root of 128. Blocks inside the local window, in the future, or off the query's page are masked. At inference, entmax-1.5 runs on the best 16 scores and the first 12 slots are the read set. Each kept block gets bias (log p - mean(log p)), with p floored at 1e-4.

The attention itself then adds that bias to the logits of the tokens in the selected blocks, applies no bias to the 128 local tokens, and softmaxes over the union (reasoned from _decode_attend, and equation 7 of the report).

Counting what a query reads

Here is the part nobody states. Page Gate restricts candidates to blocks that hold an image token of the current page. In multi-page mode the README says each page is "one global view (base mode)", which is 273 tokens. The prompt is <PAGE> then 273 image tokens, repeated per page. Blocks are 32 tokens and ignore page boundaries.

So a page spans 273 / 32 = 8.5 blocks. With the <PAGE> marker shifting the alignment, a page touches 9 or 10 blocks, never more. The cap of 12 never binds.

I wrote the layout, the Page Gate rule and the local window as a short NumPy program, called sr_swa_sim.py here. This is our re-derivation, not Baidu's kernel. Its core:

BS, LOCAL, K_REMOTE, PER_PAGE = 32, 128, 12, 273
 
def layout(n_pages, n_text):          # BOS, then <PAGE> + 273 image tokens a page, 5 instruction tokens
    kind = [-1]
    for p in range(n_pages):
        kind.append(-1); kind += [p] * PER_PAGE
    kind += [-1] * 5
    return np.array(kind + [-1] * n_text), len(kind)
 
def page_candidates(kind, n_pages):   # Page Gate: blocks holding >= 1 image token of page p
    nb = -(-len(kind) // BS)
    pad = np.pad(kind, (0, nb * BS - len(kind)), constant_values=-1).reshape(nb, BS)
    return [np.where((pad == p).any(1))[0] for p in range(n_pages)]
 
def read_set(pos, page, cands):       # local window U the (at most 12) selected blocks
    keys = set(range(max(0, pos - LOCAL + 1), pos + 1))
    for b in cands[page][:K_REMOTE]:
        keys.update(range(b * BS, (b + 1) * BS))
    return keys

Which blocks the learned router would pick when there are more candidates than slots is unknowable without the weights. The count does not depend on it, because the cap is above the candidate count. Run at 1 to 300 pages, with 1,024 generated tokens per page (the setting in the report's latency figure) and the query halfway through the last page, it gives this (measured on the re-derivation):

Keys read per decode step. Of SR-SWA's 416 to 448 keys, 273 to 296 are image tokens and 128 are the local text window; the rest are the neighbouring page's tokens and the <PAGE> marker that share a block with the current page's edge. Across 10, 100 and 300 pages every page had 9 or 10 candidate blocks.

So the fixed budget is 416 to 448 keys in this configuration, not 512 (measured on the re-derivation; reasoned about the model). That is slightly better than advertised, and it comes with a consequence: with 9 or 10 candidates and 12 slots, Token Sieve selects nothing in multi-page base mode. It keeps everything on the page and learns only the bias that reweights the blocks. The selection matters in gundam mode, where a page carries more image tokens (a code comment says so) and the candidate count can exceed 12. The report's latency figure is in gundam mode; the README's multi-page example is base mode. I did not count gundam pages.

There is a second consequence for the claim "looks only at what it transcribes." In base mode, SR-SWA is close to "attend to the whole current page and the last 128 tokens of text." That is a sound inductive bias for transcription and it is what a person copying a book does. It also means everything that makes the page-to-page structure work, such as a table continuing on the next page or a heading seven pages back, rides on the text window and on the model's own earlier text. The report's limitations paragraph says it directly: the mechanism "does not preserve global document information." Its HeadingTEDS scores are the lowest column in both tables: 60.10 on MPDocBench and 48.20 on MarathonDocBench (reported).

Try the pattern yourself. The strip below builds the prompt for 2 to 12 pages and shows which blocks each rule reads for a query on the page you pick. Switch to R-SWA and every blue and amber block lights up; switch to SR-SWA and only the query page's 9 or 10 do.

What a constant read set buys

A decode step in a model this size is limited by bytes moved more than by arithmetic. For a bf16 KV cache with 12 layers, 10 heads of 128, one key costs 12 x 2 x 1,280 x 2 = 61,440 bytes (reasoned from the config). Attention costs 12 x 4 x 1,280 = 61,440 FLOPs per key as well, which is a coincidence of the numbers, so the two scales below move together.

The weights cost 2 x 578M = 1.16 GFLOP and 1.16 GB per token at batch 1. Full attention at 100 pages reads 129,295 keys, 7.9 GB of KV and 7.9 GFLOP, 6.9 times the weights. R-SWA reads 1.7 GB. SR-SWA reads 27.5 MB, 2.4% of the weights. Put another way, SR-SWA's attention is a rounding error on the step and the step becomes the cost of the experts.

If step time were weights plus KV bytes, the full-to-SR-SWA ratio at 100 pages would be (1.16 + 7.94) / (1.16 + 0.03) = 7.7 (reasoned). The report measures 7.0x per step over full attention and 5.6x over R-SWA at 100 pages, with batch 3 and gundam pages, and reads the SR-SWA curve at about 3.9 ms from 2 pages to 100 (my reading off Figure 6b; it is a plot, not a table). A byte model and a measurement in a different mode agreeing to within 10% is encouraging and proves nothing: the settings differ, and a real kernel has launch and gather overheads.

expand

I timed the part I could. One layer, one step, 10 heads of 128, float32 NumPy on one CPU thread, median of 15 runs on a shared box. The dense version scores every cached key; the sparse version gathers 448 keys first and attends only those. Both compute the same thing with the same mask: on random data at 1,950 keys the two outputs differed by 3e-16 (measured).

Dense time is linear in length. Sparse time is flat within the noise of a shared machine, at 3 to 5 ms, almost all of it NumPy's fancy-index copy. Below about 3,000 keys the gather loses, roughly ten base-mode pages of context; for single-page parsing there is nothing to skip. These are CPU timings of my own code and say nothing about Baidu's Triton or SGLang kernels.

Where "constant" stops

The sparse read set is constant. Three other things are not, and the code shows each (read, then reasoned).

The Transformers reference path is dense. _decode_attend in modeling_deepseekv2.py builds a [B, Lq, S] mask over the whole cache and computes q @ k^T against every key, then masks. It reproduces the read set exactly, so it is a correctness reference, and it does no less work than full attention. The speed in the report comes from the SGLang wheel in the repo, which I did not unpack. The README recommends SGLang for serving and calls the Transformers path "native, for quick tests."

The router scores every block, every step. In the reference code, _block_scores_infer recomputes the head_cls summary over all cached index keys on each decode step, then scores all S / 32 blocks. That is about 4 x 128 FLOPs per cached key per layer. At 388,895 keys (300 pages) the summary pass costs about 2.4 GFLOP a step across 12 layers, 93 times the 26 MFLOP the sparse attention itself needs. If the SGLang kernel caches block summaries, the step cost really is flat apart from scoring S / 32 dot products. If it does not, "constant" is a statement about attention and not about the step. I could not tell which from the files I read, and the report's flat curve, from 2 pages to 100, does not settle it, since a 10% term hides inside a 3.9 ms step.

The KV cache still grows. Attending 448 keys does not free the 388,895 stored ones. The report handles this separately with the Unlimited-Horizon Decoding Paradigm (its Algorithm 1): split the document into overlapping page chunks with one shared boundary page, keep the KV of the active chunk, reuse the boundary page's KV, and release the rest. It reports peak KV memory at batch 1 on MarathonDocBench falling from 47.00 GB to 1.77 GB, and throughput at batch 16 rising from 360 to 1,315 tokens per second (reported, Table 4). Divided by 61,440 bytes a token, those are about 765,000 and 28,800 cached tokens (reasoned). The second figure is a resident window of about 29,000 tokens. That is on the order of the README's ten-page window (SGLANG_UOCR_DOC_*_PAGE_WINDOW=10) if a gundam page costs around 2,900 tokens, which is my assumption and not a number I found. So the paper's "unlimited" has two halves: SR-SWA makes the compute per token flat, and chunking makes the memory flat. Neither is a property of the attention rule alone.

The benchmark numbers

All of these are reported; I reproduced none.

Three readings. First, the Marathon gap of 43 points is real in the table and should not be read as better reading. The rivals are general models given hundreds of pages in one context; Figure 6a shows one failing outright past 223 pages. The comparison measures who can hold a 300-page job at all. The benchmark is also the authors' own, built from page-by-page parser output refined by annotators (reported), so it rewards the model that shares the pseudo-labelers' conventions, and the training data came from the same two parsers, MinerU2.5-Pro and PaddleOCR-VL (reported).

Second, v1 to v2 rises 22.7 points on MPDocBench (61.89 to 84.59), and the report credits SR-SWA. The change also includes a post-training run on 2M multi-page and 10M single-page documents, 25 x 8 H800 GPUs, one epoch, sequences to 64K tokens (reported). I cannot separate the attention rule from the data. The report has no ablation holding the data fixed.

Third, single-page quality barely moved, 93.92 to 94.02, and sits below two models in its own Table 3, TeleOCR (96.91) and OvisOCR2 (96.47). That is the sane result for a change that touches long documents only, and it says the sparse rule does not hurt local reading.

Running it

The model is MIT and 6.69 GB of bf16 weights, so the weights alone fit a 12 GB card (reasoned, untested; the README tests an 80 GB A800). The README's tested setup is an A800 80GB, Python 3.10, torch==2.13.0, transformers==4.52.1. Serving the speedups needs the SGLang wheel in the repo and its environment variables (SGLANG_UOCR_DOC_BASE_PAGE_WINDOW=10, SGLANG_UOCR_MAX_PAGE_TOKENS=20000), and the sample client sets a no-repeat n-gram logit processor with ngram_size 35 and a window of 1,024 for multi-page requests. The custom code requires trust_remote_code=True: read modeling_unlimitedocr.py before you point it at documents you care about.

What I would do with it

If you parse single pages, none of this changes anything: there is nothing to skip, and v2 is a point above v1 on OmniDocBench. If you parse long scanned books, the useful property is that cost per step stops depending on how far in you are, as long as you use the chunked serving path and the block summaries are cached. Test one thing before trusting it on a contract or a ledger: whether content that continues across a window boundary survives. The README says "a smaller page window lowers peak memory but hurts cross-page merging," which is the same cost the limitations paragraph describes, stated as a tuning knob.

What I did not do

I did not run the model, the SGLang wheel or any benchmark. I did not measure gundam-mode token counts, so my read-set count covers base mode only. I did not see the trained router choose among more than 12 candidates, so what Token Sieve selects, as opposed to how many keys it keeps, is untested here. The NumPy timings are one machine, one thread, and a shared one. The figures I read off the report's plots are approximate.

What I would run next, on a GPU: the same read-set count taken from a live topi tensor during a 20-page decode, in both modes, plus the per-step time of the Transformers reference against SGLang at 10, 50 and 100 pages. That would settle the two open items above, whether the selection ever binds and whether the router's summary pass is cached.

Sources

• Baidu ERNIE Team, "Unlimited OCR Works 2" (the PDF in the model repo); the README links version 1 as arXiv:2606.23050
• Launch post, 27 October 2026
• baidu/Unlimited-OCR-2: config.json, dash_attention.py, modeling_deepseekv2.py, modeling_unlimitedocr.py, _sparse_base_impl.py, both safetensors headers
• baidu/Unlimited-OCR