Xiaomi-OCR-0: what a 0.8B document parser's headline number covers, and what its training did to Qwen's weights

mdjsonmcpcopy↗ai

2026-10-31 · 20 min · ocr · document-parsing · vision-language-models · qwen · reinforcement-learning · xiaomi

Why read this

High· top 30%

Weights compared across three checkpoints, ranks recounted, CPU runs of the model and the real layout detector, one stage vs two. Synthetic pages.

• Checked against the source
• Runs on a laptop CPU
• A new technique

·Vision & multimodal·Apache-2.0·Practitioner model

How this was scored

Is it new?

2 of 3: A real new idea, method or capability

Can I trust it?

3 of 3: Reproduces the headline result, or shows from primary files it is wrong

Can I run it?

2 of 3: Open code or weights with real limits

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

1 of 3: Relevant for months

Does it affect many?

1 of 3: A specialist community

Only here?

2 of 3: A teardown or measurement few others did

Score 69 of 100, ranked 155 of 666 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

A post on 30 October said Xiaomi had dropped the idea that document readers have to be big: an open 0.8B model that parses documents, extracts key information and answers questions about them, first on OmniDocBench v1.6 among models under 1B, 95.24 on Real5-OmniDocBench, 83.2 on average across five visual question answering benchmarks, with weights, code, a demo, an MCP service and an agent Skill. The model is xiaomi-research/Xiaomi-OCR-0, the code is at SeerRay-Lab/Xiaomi-OCR-0, and the technical report is arXiv 2609.36136.

All of that checks out against the paper, with one condition on the headline that the post leaves out. The more interesting things are in the files: what kind of 0.8B this is, and what its training did to the weights it started from.

Every claim carries a label. Measured means I counted or timed it from files or from a run. Reported means Xiaomi says it and I did not reproduce it. Reasoned means I derived it from the numbers.

What the 0.8B is

It is Qwen3.5-0.8B with every weight retrained. Reading the safetensors header of model.safetensors over HTTP range requests, with no weights downloaded, gives 488 tensors and 873,438,784 values, all bf16 (measured). Qwen3.5-0.8B's header has the same 488 names with the same 488 shapes and the same total. Nothing was added or resized.

The split is measured from the shapes. One reading of it is reasoned: for a model sold as 0.8B, 29% of the stored values are a lookup table, and the part that does the reading, the vision tower and the 24 layers together, is about 0.6B. For scale against another document parser on this site: Unlimited-OCR-2 stores 3.34B parameters and touches about 0.58B per decoded token (my counts there), and was built to keep decoding cost flat across very long documents. Xiaomi-OCR-0 is dense and small, and its argument is accuracy per parameter on single pages.

What training did to the weights

The model card says it starts from Qwen/Qwen3.5-0.8B-Base. The paper says "starting from Qwen3.5-0.8B". Those are different checkpoints, and the same tensors exist in both, so the weights can say which.

I read 13 tensors, spread over the language layers, the vision tower, the merger and the multi-token head, from three checkpoints: Xiaomi-OCR-0, Qwen3.5-0.8B-Base and the post-trained Qwen3.5-0.8B, each over range requests (measured, bf16 converted to float32, relative L2 distance between the same tensor in two checkpoints). Then I drew a triangle for each, with Base at one corner.

Three things come out of it.

Xiaomi moved the weights much further than Qwen did. Take the 10 tensors where Qwen's own post-training changed anything. Qwen moved them by 0.04 to 0.18 relative to their size, median 0.068. Xiaomi's checkpoint is 0.39 to 0.90 away from Base on the same tensors, median 0.57, so on each tensor it is 4.7 to 11.1 times as far, median about 7 times (measured). All 600 embedding rows I sampled differ from Base too. This is a full retraining of a small model on about 170 million samples, not an adapter.

The vision tower was unfrozen. I compared 12 vision tensors byte for byte between Qwen3.5-0.8B and Qwen3.5-0.8B-Base: 10 randomly chosen block weights and the patch embedding are identical, so Qwen's post-training left the encoder blocks alone. Only the position embedding differed (measured). Xiaomi's three block tensors in the triangle above are 0.31 to 0.40 away from both (measured). The merger between the vision tower and the language model did move in Qwen's post-training, by about 0.065.

It started from Base, and the weights show it. If Xiaomi had started from the post-trained model, Qwen's step would still be part of Xiaomi's change, and the angle at Base between "Qwen's move" and "Xiaomi's move" would not be a right angle: its cosine would be about a/c, where a is Qwen's distance and c is Xiaomi's, which is 0.09 to 0.21 on these tensors. Measured, the cosine is between −0.006 and +0.017 for nine of the ten tensors, so the angle is 89.0 to 90.3 degrees, and the distance from Xiaomi to the post-trained model matches the right-angle prediction to within 0.001 on nine of the ten (layer 0 down-projection: predicted 0.6157, measured 0.6161). The tenth, the MLP of the multi-token head, is at 84 degrees (cosine +0.105), which is the one place the two updates are visibly related.

A caveat on how much to read into perpendicularity. Two independent updates to a matrix with millions of entries are close to perpendicular by default, so a right angle is what "unrelated" looks like. It is the comparison with the 0.09 to 0.21 that carries the result: the post-trained model's step is not inside Xiaomi's weights. Thirteen tensors are a sample, not the whole checkpoint, and this is a statement about geometry, not about what either training run learned.

How it was trained

The recipe is in the technical report. Everything in this section is reported.

expand

Data. About 170 million samples. A pool of eight existing OCR systems (PaddleOCR-VL-1.6, MinerU2.5-Pro, GLM-OCR, dots.mocr, HunyuanOCR-1.5, TeleOCR, OvisOCR2, Qianfan-OCR) annotates each region. Ensemble triplet consensus ranks candidates by agreement across the pool and keeps a three-expert subset. Where experts disagree, the candidate annotation is rendered back to an image and compared with the source region, which catches structural errors in tables and formulas that a text comparison misses. Hard samples are mined by expert agreement, the model's own self-consistency, semantic clustering and verified failure patterns, and synthesis fills under-covered structures and recurring errors.

Three stages. Text-anchoring pretraining follows Q-Mask: a query-conditioned mask decoder is trained alongside next-token prediction so the model learns where text sits, with a loss that adds a next-token term to Dice and cross-entropy mask terms. Continued pretraining drops the mask decoder and trains the shared autoregressive format over parsing, understanding, captioning, scene text, handwriting and long-tail tasks, keeping the base model's single multi-token-prediction head and training it jointly. Then mixed-task reinforcement learning, a DAPO-style variant of GRPO with token-level loss aggregation, clip-higher, dynamic sampling that discards rollout groups with zero reward variance, and no KL term.

Rewards. Region-level samples score by text edit similarity, table TEDS or formula CDM. Full pages are aligned with multi-granularity adaptive matching and the three scores are weighted by character coverage. Key-information extraction checks the JSON first, then fields. Question answering uses ANLS. Anything unparseable, malformed or over length gets zero.

What the ablations say, all on OmniDocBench v1.6 and all reported:

The last three rows are the paper's claim that understanding data helps parsing only once parsing is already strong: it hurt at the start, helped most in the middle, and added little at the end. The 0.8B baseline in the paper's ablation table scores 72.21 on v1.6 before any of this. Scaling to a 4B teacher with the same data helps understanding much more than parsing (88.1 against 83.2 on the five question-answering benchmarks, but 96.97 against 96.83 on parsing), which is the paper's argument that parsing is the part a small model can already do.

expand

What the headline number covers

Here is the check the post invites. The paper's tables list every system it compares, with its own score.

From those tables (reported, ranks counted by me): on OmniDocBench v1.6 it is second of 34 rows, 0.04 points behind TeleOCR at 1.2B and 0.25 ahead of the next 0.8B system, which makes it first of the 8 rows under 1B. On Real5-OmniDocBench, whose pages are recaptured by scanning, warping, screen photography, uneven light and skew, it is first of 30 rows, ahead of the next system by 2.05. On Wild-OmniDocBench it is second of 10 behind TeleOCR. So the post's "first among models under 1B on v1.6" is right, and so is the card's "first on Real5". The margins are small: a quarter of a point on the main benchmark.

The condition is in section 4.1 of the report. The v1.6, Real5 and Wild scores are two-stage. A separate layout detector, PP-DocLayoutV3, finds the regions on the page, Xiaomi-OCR-0 reads each crop, and the pieces are assembled into a page. The report says the 0.8B count "excludes the external layout detector." That is how most of the systems in the table are scored, so the comparison is fair, but it means the headline is not a single 0.8B model reading a page. The model can read a whole page from one prompt, and the repository ships that path as e2e_img2md.py next to twostage_img2md.py; the paper's benchmark numbers are for the two-stage one.

I did not re-run any of these benchmarks. OmniDocBench v1.6 is 1,651 pages scored with several matching and edit-distance tools, and the two-stage path needs the layout detector. An earlier version of this article said that meant the Paddle runtime; that was wrong, see "What running it showed" below.

What it outputs

The model takes a different prompt for each task and answers in a different format. These are the prompts from the model card (read):

The table format is the unfamiliar part. OTSL writes a table as a stream of cell tags, six of them, and lets spans fall out of the tags: a cell that continues the one to its left, the one above, or both. postprocess/otsl.py in the repository turns it into HTML with row and column spans. The widget below is my own decoder, written from the tag meanings in that file's docstring; edit the string and the table follows.

Two settings matter when reproducing results (read from the evaluation scripts): decoding is greedy, temperature 0 with top_p 0.8, at most 16,384 new tokens, with the chat template's thinking mode off. And the post-processing includes a repetition guard, postprocess/repetition_guard.py, that flags a generation as suspicious in four cases: the generation stopped because it hit the token limit; one character repeats 128 or more times in a row and that run is at least 512 long or at least half the non-whitespace text; the text ends in a unit of up to 256 characters repeated at least 8 times, covering at least 512 characters and half the text; or a single line makes up at least 80% of the lines and repeats at least 8 times. Small OCR models loop, and the repository treats that as a normal failure to detect, not an exception.

The agent Skill

The post mentions an agent Skill. The repository's SKILL.md is unusual in what it asks the agent to do. The one prompt a user is meant to give is "Read and execute" a raw GitHub URL pointing at that file, and the file then tells the agent to install a Skill with the skills CLI, create a private virtual environment, install the MCP service's dependencies, and register one MCP entry. It tells the agent to explain those steps and ask for authorization before running them, not to download model weights unless authorized, to keep any inference listener bound to loopback, and not to claim inference is ready until an image request has succeeded. The MCP service does the task-specific prompting, page and region preprocessing and result assembly; the model itself is served locally through SGLang, vLLM or Ollama.

That is a reasonable design for a tool whose installation is itself an agent task, and the file's own caution is a good sign. It also means the trust decision is in the file, so it is worth reading it before pointing an agent at it.

What I ran

I ran the model itself, on CPU, on pages I rendered, so the ground truth is known. The model loads with the standard qwen3_5 classes in transformers 5.20.2, straight from the repository, with no remote code and no trust_remote_code. I used bf16, two threads, greedy decoding and the prompts from the model card; peak resident memory was 2.47 GB (measured). The five test inputs, the scripts and the raw outputs are in public/articles/xiaomi-ocr/run/.

The pages: a four-line paragraph in DejaVu Sans (860 by 246 pixels); a five-row table with a title cell merged across three columns (600 by 384); and two formulas typeset by matplotlib (832 by 169), an integral and a sum. I scored text by character error rate, the table by comparing every cell and its span against the grid I drew, and the formulas by edit distance after normalising whitespace and brace style.

Three things worth saying about those results.

The formats are as documented. The table came back as OTSL with the span tags in the right place: the merged title is <fcel>Benchmark results<lcel><lcel>. The decoder widget above turns the model's actual output back into the grid I drew; the second preset in it is that output, unedited.

The LaTeX is in a different style from the source. The model wrote the integral as \[\int_ {0} ^ {\infty} e ^ {- x ^ {2}} d x = \frac {\sqrt {\pi}}{2}\]: display delimiters, a space between every token, and braces around every subscript and superscript, where my source had \int_0^\infty e^{-x^2}\,dx. The two typeset to the same formula, but a string comparison scored the first 6 characters off until I normalised braces and spaces, and then 0. That is a property of the output to plan for when scoring it. It is also why formula benchmarks like CDM render both sides before comparing.

The timing is not a benchmark. The 4.8 to 13 seconds per new token is for generate() including the image prefill, on a four-core machine that was running other jobs at a load average of 16 to 20, on a CPU without bf16 hardware, with transformers falling back to its unfused reference code for the 18 Gated DeltaNet layers. The first two rows produced the same 48 tokens and took 308 and 622 seconds, which is the machine, not the prompts. An image is cheap in tokens: the processor gives one image token per 32 by 32 pixel block (patch 16, merged 2 by 2), so a 860 by 246 page is 216 tokens.

What this does not show: five clean, synthetic inputs in one font are nothing like the scanned, warped and photographed pages the benchmarks score, and a character error rate of 0 on one paragraph says nothing about the rates in the paper. What it does show is that a 0.8B model on a laptop-class CPU reads a clean page, a table with a merged cell and two formulas correctly, and that the documented formats are what comes out.

What running it showed: one stage against two

The headline is two-stage, so I ran both paths on pages I drew, with ground truth, on CPU. Everything here is measured; scripts, pages, raw outputs and the log are in public/articles/xiaomi-ocr/verify/.

The detector runs on CPU, and not through Paddle. An earlier version of this article implied the two-stage path needs PaddlePaddle. The repository's twostage_img2md.py has a second route: load_layout_model loads PPDocLayoutV3ForObjectDetection from transformers, and the PaddleX script is the alternative. PaddlePaddle/PP-DocLayoutV3_safetensors is that checkpoint: 33,175,165 parameters, 25 class ids, loaded with transformers 5.20.2 plus opencv-python-headless (its post-processing asks for OpenCV). It took 7 to 16 seconds a page on two threads, about 2.1 GB peak with the interpreter and torch. The detector is about 4% of the VLM's size, so "0.8B excludes the detector" is true and also small: the second model is a 33M one, not a second 0.8B. One thing to know if you read the repo: the checkpoint's own config calls ids 5 and 15 both formula, while the repository maps ids itself (5 is display_formula, 15 inline_formula) and its task table has no entry for the bare name formula. Going by id, as the repo does, is what routes a formula crop to the formula prompt.

What was refused. I wanted to import the repository's own run_layout_detection, region post-processing and merge_page_to_markdown. The permission layer denied that twice, so none of the repository's code ran. What I ran is the same transformers classes it calls, the prompts and task mapping I read in it, and its crop rule (box normalised to 0 to 1000, rectangle crop), with regions ordered by the detector's own reading-order score. I did not apply its box merging, its repetition guard or its page assembler. So this is the model and the detector on the repository's recipe, not the repository's script.

Three pages, each read twice: once whole with the document prompt, once region by region.

The detector found every region I drew on all three pages: a title, two columns, a table, a formula, two paragraphs, at confidences of 0.83 to 0.98. In the two-column case the detector's own order put the left column first, and the whole-page read did too; had the model swapped the columns, the error rate would have been 0.68, so reading order was a real thing to get right and both paths did.

What that says about the caveat: on a clean page and on a mildly damaged one, the whole-page path gave the same answer as the two-stage path, character for character. The paper's condition is real, since its scores come from the two-stage path, but I cannot say how large it is: my pages are one font, one column structure and one table, and a gap that exists would show on pages I did not draw. The cost difference is easier to see. Two-stage fed the model 35 to 49% fewer image tokens, because crops drop the whitespace between regions. About the same number of tokens came out: 76 and 76, 75 and 74, 167 and 167 for whole page and crops. Wall time did not show a consistent win: the whole page took 108, 30 and 77 seconds, the crops 70, 79 and 51 seconds in total, and that is before adding 7 to 16 seconds of detector time per page. The box was shared and its load changed between runs, so I would not read speed into it.

One correction to the table of runs above, which comes from the same machine. I reported 4.8 to 13 seconds per new token. That was a box at a load average of 16 to 20. On this run, at a load of about 6, the same pipeline made 0.4 to 1.0 seconds per new token for whole pages and 1 to 2 for short crops (prefill dominates a 7-token title). The first figure is the load, not the model. Peak memory this time was 2.81 GB for the VLM process, not the 2.47 GB I reported before; I did not find out why (the pages and the load differ); it is over the 2.5 GB I was working to, so I say so.

What I did not check

• The benchmarks. None of the paper's scores was re-run. Ranks were counted from the paper's tables. The one-stage against two-stage gap on real pages is untested: my three pages are synthetic and mostly easy.
• The repository's own scripts. They were read, not run (the run was refused); box merging, the repetition guard and page assembly were not exercised.
• The training claims. The data engine, the 170 million samples and the reinforcement-learning recipe are the report's description. What I can say from the weights is that the training was large and started from Base.
• Thirteen tensors. The weight comparison is a sample, spread over the components, not the whole checkpoint.
• Speed and real pages. My runs used eight synthetic or lightly degraded inputs and a shared machine. It says the pipeline works, not how fast or how accurate it is on hard pages.
• Community conversions. The GGUF and Ollama builds are community work the model card says it has not evaluated; I did not look at them.
• The Xiaomi-OCR name. The card links to SeerRay-Lab/Xiaomi-OCR-0 for the code and Hub pages while the weights I read are under xiaomi-research. They are the same model; I did not look into why there are two organizations.

Sources

• Technical report: Xiaomi-OCR-0 Technical Report, arXiv 2609.36136, including Tables 1 to 3 and 10 to 12 and Figures 2 and 4.
• Model: xiaomi-research/Xiaomi-OCR-0 (Apache-2.0), compared with Qwen/Qwen3.5-0.8B-Base and Qwen/Qwen3.5-0.8B.
• Code: SeerRay-Lab/Xiaomi-OCR-0 (Apache-2.0): postprocess/otsl.py, postprocess/repetition_guard.py, evaluation_pipeline/, SKILL.md.
• The post: @EngMoElgaraihy, 30 October 2026.
• Related on this site: Unlimited-OCR-2.