# GroundingPI: a 4.84B grounding model that answers in 1,000 coordinate tokens
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/groundingpi
> date: 2026-10-13
> tags: vision-language-models, object-detection, robotics, benchmarks, licensing, vla
[GroundingPI](https://huggingface.co/GroundingPI/GroundingPI) is a grounding model: give it an image and a phrase, and it answers with where things are, as boxes or as points. The [paper](https://arxiv.org/abs/2609.39601) (arXiv 2609.39601, "GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives", 30 September 2026) makes two claims. First, that this 4B model beats 44 baselines on average across 34 grounding benchmarks, GPT-6 Astra included. Second, that plugging it into a robot policy or a driving planner as the visual backbone beats the usual vision-language and video backbones.
The repost that brought it to me called it "Released by XPeng's AI team". The paper says something narrower; more below. I did not download the weights. Everything here comes from the config, the processor and modelling code, the chat template, the token table, the [release repository](https://github.com/groundingpi/GroundingPI) at commit `0047d45`, HTTP range reads of the two safetensors headers, and the paper's own tables. Labels: **measured** means I computed it from a file; **reported** means it is the authors' number, not re-run; **reasoned** means my arithmetic on the other two.
## What is in the checkpoint
`config.json` names the parts directly. The language model's `text_config` carries `"_name_or_path": "Qwen3-4B-Instruct-2507"`: 36 layers, hidden width 2560, 32 query heads over 8 KV heads. The vision side is `MoonViT3dPretrainedModel`, from `modeling_groundingpi_vision.py`, and the image processor's docstring still reads "Image processor class for Kimi-K3". Between them sits a projector the code calls "the freshly initialized GroundingPI projection connector": a LayerNorm, a 2×2 merge of neighbouring patch features into one 4096-wide vector, then two bias-free linear layers (4096 to 4096 to 2560) with a GELU and an RMSNorm.
I read both shard headers with range requests and summed every tensor:
| Part | Parameters | Share |
|---|---:|---:|
| MoonViT-V2 vision tower (27 layers, width 1024) | 401,214,464 | 8.3% |
| Projector | 27,267,584 | 0.6% |
| Qwen3 decoder body (36 layers) | 3,633,511,936 | 75.0% |
| Input embedding (152,670 × 2560) | 390,835,200 | 8.1% |
| Output head, untied (152,670 × 2560) | 390,835,200 | 8.1% |
| **Total** | **4,843,664,384** | |
All of it is BF16, 9,687,328,768 bytes (**measured**). The total matches the Hub API and the paper's Table 9 to the parameter. So "4B" is the language-backbone family, which the paper's appendix says itself. The checkpoint is 4.84B. Of the 0.8B gap, 390.8M is an output head that Qwen3-4B-Instruct-2507 does not have: upstream ties its embeddings (`tie_word_embeddings: true`, vocabulary 151,936), and GroundingPI unties them (**measured**, from both configs).
Is the vision tower really Kimi K3's? Kimi K3's own `config.json` describes a vision tower with the same 27 layers, width 1024, FFN 4096, 12 heads and QKV width 1536. I pulled four small norm vectors from each checkpoint by byte range. They have identical shapes and are close but not identical: the mean absolute difference is 0.1% to 0.8% of the weights' mean magnitude (**measured**). That fits the recipe: K3's encoder, frozen in Stage 1, then trained at a learning rate of 1e-6 (**reported**). The same probe against Qwen3-4B-Instruct-2507's norms shows bigger moves, 0.8% to 8%, consistent with a decoder trained at 1e-5 (**measured**). The site's [Kimi K3 article](/articles/kimi-k3) covers where MoonViT-V2 came from.
## Points and boxes as vocabulary
The interesting part is the output. `added_tokens.json` holds 1,027 added entries. 1,000 of them are the strings `<0>` through `<999>`, at token ids 151669 to 152668, followed by `` at 152669 (**measured**). Each coordinate is one atomic token with its own embedding row and its own output logit. The model does not spell "4", "2", "1" in digits. It emits one token that means 421.
The rest of the grammar reuses Qwen's existing control tokens. From `grounding_pi/protocol.py` in the release:
```text
answer := entry (", " entry)*
entry := <|object_ref_start|> phrase <|object_ref_end|> <|box_start|> payload <|box_end|>
payload := "None" | tuple ("," tuple)*
tuple := coord coord # point (x, y)
| coord coord coord coord # box (x1, y1, x2, y2)
coord := "<0>" | ... | "<999>"
```
Three conventions do a lot of work:
- **Instances of one label share a wrapper.** Two mugs are one entry with two tuples, joined by a bare comma. Boxes are sorted by x1 and points by x, so the order is deterministic and teacher forcing has one right answer.
- **Absence is an answer.** A queried category that is not in the image gets its wrapper with the literal text `None` inside. The data engine treats "absent" as a fact it must verify before a sample is accepted, not as the default when a teacher returned nothing.
- **Queries use ``.** Categories in the prompt are joined by that token, so a phrase can contain an ordinary comma.
The coordinate mapping is in the paper's Section 7.1. A coordinate sits on an intermediate integer grid v from 0 to 1000, normalised by the image's width or height. It is mapped onto the 1,000-entry vocabulary by
$$
q(v) = \left\lfloor \frac{999\,v + 500}{1000} \right\rfloor
$$
and decoded back as $x = q/999 \cdot W$. The paper does not say how pixels reach the 0..1000 grid. Rounding to nearest is the obvious reading, and it is what the widget below does.
Here is a worked example on a 1280×720 frame. The left mug's box is (180, 430, 300, 590) in pixels. The x1 normalises to 1000 × 180 / 1280 = 140.6, which rounds to v = 141 and maps to q = 141. The y1 gives 1000 × 430 / 720 = 597.2, so v = 597, and q = floor(596,903 / 1000) = 596. Then x2 = 300 gives `<234>` and y2 = 590 gives `<818>`. With a second mug at (860, 450, 960, 580) and an absent car, the query `Locate all the instances that match the following categories: mugcar.` should produce:
```text
<|object_ref_start|>mug<|object_ref_end|><|box_start|><141><596><234><818>,<671><624><749><805><|box_end|>, <|object_ref_start|>car<|object_ref_end|><|box_start|>None<|box_end|><|im_end|>
```
That is two boxes and a negative in 8 coordinate tokens and 9 protocol tokens, plus the two labels' own text tokens. Decoding `<141>` gives 141 / 999 × 1280 = 180.7 px, which is 0.7 px off the original (**reasoned**, by hand from the paper's formula).
The widget runs both directions: build a query and read the string, or edit the string and watch the outlines follow. It flags what the release's parser rejects: a coordinate above `<999>`, a tuple of three, an inverted box, a space between coordinates, a missing `<|im_end|>`.
### What the grid costs
Two consequences are worth stating with numbers.
**The grid is a resolution limit.** One step is W/999 of the image: 1.28 px at 1280 wide, 3.84 px at 3840. The two-stage rounding (pixels to 0..1000, then 0..1000 to 0..999) makes it slightly worse than a single rounding would. I swept every tenth of a pixel across widths of 640, 1000, 1920 and 3840, and the worst round-trip error came out at 0.639, 1.001, 1.917 and 3.834 px. That is very nearly a full grid step, where a single rounding straight onto 0..999 would cap it at half a step (**measured**). The intermediate grid also has 1,001 values feeding 1,000 tokens, so exactly one collision exists: v = 500 and v = 501 both become `<500>` (**measured**). This affects the training labels, not the model, and at the encoder's own resolution it is sub-pixel. But it is a design choice with a measurable cost, and the paper does not discuss it.
**The encoder sees roughly 1,170 px, whatever you send.** The processor caps input at 4,096 patches of 14 px, and the 2×2 merge turns that into at most 1,024 visual tokens. Running the processor's resize logic on its own (no weights involved), a 1920×1080 frame and a 3840×2160 frame both arrive at the encoder as 1172×659, padded to a grid of 1,008 visual tokens. A 1280×720 frame arrives at 1173×660 (**measured**). The 1,000-step output grid is therefore matched to what the encoder actually sees: about 1.17 resized pixels per step, or 24 steps per merged visual token (**reasoned**). The flip side is tiny objects. A 20-px object in a 4K frame is 6 px wide by the time the encoder sees it (**reasoned**). VisDrone, the paper's tiny-object aerial set, is the detection column where GroundingPI scores lowest (40.44 F1mIoU) and where SenseNova-Vision (42.35) and Qwen3.7-Max (41.59) beat it (**reported**).
The paper's own ablation is the reason to pay this cost. Atomic coordinates against coordinates written as text: grounding average 73.68 against 71.21, and text coordinates run at 0.25× the speed (**reported**, Figure 10). Output length is the other half. Table 5 puts GroundingPI at 7.6 tokens per box on COCO and 5.1 on Dense200, against 148.8 and 74.5 for SEED1.5-VL (**reported**). That is roughly 19.6× and 14.6× fewer, and the paper itself calls these serialisation ratios, not measured speedups.
## How it was trained
Three phases, all in the appendix with hyperparameters; none re-runnable, because the training data is not published.
1. **Base VLM training (Stages 1-3).** First the projector alone on image captions, with both backbones frozen. Then everything, on text plus image-text data, at learning rates of 1e-5 for the language model and 1e-6 for the encoder. Then general visual and video QA at a 32,768-token context. 128 GPUs, BF16, ZeRO-1 (**reported**).
2. **Coordinate alignment, called SFT (Stage 4).** Eight spatial task groups: detection, GUI, referring boxes, referring points, OCR, layout, dense pointing and counting, and visual prompting. Coordinate tokens get the same cross-entropy as every other token. There is no IoU regression loss (**reported**).
3. **GRPO.** 8 responses per prompt, 256 GPUs, only language parameters trained, learning rate 5e-7. For grounding, the advantage is 0.7 × a coverage reward plus 0.3 × a strict reward. The strict reward combines F1 at IoU 0.50, 0.75 and 0.95, matched IoU, count, format and ordering, with penalties for oversized and duplicate boxes. OCR gets a Hungarian-matched text-plus-geometry reward (**reported**). Removing this stage costs 0.82 points of grounding average (**reported**).
Total exposure is 500.41B plus 221.17B tokens (**reported**). Labels come from a "data engine" that fuses several teachers' outputs field by field, validates them per task, and iterates through a grounding expert. No data, per-source counts or teacher list is released, which is the largest reproducibility gap.
## The 34-benchmark claim, recomputed
The abstract says GroundingPI beats 44 baselines across 34 benchmarks spanning 11 capabilities, averaging 73.68% against GPT-6 Astra's 71.54%. The appendix defines that number precisely: "The headline Avg is the mean of 11 independently computed capability scores." It is not a mean over 34 benchmarks.
Each capability score is itself an unweighted mean of table columns. I rebuilt all eleven for both models from the per-benchmark tables. Visual prompting, for example, is the mean of the FSC147, Dense200, COCO and LVIS F1mIoU columns in Tables 36-39: 73.59 for GroundingPI and 65.84 for Astra. Two need a guess. Referring is the mean of seven columns (HumanRef, RefCOCO, RefCOCO+, RefCOCOg, RefCOCOg val, RefCOCOg test, and the "RefCOCO avg" column, which is itself a mean of three of those). Robo Point averages five columns, again including an average column. Only those compositions reproduce the published 84.89 and 75.12. The result: 73.68 for GroundingPI and 71.54 for Astra, both exact to two decimals (**measured**, from the reported tables).
What the average hides:
- **It is 8 wins out of 11 against Astra, not a sweep.** Astra leads GUI by 13.67 points (92.58 against 78.92), Robo Point by 5.96 and Dense Point by 1.72 (**reasoned**). On ScreenSpot-Pro alone the gap is 93.17 against 65.78 (**reported**). If you want a GUI-clicking model, the other side of this comparison is the one to read: see [Holo4](/articles/holo4), [Fara 1.5](/articles/fara-1-5) and [LFM2.5-VL-3B](/articles/lfm2-5-vl-3b).
- **Against everyone, it is 16 of 27 columns.** Across the main-text Tables 1-4, GroundingPI has the best score in 16 of 27 benchmark columns (**reasoned**, by counting). It loses LVIS (56.02 against SenseNova-Vision's 56.12), VisDrone, HierText (41.70 against Qwen3.7-Max's 42.95), TotalText and DocLayNet (85.08 against 85.53). To Astra it also loses Dense200 pointing and five of the six robot-spatial and GUI columns. Its wins are large where they land: Dense200 boxes at 74.53 against Astra's 65.04, and Qwen3-VL-4B's 14.02 (**reported**).
- **"34 benchmarks" counts task-dataset pairs.** COCO appears as box grounding, as pointing and as visual prompting. The suite uses 20 distinct datasets (**reasoned**, from the benchmark list in Section 12).
- **The protocol is uneven by necessity.** GroundingPI's numbers are the mean of ten runs; locally evaluated baselines get three; starred rows are copied from other papers. Kimi K3 is marked N/A on pointing, OCR and layout because its outputs did not fit the parser. The paper says plainly that this "does not establish a lack" of the capability (**reported**).
None of this makes the result wrong. On RefCOCO, RefCOCOg and RefCOCO+ it scores 85.79, 81.96 and 84.01 F1mIoU against Astra's 79.45, 76.15 and 77.84 (**reported**); a 4.84B model doing that is a real result. It is an average claim, and the appendix is honest about it.
## The physical-intelligence claim
This is the half of the title that "PI" refers to, and it is the half the release does not let you check.
The manipulation setup is careful. Every backbone, GroundingPI and eight others, is run once per observation. Eight hidden states are sampled at normalised depths, projected to width 1024, resampled to 64 tokens each, and fed into the cross-attention blocks of one fixed 16-block flow-matching Action DiT. Same data, same 40,000 updates, same optimiser (**reported**, Tables 14-15). The only variable is the backbone.
Reading the bars rather than the summary (all **reported**, differences **reasoned**):
| Setting | GroundingPI | Best other backbone | Gap |
|---|---:|---:|---:|
| RoboTwin 2.0, in-distribution | 66.20 | 60.62 (LocateAnything) | +5.58 |
| RoboTwin 2.0, Clean2Random | 17.60 | 14.10 (Rex-Omni) | +3.50 |
| RoboCasa-GR1, in-distribution | 37.75 | 39.00 (RynnBrain) | −1.25 |
| RoboCasa-GR1, unseen object types | 28.94 | 28.06 (RynnBrain) | +0.88 |
| RoboCasa-GR1, unseen appearance | 37.78 | 36.93 (RynnBrain) | +0.85 |
| RoboCasa-GR1, unseen combinations | 37.86 | 37.42 (RynnBrain) | +0.44 |
The abstract's "by up to 24.8% relative" is the Clean2Random row: 17.60 / 14.10 = 1.248, which is 3.5 points absolute on a split where every model fails more than four times in five. Three of the four "wins in every OOD setting" are under one point. The paper does not report seeds or variance for these runs. Its own scaling appendix calls four settings "without repeated-seed uncertainty estimates" too few to establish a trend. The data-efficiency claim has the same shape: 28.75% with half the demonstrations against RynnBrain's 27.75% with three quarters (**reported**), a one-point margin.
Driving is a smaller claim than it sounds. GroundingPI predicts six waypoints over 3 s, serialised through the same coordinate tokens, from the front camera plus up to seven past ego states that include velocity, acceleration and steering. It reaches an average open-loop L2 of 0.296 m, against 0.301 for Qwen3-VL-4B and 0.308 for RynnBrain (**reported**). That is 5 mm better than the runner-up, on nuScenes open-loop L2, the metric where ego history alone is known to carry much of the signal: a shortcut discussed in the [Qwen-Drive-1.0](/articles/qwen-drive-1-0) write-up. No ego-state ablation is reported, so how much of 0.296 m the image contributes is not measured. The authors' own wording is "modest", and they say it does not establish closed-loop performance.
Then the release. The `vla/` directory is a trimmed copy of OpenWAM for RoboTwin and StarVLA for RoboCasa. Its backbone switch lists `wan22_ti2v_5b | cosmos_predict25 | qwen3_vl_4b | paligemma_3b | rex_omni_3b | locate_anything_3b | rynnbrain_2b`, and the RoboCasa launcher's `case` has the same seven. Neither has a GroundingPI entry, and no file under either code tree names it outside READMEs and licences (**measured**, by grep). The documented RoboTwin protocol is "global batch 256, 10 epochs", while the paper's Table 15 says 1280 global and 40,000 updates; the RoboCasa launcher does default to 40,000 steps at full data. There is no nuScenes code at all, and no action-policy checkpoint (**measured**). The baselines are reproducible from the release; the row they are compared against is not. The model card says as much in one sentence: "downstream action-policy results are described in the paper."
The broader argument, a grounding model as the "System 1" backbone, is labelled in the paper as a hypothesis it has not tested. [Flex-π](/articles/flex-pi) takes the opposite route, predicting geometry and semantics jointly with actions.
## XPeng, and what "released by" means
XPeng Inc. is the first of eight affiliations, with Peking University, HKU, Berkeley, Princeton, NUS, Tsinghua and HKUST (GZ). The paper's disclaimer is unusually explicit: the work "originated from exploratory academic research undertaken by the project leader Qize Yu during their internship at Xpeng Inc.", "does not involve commercial applications, product development, or commercial deployment", and does "not represent the official positions, technical direction, product roadmap" of the company (**reported**). XPeng supplied compute and data. Reading it as a signal about XPeng's cars or robots goes further than the authors do.
## The licence is two licences
The Hub tags the model `license: other`, named `apache-2.0-with-upstream-kimi-k3`. The `LICENSE` file explains the split. The authors' own contributions are Apache-2.0. "Kimi-derived material remains subject to the Kimi K3 License", whose conditions "continue to apply to the combined model package where applicable." The Kimi-derived material is the vision tower and its code, 8.3% of the parameters (**measured**), and the code notices in the repo say the same. Qwen3-4B-Instruct-2507 is Apache-2.0 upstream.
The Kimi K3 terms are permissive with two conditions, both scoped to large commercial use:
- **Model as a Service.** If you, with your affiliates, run a business that gives third parties API-style "language model inference or fine-tuning" and have more than US\$20M in revenue over 12 months, you need a separate agreement with Moonshot before commercial use.
- **Attribution.** A commercial product with more than 100M monthly active users or more than US\$20M in monthly revenue must display "Kimi K3" prominently in its UI.
- **Exemptions.** Internal use that exposes nothing to third parties is exempt from both.
Whether a hosted grounding API counts as "language model inference" the text does not settle. Below those thresholds it behaves like a permissive licence with a notice to keep.
## Running it
The weights are 9.69 GB in BF16, so a 24 GB consumer card holds them with room for the 16,384-token serving context (**reasoned**). The documented path is narrower: the repo's `run.py` builds a bundled Transformers 5.7.0 fork and serves through vLLM 0.18.x in eager mode, one image per request, tested on B300, B200, H200, H800 and PPU. If you write your own client, decode with `skip_special_tokens=False` or every coordinate disappears, and never insert spaces between coordinate tokens.
One small inconsistency for client authors. The paper decodes with q / 999 × W. The release's `visualize()` uses q / 999 × (W − 1), so the two disagree by up to one pixel at the right and bottom edges (**measured**, from `client.py`). Pick one and be consistent with whatever produced your ground truth.
The Hub repo also carries a stray `streammind_gate.py`, a Mamba-based gate nothing else references (**measured**). Harmless.
## What holds
- **The architecture matches the card to the parameter**: a 4.84B checkpoint whose vision tower is Kimi K3's, lightly retrained (measured).
- **The protocol is clean**: one token per coordinate, deterministic order, explicit negatives, a reference parser. Its cost is a 1,000-step grid, and two-stage label rounding nearly doubles worst-case error (measured).
- **73.68 against 71.54 reproduces exactly**, as a mean of 11 capabilities, 8 of them won, with a 13.67-point GUI deficit inside (measured, reasoned).
- **The robot and driving results are small and not reproducible from the release**: under a point on three of four OOD splits, a 5 mm L2 margin with no ego-state ablation, no variance, and no GroundingPI backbone in the released VLA code (reported, measured).
As a grounding model that writes a box in about 8 tokens and holds its own against a frontier model on dense detection and referring, it is worth having. As evidence that robot policies need grounding pretraining, it is a promising first experiment nobody outside can re-run yet.