# Jais 3: adapting K2 Horizon to Arabic without touching the tokenizer

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/jais-3
> date: 2026-10-16
> tags: pretraining, tokenization, benchmarks, asr, speech

Six weeks ago IFM shipped [K2 Horizon](/articles/k2-horizon-mova): six open
models from 0.9B to 375B, with the data, the training code and every
intermediate checkpoint public. On 16 October it announced the first thing
built on top of them. The announcement is three sentences:

> Jais 3 is built on K2 Horizon, our six-fleet fully open model family.
> Stronger in Arabic at every size, without starting from scratch. Plus Jais
> Speech, the most accurate open Arabic ASR.

"Without starting from scratch" is the engineering claim, and it has a known
hard part. A base model's tokenizer decides how many tokens a sentence costs.
If that tokenizer was built mostly from English, Arabic text comes out in
fragments, and every fragment is context you cannot use and compute you pay
for. The usual fix is to extend the vocabulary with new Arabic tokens, which
means new embedding rows nobody trained. A reply under IFM's post asked
exactly this, and nobody answered it.

So I answered it from the files. Everything below is labelled: **measured**
(I computed it from a file or a local run), **reported** (IFM's figure, not
re-run) or **reasoned** (my arithmetic on the other two). No language model
was run; I had a shared 4-core CPU VM and no GPU. The tokenizers were run.

## The release, mapped

Jais 3 lives under the `inception42` organisation; the quantised builds and
Jais Speech are under `IFM`. Each Jais 3 model is the same-named K2 Horizon
model, continued: `K2-Horizon-7B` became `Jais-3-7B`, and so on down the list.

<ModelCard repo="inception42/Jais-3-7B" />

The first thing I checked is what a "7B" is. The safetensors headers of
`Jais-3-7B` sum to **8,999,178,240 parameters** (measured). The gap is the
vocabulary: two matrices of 250,624 rows by 4,096 columns, the input embedding
and the output head, hold 2,053,111,808 of them, and the rest is
6,946,066,432 (reasoned). The same subtraction gives 3.78B for the 3.7B and
881M for the 0.9B, so the names count the transformer, not the vocabulary
(reasoned). That matters later, because the vocabulary is where language
adaptation usually happens.

## Why adapting a language usually starts with the tokenizer

A byte-level BPE tokenizer learns merges from a corpus. Frequent byte
sequences become single tokens; rare ones stay split. Train it on English and
`" destination"` is one token, while an Arabic word is a pile of byte pairs.
The number of tokens a tokenizer spends per word is its **fertility**, and it
sets three things at once: how much text fits in the context window, how
many forward passes it takes to generate a sentence, and what an API bills
per sentence.

The previous generation of Jais faced this head-on. The 2024
`jais-adapted` models were Llama 2 continued into Arabic, and their card
describes the surgery (reported): 32,000 Arabic tokens from the Jais-30b
vocabulary were added to the Llama 2 tokenizer; the new embedding rows were
initialised by learning a linear map from Jais-30b's embedding space to
Llama's on the tokens the two vocabularies shared; and because the new rows
came in with "higher norms", the first training stage froze the whole
backbone and trained only the embeddings on about 15 billion bilingual
tokens. The card credits the new tokens with improving "fertility and compute
efficiency by over 3x". That is what adaptation costs when the base was not
built for the language.

## Jais 3 did not extend the tokenizer

At every one of the six sizes, `tokenizer.json` in the Jais 3 repository has
**the same SHA-256 as the K2 Horizon base it came from** (measured, from the
Hub's LFS hashes). `vocab_size` is unchanged in every `config.json`: 64,256
for the 0.9B and 250,624 for the other five. The 32B, 36B and 375B ship a
29 MB `tokenizer.json` and the 3.7B and 7B a 20 MB one, but loaded, the two
have identical vocabularies, merges, added tokens and pre-tokenizer
(measured); the difference is whitespace in the JSON.

It did not need to. Of the 250,624 entries in the K2 Horizon vocabulary,
**49,568 decode to Arabic script** (measured), about one in five. GPT-2 has
22. The pre-tokenizer regex also keeps Unicode combining marks attached to
the letters they modify, so a written vowel does not break a word in two at
the split stage. Somebody at IFM built the base with Arabic already in mind.

To see what that buys, I ran four tokenizers over Belebele, which is the
FLORES passages professionally translated into many languages: 488 unique
passages, the same text in English, Modern Standard Arabic, Egyptian Arabic
and Moroccan Arabic. Pick a sentence and a tokenizer below.

<FertilityComparator />

Over the whole corpus (measured):

| tokenizer | vocab | English tok/word | MSA tok/word | Egyptian | Moroccan | MSA tokens ÷ English tokens |
|---|---:|---:|---:|---:|---:|---:|
| GPT-2 | 50,257 | 1.23 | 6.02 | 5.83 | 5.90 | 4.39× |
| Qwen3 | 151,669 | 1.26 | 2.29 | 2.31 | 2.44 | 1.63× |
| K2 Horizon / Jais 3, 0.9B | 64,256 | 1.31 | 1.69 | 1.71 | 1.89 | 1.16× |
| K2 Horizon / Jais 3, 3.7B and up | 250,624 | 1.21 | 1.41 | 1.42 | 1.57 | 1.05× |

The last column is the one to read: the same 488 passages cost 48,675 tokens
in Arabic and 46,409 in English with the Jais 3 tokenizer, and 207,591
against 47,265 with GPT-2's (measured). At 512K context, 1.41 tokens per word
is room for about 371,000 words of Arabic; at GPT-2's fertility it would be
about 87,000 (reasoned). The 0.9B pays for its smaller vocabulary, 1.69
tokens per word, and Moroccan, the dialect furthest from MSA in this set,
costs about 11% more per word than MSA under the large vocabulary (reasoned).

Two places it is still expensive. Fully voweled text: my five-word test
sentence with every haraka written costs 21 tokens even in the large
vocabulary, because vowel marks rarely appear in the web text BPE merges are
learned from (measured). That matters for Jais Speech, which can emit vowel
marks on request. And the comparison is between tokenizers, not models; a
low fertility says the context and the bill are fair, not that the model
understands what it reads.

## What did change: the weights, and only the weights

If the tokenizer is untouched, "adaptation" is continued training of the
existing network. The repositories say exactly that.

<AdaptationLedger />

Every Jais 3 checkpoint has **the same tensor names and the same shapes** as
its base: 255 tensors in the 0.9B, 327 in the 3.7B and 7B, 579 in the 32B,
16,998 in the 36B and 34,076 in the 375B (measured, from the safetensors
headers). The `config.json` diffs are cosmetic: a `dtype` field on the 0.9B,
serialisation defaults on the 3.7B and 7B, an `initializer_range` of 0.01
becoming 0.02 on the 32B and 375B, and `router_aux_loss_coef` going from
0.0001 to 0.001 on the two MoE models, which is a training-time setting that
does nothing at inference (measured). Same architecture, same 512K context
(the 0.9B stays at 128K), same `modeling_k2_horizon.py` byte for byte.

The values are another story. I pulled the first 16,384 values of every
attention and MLP projection from both repositories by HTTP range request and
compared them (measured, so a sample of each matrix, not the whole matrix).
The median relative change, the norm of the difference over the norm of the
base, is about 0.40 at 0.9B, 0.19 at 3.7B, 0.16 at 7B and 0.08 at 32B (its
first 28 of 64 layers). The smaller the model, the further it moved.
Unrelated matrices of equal norm would sit near 1.41 on this measure
(reasoned), and a light fine-tune well under 0.01, as two of the stages
below show, so this is a heavy continued pre-training that stays clearly
descended from its base.

For the 0.9B I could afford to download the full embedding and output-head
matrices from both sides, 197 MB each, and split the rows by what the token
decodes to (measured):

- output-head rows for Arabic-script tokens moved a median **0.71** relative
  to their base; rows for plain-Latin tokens moved **0.37**, so nearly
  twice as far (reasoned);
- input-embedding rows: 0.47 for Arabic against 0.38 for Latin.

So the training spent most of its effort where you would hope: on the rows
that predict Arabic. Those rows existed in K2 Horizon, but a base that saw
mostly English and code had little reason to make them good.

### Where the continued training started

Jais 3 publishes its stage checkpoints as branches on each repository:
`CPT`, `Cooldown`, `SFT`, `FTPO` and `main`. K2 Horizon publishes its own as
tags. Comparing them on the 0.9B (measured, same sampling):

| from | to | median relative change |
|---|---|---:|
| K2 `pretrain_600000` | Jais `CPT` | 0.571 |
| K2 `mid_1_75000` | Jais `CPT` | 0.405 |
| K2 `mid_2_47684` | Jais `CPT` | 0.398 |
| K2 `mid_2_47684` | K2 `main` (its RL stages) | 0.0017 |
| Jais `CPT` | Jais `Cooldown` | 0.032 |
| Jais `Cooldown` | Jais `SFT` | 0.199 |
| Jais `SFT` | Jais `FTPO` | 0.0006 |

The Jais run is nearest the end of K2 Horizon's mid-training, not its
pre-training (measured). Whether it started from `mid_2_47684` or from the
released `main` I cannot tell, because K2's own RL stages moved the 0.9B by
less than 0.2%. The rest of the shape is the recipe: one large continued
pre-training step, a small cool-down, a supervised fine-tune that moves a
fifth of the norm again, and an anti-repetition preference stage
(FTPO, the same one the K2 Horizon cards describe) that barely registers.

IFM's blog gives the budget, but it was behind a bot wall from my sandbox,
so these come from AlphaSignal's summary of it (reported, second-hand):
**201.45B tokens of continued pre-training**, 151.09B of them Arabic and
50.36B replayed from K2 Horizon's own mid-training data; mixes of 90/10,
75/25 and 50/50 tested, 75/25 chosen; a 3.47B-token cool-down; and 41.1B
tokens of supervised fine-tuning. A quarter of every batch being the base's
old data is the standard defence against forgetting. If that budget applies
at 0.9B, it is about 3.6% of the 5.59T tokens K2 Horizon-0.9B was trained on
before it (reasoned, from the 0.9B card's 5T + 393B + 200B).

## "Stronger in Arabic at every size"

The claim compares Jais 3 with K2 Horizon. The only base-against-Jais number
I could find in public is the launch release's: at 7B, IFM's Arabic average
"rises from 56.4 for the K2 Horizon base model to 79.6 for Jais 3"
(reported). Per-size tables reportedly sit in the blog's appendix, which I
could not load, so **the "every size" half is unverified here**. It is not a
bold claim, though. A base trained mostly on English and code, given 151B
Arabic tokens, would be surprising if it did not improve at every size.

The stronger reading, which the coverage mostly runs with, is that Jais 3 is
the best open Arabic model at every size. The model cards answer that one,
because each compares its Jais 3 model with open peers of a similar size. I
counted the leaf Arabic rows (dropping the AraGen sub-scores and the rows
that are means of other rows) and averaged them, unweighted, over the rows
every model has (reasoned, from reported scores):

| size | Jais 3 leads (rows) | Jais 3 mean | best peer mean |
|---|---:|---:|---|
| 0.9B | 24 of 26 | 67.0 | 50.1, Qwen3.5-2B |
| 3.7B | 25 of 26 | 77.9 | 64.9, Qwen3.5-4B |
| 7B | 14 of 28 | 80.1 | 78.2, Gemma-4-12B-it |
| 32B | 7 of 28 | 84.0 | 84.2, Gemma-4-31B-it |
| MoVA-36B-A4B | 5 of 28 | 83.6 | 84.2, Gemma-4-31B-it |
| 375B-A23B | 12 of 28 | 84.6 | 83.0, Qwen3.5-397B-A17B |

The small models win outright. At 7B it is a fair fight that Jais 3 wins
against a 12B. At 32B and 36B, by IFM's own numbers, Gemma 4 31B is level or
slightly ahead, and leads 21 of 28 rows against the 32B. Jais 3 leads at
every size on only two rows, IFM's own unreleased Arabic IFBench and the
QIAS-1 Islamic-knowledge set; it usually, not always, leads on dialect
identification and on the non-hallucination half of Arabic SimpleQA, where
it abstains instead of guessing. Starred
benchmarks were run on sets IFM audited and cleaned itself, and it publishes
the excluded items, which is more than most labs do.

<Figure
  src="https://media.thesatyajit.com/ai/articles/jais-3/fig1.png?v=fafc12fa0ca711f9"
  alt="Eight bar-chart panels for Jais-3-7B. Arabic panels: General Knowledge 89.1 against Gemma-4-12B-it 88.8; STEM and Math 93.9 against Gemma-4-12B-it 94.8; Islamic Knowledge and Arab Culture 83.8 against 82.7; Dialects 77.8 against 76.0. Agentic panels: Terminal Bench 2.1 46.4, GDPVal 60.5, SWE-Bench Verified 70.2 and BrowseComp-Plus 67.7, each well ahead of Gemma 4 and Qwen3.5-9B."
  caption="Jais-3-7B against its peers. The four Arabic panels are means of the card's rows; the 12B Gemma beats it on STEM, and the agentic margins are large, presumably because the 7B inherits K2 Horizon's agent training (Jais-3-7B model card, benchmark figure)."
/>

### Is English held?

The Jais 3 cards report no English knowledge or reasoning benchmark at all:
no MMLU, no GPQA, no AIME. The only rows I can line up against the K2
Horizon cards are agentic ones, partly English (reported, across two cards
that may not share a harness):

| size | benchmark | K2 Horizon | Jais 3 |
|---|---|---:|---:|
| 7B | Terminal-Bench 2.1 | 39.1 | 46.4 |
| 7B | SWE-bench Verified | 70.6 | 70.2 |
| 32B | Terminal-Bench 2.1 | 62.6 | 63.7 |
| 36B | Terminal-Bench 2.1 | 58.6 | 67.3 |
| 375B | Terminal-Bench 2.1 | 70.2 | 73.8 |
| 375B | MCPMark | 67.7 | 68.5 |
| 375B | WildClawBench | 50.9 | 50.4 |
| 375B | Apex-Agents | 24.8 | 18.6 |

The K2 Horizon 7B card is not consistent with itself: its main table gives
39.1 and 70.6, and its FTPO table gives 44.6 and 72.4 for the released
checkpoint. Either way, agentic skill survived the Arabic training, with one
real drop on Apex-Agents at 375B. Whether English knowledge survived is not something
these cards let anyone check.

<Figure
  src="https://media.thesatyajit.com/ai/articles/jais-3/fig3.png?v=1597f0d99e2a4abe"
  alt="Eight bar-chart panels for Jais-3-375B-A23B against MiniMax M3, Nemotron 3 Ultra, Qwen3.5-397B-A17B and Qwen3.8-Flash-Next. It ties Qwen3.8-Flash-Next on General Knowledge at 92.4 and leads Islamic Knowledge at 89.4, Dialects 82.8 and MCPMark 68.5; it is last on STEM and Math at 93.6 and behind Qwen3.8-Flash-Next on Terminal Bench 2.1 (86.1), GDPVal (76.8) and WildClawBench (59.0)."
  caption="The flagship. IFM's own chart shows it last on STEM and Math and second or lower on three agentic panels, which is to its credit (Jais-3-375B-A23B model card, benchmark figure)."
/>

## What "fully open" covers here

K2 Horizon earned the phrase: its cards link the pre-training data
(`IFM/TxT360-v2`), the training code (`ifm-ai/xllm`) and dozens of
intermediate checkpoints. Jais 3 is open in a narrower sense (measured, from
the cards and the repositories on 16 October):

- **Weights:** all six under Apache 2.0, plus the stage checkpoints as
  branches. The branch checkpoints are FP32 masters, and the 3.7B's `CPT`
  branch is `pytorch_model-*.bin`, not safetensors.
- **Training logs:** a W&B project per model.
- **Code and technical report:** "Coming soon" in every card's artifact
  index.
- **Data:** the artifact index has no data row. The 151B Arabic tokens are
  not released, as far as the cards show.
- **Evaluation:** the audited benchmark sets are a public collection; the
  benchmarks marked ^ "will be released by IFM soon".

One packaging oddity: `Jais-3-MoVA-36B-A4B` on `main` is stored in FP32,
150 GB in three shards, while its config says `bfloat16` and its base is
75 GB (measured). It loads the same; it downloads twice the bytes.

## Jais Speech: an audio encoder bolted onto Jais 3.7B

The ASR model is `IFM/Jais-ASR`, and its modeling file names its parts in its
own docstring: "explicitly attributed Qwen3-Omni and K2-Horizon components".

<ModelCard repo="IFM/Jais-ASR" />

From the config and headers (measured):

- **Audio encoder:** Qwen3-Omni's audio tower, 32 layers, width 1,280, 128
  mel bins, **647,927,168 parameters**. Three stride-2 convolutions take 100
  mel frames a second down to about 13 audio tokens a second.
- **Projector:** a fresh LayerNorm, Linear, GELU, Linear from the encoder's
  2,048-wide output to the decoder's 2,560, **11,805,696 parameters**.
- **Decoder:** a dense K2 Horizon of exactly the 3.7B's shape, 36 layers,
  width 2,560, the same 250,624-token vocabulary, **5,058,255,360
  parameters**. The card says "Based on Jais-3-4B"; there is no such
  repository, and the shape is the 3.7B's.

Sampled the same way as above, the ASR decoder sits closer to `Jais-3-3.7B`
(median relative change 0.22) than to `K2-Horizon-3.7B` (0.26), which fits it
being trained from the Arabic model rather than the base (measured). The
audio tower sits at a median 0.09 from the one inside
`Qwen3-Omni-30B-A3B-Instruct`, so it is Qwen's encoder, fine-tuned, not one
trained from scratch (measured). The
whole model is 5.72B parameters and 11.4 GB in bf16, so I did not run it: it
is over the 2 GB this sandbox allows.

The claim is "the most accurate open Arabic ASR". The card's evidence is the
[Open Universal Arabic ASR Leaderboard](https://github.com/Natural-Language-Processing-Elm/open_universal_arabic_asr_leaderboard)'s
six test sets (reported):

<Figure
  src="https://media.thesatyajit.com/ai/articles/jais-3/fig2.png?v=54ca9cf246ff59db"
  alt="Horizontal bar chart of macro-average word and character error rates over six test sets. Jais-ASR 22.85 WER and 8.37 CER; Audar-ASR-V1-Turbo 23.17 and 9.23; Speechmatics Melia 1 23.51 and 10.62; ElevenLabs Scribe v2 24.54 and 9.96; cohere-transcribe-arabic-07-2026 25.87 and 11.80; omniASR-LLM-7B 28.32 and 12.52; Qwen3-Omni-30B-A3B 30.71 and 13.67; Qwen3-ASR-1.7B 33.36 and 12.33; Voxtral-Small-24B 34.47 and 15.29."
  caption="Macro-average WER and CER over SADA, Common Voice 18, MASC clean, MASC noisy, MGB-2 and Casablanca. Jais-ASR was scored by IFM with the leaderboard's script; the two commercial APIs were also measured by IFM (Jais-ASR model card, benchmark figure)."
/>

I checked the open baselines against the leaderboard Space's own results
table, last updated 20 August 2026: Audar-ASR-V1-Turbo 23.17, cohere 25.87,
omniASR-LLM-7B 28.32, Qwen3-Omni 30.71, Qwen3-ASR-1.7B 33.36 and Voxtral-Small
34.47 all match (measured). Audar-ASR-V1-Turbo has public weights on the Hub,
so it is a fair open comparator, and it is the one that matters: the margin
over it is **0.32 WER points** on the average (reasoned). Per test set,
Jais-ASR is best of the nine on three (MASC clean 16.12, MASC noisy 25.44,
MGB-2 9.59), and loses Saudi broadcast SADA to Audar (28.92 against 30.85),
Common Voice to Speechmatics (5.19), cohere (5.82) and Audar (8.09) against
its 8.56, and
the Casablanca dialect set to Speechmatics by 0.15 points (reported). On
character error rate its lead is wider, 8.37 against Audar's 9.23.

Three things the card does not settle. Jais-ASR is not on the leaderboard;
the number is IFM's run of the leaderboard's script. The leaderboard asks
entrants not to train on the training splits of its test sets, and Jais-ASR's
training data is not described, so I cannot say whether it did; MGB-2 at 9.59
against the next-best 10.30 is where I would look first. And "19 Arab
countries" is the dialect coverage claim, while the six test sets cover Saudi
broadcast, eight dialects in Casablanca, and mostly MSA elsewhere.

## The take

Jais 3 is a clean demonstration of the cheapest kind of language adaptation:
**if the base's vocabulary already covers the language, adapting it is just
training**. No new tokens, no embedding surgery, no frozen-backbone warm-up,
no change to a single tensor shape; 151B Arabic tokens mixed three to one
with the base's own data, and every downstream tool that ran K2 Horizon runs
Jais 3 unmodified. The 2024 `jais-adapted` models needed 32,000 new tokens to
get Llama 2 to the same place. The reason Jais 3 did not is a decision IFM
made months earlier, when it gave K2 Horizon a vocabulary that is one-fifth
Arabic.

The claims are mostly fine and slightly loose. "Stronger in Arabic at every
size" than the base is surely true and only shown at 7B. "Best at every size"
is what readers hear, and IFM's own tables show Gemma 4 31B level with the
32B and 36B. "Most accurate open Arabic ASR" holds by a third of a point on
IFM's own run. And "fully open" describes the family Jais 3 is built on more
than Jais 3 itself, until the data and the code arrive.

Related: [K2 Horizon MoVA](/articles/k2-horizon-mova) for the base and its
value-expert attention, [Uno](/articles/uno-diffusion-augmented) for the
lossless speed-up adapters shipped alongside every Jais 3 size,
[Tokenizers v1 and Combinatorial BPE](/articles/tokenizers-v1-combinatorial-bpe)
for how BPE merges are learned, and
[Qwen-Audio-3.1](/articles/qwen-audio-3-1) for the Qwen audio family whose
encoder Jais Speech borrows.

<ChangeMyMind>

<Falsifier claim="Jais 3 did not extend or change K2 Horizon's tokenizer at any size.">
Compared by the SHA-256 of `tokenizer.json` on the Hub for all six pairs, and by loading the 250K and 64K files. If IFM ships a revised tokenizer on a later commit, or a model loads a different tokenizer through `tokenizer_config.json`, this is wrong for that revision.
</Falsifier>

<Falsifier claim="The 250K K2 Horizon vocabulary spends 1.41 tokens per word on Modern Standard Arabic and 1.21 on English, on the same Belebele passages.">
Load `tokenizer.json` from any Jais 3 or K2 Horizon repository above 0.9B with `tokenizers`, encode Belebele's 488 unique `flores_passage` values for `arb_Arab` and `eng_Latn` without special tokens, and divide by whitespace-split words. A different corpus will give different numbers; the ratio between tokenizers is the robust part.
</Falsifier>

<Falsifier claim="Jais 3's continued pre-training started near the end of K2 Horizon's mid-training, not from its pre-training checkpoint.">
Measured only at 0.9B, and only on the first 16,384 values of each projection. A full-matrix comparison, or the same comparison at 7B with the `.bin` CPT branch converted, could move the numbers; if the technical report names a different starting checkpoint, the report wins.
</Falsifier>

<Falsifier claim="By the cards' own tables, Gemma-4-31B-it is level with or ahead of Jais-3-32B and Jais-3-MoVA-36B-A4B on Arabic.">
An unweighted mean of leaf rows. IFM's blog uses its own average and category weights, which may reverse a 0.2-point gap; if its per-size appendix shows Jais 3 ahead under a stated weighting, that is a fair reading too, just a different one.
</Falsifier>

</ChangeMyMind>

---

*Measured: tokenizer hashes, configs and safetensors headers for all twelve
base and Jais 3 repositories and for `IFM/Jais-ASR`, read on 2026-10-16;
weight deltas from HTTP range requests (the first 16,384 values of each
projection, plus the full embedding and output head of the 0.9B); fertility
from `tokenizers` 0.23.3 on a shared 4-core CPU VM over
[Belebele](https://huggingface.co/datasets/facebook/belebele) (CC-BY-SA 4.0),
with GPT-2 and Qwen3-8B as references; the ASR baselines against the
leaderboard Space's table. Reported: every benchmark score, from the Jais 3,
K2 Horizon and Jais-ASR model cards; the 7B base-against-Jais average from
the launch press release; the training-token budget from AlphaSignal's
summary of [IFM's blog](https://ifm.ai/blog/jais3/), which returned HTTP 403
to every client I had, as did [the Jais Speech post](https://ifm.ai/blog/jais-asr).
No language or speech model was run. The figures are IFM's; the fertility
comparator and the adaptation ledger are mine.*