Jais 3: adapting K2 Horizon to Arabic without touching the tokenizer

Six weeks ago IFM shipped K2 Horizon: six open models from 0.9B to 375B, with the data, the training code and every intermediate checkpoint public. On 16 October it announced the first thing built on top of them. The announcement is three sentences:

Jais 3 is built on K2 Horizon, our six-fleet fully open model family. Stronger in Arabic at every size, without starting from scratch. Plus Jais Speech, the most accurate open Arabic ASR.

"Without starting from scratch" is the engineering claim, and it has a known hard part. A base model's tokenizer decides how many tokens a sentence costs. If that tokenizer was built mostly from English, Arabic text comes out in fragments, and every fragment is context you cannot use and compute you pay for. The usual fix is to extend the vocabulary with new Arabic tokens, which means new embedding rows nobody trained. A reply under IFM's post asked exactly this, and nobody answered it.

So I answered it from the files. Everything below is labelled: measured (I computed it from a file or a local run), reported (IFM's figure, not re-run) or reasoned (my arithmetic on the other two). No language model was run; I had a shared 4-core CPU VM and no GPU. The tokenizers were run.

The release, mapped

Jais 3 lives under the inception42 organisation; the quantised builds and Jais Speech are under IFM. Each Jais 3 model is the same-named K2 Horizon model, continued: K2-Horizon-7B became Jais-3-7B, and so on down the list.

The first thing I checked is what a "7B" is. The safetensors headers of Jais-3-7B sum to 8,999,178,240 parameters (measured). The gap is the vocabulary: two matrices of 250,624 rows by 4,096 columns, the input embedding and the output head, hold 2,053,111,808 of them, and the rest is 6,946,066,432 (reasoned). The same subtraction gives 3.78B for the 3.7B and 881M for the 0.9B, so the names count the transformer, not the vocabulary (reasoned). That matters later, because the vocabulary is where language adaptation usually happens.

Why adapting a language usually starts with the tokenizer

A byte-level BPE tokenizer learns merges from a corpus. Frequent byte sequences become single tokens; rare ones stay split. Train it on English and " destination" is one token, while an Arabic word is a pile of byte pairs. The number of tokens a tokenizer spends per word is its fertility, and it sets three things at once: how much text fits in the context window, how many forward passes it takes to generate a sentence, and what an API bills per sentence.

The previous generation of Jais faced this head-on. The 2024 jais-adapted models were Llama 2 continued into Arabic, and their card describes the surgery (reported): 32,000 Arabic tokens from the Jais-30b vocabulary were added to the Llama 2 tokenizer; the new embedding rows were initialised by learning a linear map from Jais-30b's embedding space to Llama's on the tokens the two vocabularies shared; and because the new rows came in with "higher norms", the first training stage froze the whole backbone and trained only the embeddings on about 15 billion bilingual tokens. The card credits the new tokens with improving "fertility and compute efficiency by over 3x". That is what adaptation costs when the base was not built for the language.

Jais 3 did not extend the tokenizer

At every one of the six sizes, tokenizer.json in the Jais 3 repository has the same SHA-256 as the K2 Horizon base it came from (measured, from the Hub's LFS hashes). vocab_size is unchanged in every config.json: 64,256 for the 0.9B and 250,624 for the other five. The 32B, 36B and 375B ship a 29 MB tokenizer.json and the 3.7B and 7B a 20 MB one, but loaded, the two have identical vocabularies, merges, added tokens and pre-tokenizer (measured); the difference is whitespace in the JSON.

It did not need to. Of the 250,624 entries in the K2 Horizon vocabulary, 49,568 decode to Arabic script (measured), about one in five. GPT-2 has 22. The pre-tokenizer regex also keeps Unicode combining marks attached to the letters they modify, so a written vowel does not break a word in two at the split stage. Somebody at IFM built the base with Arabic already in mind.

To see what that buys, I ran four tokenizers over Belebele, which is the FLORES passages professionally translated into many languages: 488 unique passages, the same text in English, Modern Standard Arabic, Egyptian Arabic and Moroccan Arabic. Pick a sentence and a tokenizer below.

Over the whole corpus (measured):

The last column is the one to read: the same 488 passages cost 48,675 tokens in Arabic and 46,409 in English with the Jais 3 tokenizer, and 207,591 against 47,265 with GPT-2's (measured). At 512K context, 1.41 tokens per word is room for about 371,000 words of Arabic; at GPT-2's fertility it would be about 87,000 (reasoned). The 0.9B pays for its smaller vocabulary, 1.69 tokens per word, and Moroccan, the dialect furthest from MSA in this set, costs about 11% more per word than MSA under the large vocabulary (reasoned).

Two places it is still expensive. Fully voweled text: my five-word test sentence with every haraka written costs 21 tokens even in the large vocabulary, because vowel marks rarely appear in the web text BPE merges are learned from (measured). That matters for Jais Speech, which can emit vowel marks on request. And the comparison is between tokenizers, not models; a low fertility says the context and the bill are fair, not that the model understands what it reads.

What did change: the weights, and only the weights

If the tokenizer is untouched, "adaptation" is continued training of the existing network. The repositories say exactly that.

Every Jais 3 checkpoint has the same tensor names and the same shapes as its base: 255 tensors in the 0.9B, 327 in the 3.7B and 7B, 579 in the 32B, 16,998 in the 36B and 34,076 in the 375B (measured, from the safetensors headers). The config.json diffs are cosmetic: a dtype field on the 0.9B, serialisation defaults on the 3.7B and 7B, an initializer_range of 0.01 becoming 0.02 on the 32B and 375B, and router_aux_loss_coef going from 0.0001 to 0.001 on the two MoE models, which is a training-time setting that does nothing at inference (measured). Same architecture, same 512K context (the 0.9B stays at 128K), same modeling_k2_horizon.py byte for byte.

The values are another story. I pulled the first 16,384 values of every attention and MLP projection from both repositories by HTTP range request and compared them (measured, so a sample of each matrix, not the whole matrix). The median relative change, the norm of the difference over the norm of the base, is about 0.40 at 0.9B, 0.19 at 3.7B, 0.16 at 7B and 0.08 at 32B (its first 28 of 64 layers). The smaller the model, the further it moved. Unrelated matrices of equal norm would sit near 1.41 on this measure (reasoned), and a light fine-tune well under 0.01, as two of the stages below show, so this is a heavy continued pre-training that stays clearly descended from its base.

For the 0.9B I could afford to download the full embedding and output-head matrices from both sides, 197 MB each, and split the rows by what the token decodes to (measured):

• output-head rows for Arabic-script tokens moved a median 0.71 relative to their base; rows for plain-Latin tokens moved 0.37, so nearly twice as far (reasoned);
• input-embedding rows: 0.47 for Arabic against 0.38 for Latin.

So the training spent most of its effort where you would hope: on the rows that predict Arabic. Those rows existed in K2 Horizon, but a base that saw mostly English and code had little reason to make them good.

Where the continued training started

Jais 3 publishes its stage checkpoints as branches on each repository: CPT, Cooldown, SFT, FTPO and main. K2 Horizon publishes its own as tags. Comparing them on the 0.9B (measured, same sampling):

The Jais run is nearest the end of K2 Horizon's mid-training, not its pre-training (measured). Whether it started from mid_2_47684 or from the released main I cannot tell, because K2's own RL stages moved the 0.9B by less than 0.2%. The rest of the shape is the recipe: one large continued pre-training step, a small cool-down, a supervised fine-tune that moves a fifth of the norm again, and an anti-repetition preference stage (FTPO, the same one the K2 Horizon cards describe) that barely registers.

IFM's blog gives the budget, but it was behind a bot wall from my sandbox, so these come from AlphaSignal's summary of it (reported, second-hand): 201.45B tokens of continued pre-training, 151.09B of them Arabic and 50.36B replayed from K2 Horizon's own mid-training data; mixes of 90/10, 75/25 and 50/50 tested, 75/25 chosen; a 3.47B-token cool-down; and 41.1B tokens of supervised fine-tuning. A quarter of every batch being the base's old data is the standard defence against forgetting. If that budget applies at 0.9B, it is about 3.6% of the 5.59T tokens K2 Horizon-0.9B was trained on before it (reasoned, from the 0.9B card's 5T + 393B + 200B).

"Stronger in Arabic at every size"

The claim compares Jais 3 with K2 Horizon. The only base-against-Jais number I could find in public is the launch release's: at 7B, IFM's Arabic average "rises from 56.4 for the K2 Horizon base model to 79.6 for Jais 3" (reported). Per-size tables reportedly sit in the blog's appendix, which I could not load, so the "every size" half is unverified here. It is not a bold claim, though. A base trained mostly on English and code, given 151B Arabic tokens, would be surprising if it did not improve at every size.

The stronger reading, which the coverage mostly runs with, is that Jais 3 is the best open Arabic model at every size. The model cards answer that one, because each compares its Jais 3 model with open peers of a similar size. I counted the leaf Arabic rows (dropping the AraGen sub-scores and the rows that are means of other rows) and averaged them, unweighted, over the rows every model has (reasoned, from reported scores):

The small models win outright. At 7B it is a fair fight that Jais 3 wins against a 12B. At 32B and 36B, by IFM's own numbers, Gemma 4 31B is level or slightly ahead, and leads 21 of 28 rows against the 32B. Jais 3 leads at every size on only two rows, IFM's own unreleased Arabic IFBench and the QIAS-1 Islamic-knowledge set; it usually, not always, leads on dialect identification and on the non-hallucination half of Arabic SimpleQA, where it abstains instead of guessing. Starred benchmarks were run on sets IFM audited and cleaned itself, and it publishes the excluded items, which is more than most labs do.

expand

Is English held?

The Jais 3 cards report no English knowledge or reasoning benchmark at all: no MMLU, no GPQA, no AIME. The only rows I can line up against the K2 Horizon cards are agentic ones, partly English (reported, across two cards that may not share a harness):

The K2 Horizon 7B card is not consistent with itself: its main table gives 39.1 and 70.6, and its FTPO table gives 44.6 and 72.4 for the released checkpoint. Either way, agentic skill survived the Arabic training, with one real drop on Apex-Agents at 375B. Whether English knowledge survived is not something these cards let anyone check.

expand

What "fully open" covers here

K2 Horizon earned the phrase: its cards link the pre-training data (IFM/TxT360-v2), the training code (ifm-ai/xllm) and dozens of intermediate checkpoints. Jais 3 is open in a narrower sense (measured, from the cards and the repositories on 16 October):

• Weights: all six under Apache 2.0, plus the stage checkpoints as branches. The branch checkpoints are FP32 masters, and the 3.7B's CPT branch is pytorch_model-*.bin, not safetensors.
• Training logs: a W&B project per model.
• Code and technical report: "Coming soon" in every card's artifact index.
• Data: the artifact index has no data row. The 151B Arabic tokens are not released, as far as the cards show.
• Evaluation: the audited benchmark sets are a public collection; the benchmarks marked ^ "will be released by IFM soon".

One packaging oddity: Jais-3-MoVA-36B-A4B on main is stored in FP32, 150 GB in three shards, while its config says bfloat16 and its base is 75 GB (measured). It loads the same; it downloads twice the bytes.

Jais Speech: an audio encoder bolted onto Jais 3.7B

The ASR model is IFM/Jais-ASR, and its modeling file names its parts in its own docstring: "explicitly attributed Qwen3-Omni and K2-Horizon components".

From the config and headers (measured):

• Audio encoder: Qwen3-Omni's audio tower, 32 layers, width 1,280, 128 mel bins, 647,927,168 parameters. Three stride-2 convolutions take 100 mel frames a second down to about 13 audio tokens a second.
• Projector: a fresh LayerNorm, Linear, GELU, Linear from the encoder's 2,048-wide output to the decoder's 2,560, 11,805,696 parameters.
• Decoder: a dense K2 Horizon of exactly the 3.7B's shape, 36 layers, width 2,560, the same 250,624-token vocabulary, 5,058,255,360 parameters. The card says "Based on Jais-3-4B"; there is no such repository, and the shape is the 3.7B's.

Sampled the same way as above, the ASR decoder sits closer to Jais-3-3.7B (median relative change 0.22) than to K2-Horizon-3.7B (0.26), which fits it being trained from the Arabic model rather than the base (measured). The audio tower sits at a median 0.09 from the one inside Qwen3-Omni-30B-A3B-Instruct, so it is Qwen's encoder, fine-tuned, not one trained from scratch (measured). The whole model is 5.72B parameters and 11.4 GB in bf16, so I did not run it: it is over the 2 GB this sandbox allows.

The claim is "the most accurate open Arabic ASR". The card's evidence is the Open Universal Arabic ASR Leaderboard's six test sets (reported):

expand

I checked the open baselines against the leaderboard Space's own results table, last updated 20 August 2026: Audar-ASR-V1-Turbo 23.17, cohere 25.87, omniASR-LLM-7B 28.32, Qwen3-Omni 30.71, Qwen3-ASR-1.7B 33.36 and Voxtral-Small 34.47 all match (measured). Audar-ASR-V1-Turbo has public weights on the Hub, so it is a fair open comparator, and it is the one that matters: the margin over it is 0.32 WER points on the average (reasoned). Per test set, Jais-ASR is best of the nine on three (MASC clean 16.12, MASC noisy 25.44, MGB-2 9.59), and loses Saudi broadcast SADA to Audar (28.92 against 30.85), Common Voice to Speechmatics (5.19), cohere (5.82) and Audar (8.09) against its 8.56, and the Casablanca dialect set to Speechmatics by 0.15 points (reported). On character error rate its lead is wider, 8.37 against Audar's 9.23.

Three things the card does not settle. Jais-ASR is not on the leaderboard; the number is IFM's run of the leaderboard's script. The leaderboard asks entrants not to train on the training splits of its test sets, and Jais-ASR's training data is not described, so I cannot say whether it did; MGB-2 at 9.59 against the next-best 10.30 is where I would look first. And "19 Arab countries" is the dialect coverage claim, while the six test sets cover Saudi broadcast, eight dialects in Casablanca, and mostly MSA elsewhere.

The take

Jais 3 is a clean demonstration of the cheapest kind of language adaptation: if the base's vocabulary already covers the language, adapting it is just training. No new tokens, no embedding surgery, no frozen-backbone warm-up, no change to a single tensor shape; 151B Arabic tokens mixed three to one with the base's own data, and every downstream tool that ran K2 Horizon runs Jais 3 unmodified. The 2024 jais-adapted models needed 32,000 new tokens to get Llama 2 to the same place. The reason Jais 3 did not is a decision IFM made months earlier, when it gave K2 Horizon a vocabulary that is one-fifth Arabic.

The claims are mostly fine and slightly loose. "Stronger in Arabic at every size" than the base is surely true and only shown at 7B. "Best at every size" is what readers hear, and IFM's own tables show Gemma 4 31B level with the 32B and 36B. "Most accurate open Arabic ASR" holds by a third of a point on IFM's own run. And "fully open" describes the family Jais 3 is built on more than Jais 3 itself, until the data and the code arrive.

Related: K2 Horizon MoVA for the base and its value-expert attention, Uno for the lossless speed-up adapters shipped alongside every Jais 3 size, Tokenizers v1 and Combinatorial BPE for how BPE merges are learned, and Qwen-Audio-3.1 for the Qwen audio family whose encoder Jais Speech borrows.

Measured: tokenizer hashes, configs and safetensors headers for all twelve base and Jais 3 repositories and for IFM/Jais-ASR, read on 2026-10-16; weight deltas from HTTP range requests (the first 16,384 values of each projection, plus the full embedding and output head of the 0.9B); fertility from tokenizers 0.23.3 on a shared 4-core CPU VM over Belebele (CC-BY-SA 4.0), with GPT-2 and Qwen3-8B as references; the ASR baselines against the leaderboard Space's table. Reported: every benchmark score, from the Jais 3, K2 Horizon and Jais-ASR model cards; the 7B base-against-Jais average from the launch press release; the training-token budget from AlphaSignal's summary of IFM's blog, which returned HTTP 403 to every client I had, as did the Jais Speech post. No language or speech model was run. The figures are IFM's; the fertility comparator and the adaptation ledger are mine.