# Scaling laws in 2026: the line held, the recipe didn't
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/scaling-laws-2026
> date: 2026-08-24
> tags: scaling-laws, llm, pretraining, inference, explainer
Every retelling of scaling laws has the same shape. A 2020 paper found that model error falls on a straight line as you add compute. A 2022 paper found everyone was building models too big and feeding them too little, and fixed the ratio at about twenty tokens per parameter. Therefore AI progress is predictable, therefore the labs are reading a ruler rather than conjuring miracles.
All of that is true and it stops four years ago. The models shipping this year are trained at **200** tokens per parameter, at **60,000**, at **79,000** — and those are not sloppy approximations of twenty. They are a different question being answered. Somewhere between 2022 and now, the thing the industry optimises stopped being the thing Chinchilla optimised, and the most-quoted number in applied scaling became a correct answer that no longer matches the question.
This is the walk from what "loss" actually means, through the two famous results, to the two much less famous ones that added a third axis to the budget.
| | |
|---|---|
| The line | [Kaplan et al., arXiv:2001.08361](https://arxiv.org/abs/2001.08361) · loss falls as a power law in parameters, tokens and compute |
| The exponents | **αN = 0.076** · **αD = 0.095** · **αC = 0.050** — all small, which is the whole story |
| The cost | **C ≈ 6ND** FLOPs · GPT-3 is 3.15×10²³ of them |
| The reallocation | [Hoffmann et al., arXiv:2203.15556](https://arxiv.org/abs/2203.15556) · 400+ models · N and D should scale together, ≈20 tokens/param |
| What ships now | Llama 3 70B at **214** · Qwen3-0.6B at **60,000** · LFM2.5-350M at **79,096** |
| The functional form | the additive Chinchilla law gets the *sign* of the allocation trend wrong — [covered separately](/articles/skaling-law) |
| The third axis | [Sardana et al., arXiv:2401.00448](https://arxiv.org/abs/2401.00448) puts serving in the objective; [Roberts et al., arXiv:2604.01411](https://arxiv.org/abs/2604.01411) puts inference samples in it |
| The pace | frontier LLM training compute doubles every **≈5.2 months**; cost doubles every **≈7** |
## First, what the word "loss" refers to
Before loss is a dot on a log-log plot it is a surface. Every weight in the network is an axis, the height is how badly the model predicts the next token, and training is a walk downhill.
You cannot draw a surface in a trillion dimensions, so the pictures everyone shares are two-dimensional slices. For a long time those pictures were close to meaningless, because a network is invariant to rescaling a filter and its normalisation together — so you can make a minimum look razor-sharp or comfortably wide without changing the network's behaviour at all. [Li et al.](https://arxiv.org/abs/1712.09913) fixed that with **filter normalisation**: scale each random direction to match the norm of the filter it perturbs, and two such plots become comparable.
That paper is also the origin of the genre, and its central pair is worth seeing at full size.
Here are both shapes as turntables, rendered from scratch for this piece — the chaotic one first, then a well-conditioned bowl with momentum SGD spiralling into the floor.
a chaotic landscape · one full rotation · rendered for this article
a well-conditioned bowl, with the descent path drawing itself in
Now the part that matters for everything below, because it is the thing these pictures make people get wrong. **Scaling laws say nothing about this surface.** They do not describe the route, the roughness, or whether SGD gets stuck. They predict *how low the floor sits* once the walk is over, as a function of three numbers you choose before it starts.
## The line, and why it is shallow
Kaplan et al. fit three separate power laws, each of the form `L = (X_c / X)^α`, and the fits hold across more than seven orders of magnitude:
$$
L(N) = \left(\frac{N_c}{N}\right)^{\alpha_N},\quad
L(D) = \left(\frac{D_c}{D}\right)^{\alpha_D},\quad
L(C) = \left(\frac{C_c}{C}\right)^{\alpha_C}
$$
The straightness is what makes nine-figure training runs a planning exercise rather than a gamble: fit the line on a ladder of small cheap models, read off where the big one lands, sign the cheque. That part of the standard retelling is exactly right.
The part that usually gets dropped is the size of the exponents. **0.050 for compute** means a tenfold increase in arithmetic removes about 11% of the remaining loss. Not 11 points of benchmark score — 11% of what is left of a quantity measured in bits. Flip the interactive above to linear axes and the celebrated straight line becomes a hook that flattens almost immediately, which is a more honest picture of what buying scale feels like from the inside.
The line being straight is what makes the spending rational. The line being *shallow* is what makes it enormous. Both come from the same small number.
The other half of the arithmetic is the cost model, and it is almost insultingly simple: **C ≈ 6ND** floating-point operations to train a model of N parameters on D tokens — roughly two FLOPs per parameter for the forward pass and four for the backward. Plug in GPT-3's 175B parameters and 300B tokens and you get 3.15×10²³, which matches the published estimate. That number *is* the physical content of the phrase "GPT-3": not a mind, a quantity of arithmetic.
## Chinchilla, and the number that outlived it
For two years after Kaplan the field read the paper as advice to spend on parameters. Model size became a leaderboard and MT-NLG reached 530B.
Then DeepMind trained over four hundred models to find where loss actually bottoms out, and found the field had been building models too big and feeding them too little. The compute-optimal recipe scales N and D **together**, both roughly as the square root of compute — about twenty tokens per parameter.
They then proved it in the most humiliating way available: **Chinchilla at 70B parameters on 1.4T tokens beat Gopher, a model four times its size, trained on the same compute.**
And that is where the popular version of this story stops. Here is what the ratio has actually done since.
Llama 3 70B trained on 15T tokens — 214 tokens per parameter, more than ten times Chinchilla's recommendation. Qwen3-0.6B reached 60,000. Liquid's LFM2.5-350M put **28 trillion tokens into 354 million parameters**, a ratio of about 79,000, roughly four thousand times the number everyone quotes.
None of that is a correction to Chinchilla. Chinchilla is right about the question it asked, which was: for a fixed **training** budget, where does validation loss bottom out? A lab shipping a model to production is not asking that. It is asking where the *lifetime* cost bottoms out, and lifetime cost is dominated by inference, where the bill scales with parameters and not at all with how much the model read. [Sardana et al.](https://arxiv.org/abs/2401.00448) put serving volume into the objective and the optimum moved to roughly 100–200 tokens per parameter. Push the serving slider in the widget above and you can watch the training term stop mattering.
There is a deeper problem with Chinchilla than a stale ratio, and it is worth separating from the economics. The *functional form* the paper fits — an additive law in N and D — assumes the two contributions do not interact, which forces a cross-derivative of exactly zero by construction. Fit it to a dense grid and the residuals are not noise; they have a saddle-shaped structure. Worse, it gets the **sign** of the compute-optimal allocation trend wrong: empirical exponents of −0.14 and −0.15 against Chinchilla's +0.03. I went through that in detail in [the Skaling law piece](/articles/skaling-law); the short version is that the additive law says tokens-per-parameter should *rise* with compute and two model-free estimates say it falls.
## Does intelligence switch on?
Loss is an abstract quantity in bits. What people care about is abilities, and that is where the clean story cracks into the best fight in the field.
[Wei et al.](https://arxiv.org/abs/2206.07682) reported that certain skills are absent in small models and appear abruptly past a scale threshold — near-random, near-random, then competence. They borrowed the physics word and called it a phase transition.
[Schaeffer, Miranda and Koyejo](https://arxiv.org/abs/2304.15004) replied that the jump is often an artefact of the yardstick, and won a NeurIPS 2023 best-paper award for it. Grade a task all-or-nothing and per-token skill can climb smoothly the entire time while the score stays pinned at zero, until enough tokens line up at once.
The control is the whole argument: the dashed line — what the model is actually getting better at — never does anything interesting, and the solid line detonates. Nothing about the model differs between those two curves. The metric does.
Both sides are right about different things. The underlying loss scales smoothly and predictably; whether a human-meaningful ability *appears* to jump depends on how harshly you grade it. What the mirage paper establishes is narrower than "emergence isn't real" and sharper: the standard evidence for discontinuity is also exactly what you would see if nothing discontinuous happened, so that evidence cannot distinguish the two.
## The third axis
Kaplan and Chinchilla answer the same question in different ways, and they share an assumption that has quietly stopped being true: that a model is finished when training ends. Both optimise a two-way split of a training budget between N and D.
A model that thinks before answering, or that gets sampled twenty times with the best answer kept, is not finished when training ends. Its budget has a third term.
[Roberts et al.](https://arxiv.org/abs/2604.01411) make this explicit with what they call **T² (train-to-test) scaling laws**: jointly optimise model size, training tokens *and* the number of inference samples under one fixed end-to-end budget, with pass@k modelling the test-time half. Their finding is the one that closes the loop with the ladder above — accounting for inference **shifts the optimum well into the over-training regime**, further than pretraining scaling alone would ever recommend, and the shift survives post-training.
Read the three results together and the trajectory is coherent rather than chaotic:
- **Kaplan (2020)** — for a training budget, loss is predictable. Spend on parameters.
- **Chinchilla (2022)** — for a training budget, spend on parameters *and* tokens equally. ≈20:1.
- **Sardana (2024)** — for a lifetime budget, serving is paid per parameter. Shrink N, buy D. ≈100–200:1.
- **Roberts (2026)** — for an end-to-end budget including samples, hold compute back for inference, and over-train the smaller model you can now afford.
Each step moves in the same direction, and each one is a consequence of taking a wider slice of the real cost seriously. "Twenty tokens per parameter" is not wrong. It is the answer for a world in which a model is trained once and queried never.
## Where this actually stands
Three things are worth stating plainly, because the discourse tends to collapse them.
**The line has not broken.** Frontier LLM training compute has been doubling roughly every 5.2 months since 2020, about 0.7 orders of magnitude a year, and cost has been doubling every seven to eight months. Those are Epoch AI's numbers and they have not turned over. The commonly-quoted "doubling every six months since 2010" mixes two different series — the 4–5× per year figure covers notable models from 2010 to 2024; the frontier-LLM series is steeper and starts later.
**The data is genuinely running out**, and that is a real constraint rather than a vibe. Epoch's projection puts the usable stock of public human text at a few hundred trillion tokens and its exhaustion somewhere in the window we are now inside. You cannot keep scaling D indefinitely when D is finite, and a model trained on 28T tokens is already consuming a meaningful fraction of what exists.
**The recipe changed and the law did not.** Every headline about scaling "hitting a wall" is describing the recipe: a particular allocation of a particular budget, tuned for a world where the training run was the whole cost. The underlying claim — that loss falls predictably in the resources you spend — is doing fine. What changed is which resources count.
## The thing worth carrying
The reason scaling laws matter is not that they let a lab predict a benchmark. It is that they turn a question about intelligence into a question about allocation, and allocation questions have answers.
Which is also why the stale number is worth caring about. "Twenty tokens per parameter" spread because it is short, memorable, and sounds like a law of nature. It is none of those things — it is the solution to one optimisation problem, with one budget, under one set of assumptions about what happens after training ends. Every one of those assumptions has since been relaxed by someone, and every relaxation moved the answer by an order of magnitude or more.
The lesson is not that scaling laws are unreliable. It is that a scaling law is a *conditional* statement, and the conditions have been changing faster than the quotes. If you take one thing from this: before quoting a ratio, ask what budget it was optimising and whether that budget is the one you are actually paying.
