# TabFM: a foundation model that learns tables in-context

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/tabular-foundation-model
> date: 2026-07-04
> tags: tabular, foundation-models, in-context-learning, explainer, transformers
For a decade the honest answer to "what should I use on this spreadsheet?" has been a
gradient-boosted tree — XGBoost or LightGBM — fit fresh on each dataset. Deep learning kept
losing to it on tabular data. **TabFM**, Google's tabular foundation model, is a bet that the
transformer recipe finally transfers, but not by training a network on *your* table. Instead it
does **in-context learning**: you hand it your labeled rows as *context*, and it predicts new rows
in a single forward pass — no gradient descent on your data at all. The knowledge to do this was
baked in ahead of time, by pre-training on **hundreds of millions of synthetic tables**.

The cleanest way to feel what that means is to watch it. Feed the model labeled example rows and
it answers a test row; add rows and the prediction firms up; take them all away and it can only
guess. Crucially, **no training happens** at any point below — the "training set" is just context:

<IclDemo />

If that shape looks familiar, it's because TabFM descends from **TabPFN** and **TabICL** — the
line of *prior-fitted* / in-context tabular transformers that reframed prediction as inference over
a prompt rather than a fit loop. TabFM scales that idea up: a bigger model, a richer synthetic
prior, and an architecture built specifically for the two things tables have that text doesn't —
columns with no natural order, and rows that are exchangeable examples.

## The synthetic prior: learning to predict, from causal make-believe

TabFM never sees your data in training. It is trained on tables **generated from structural causal
models (SCMs)** — random causal graphs that emit synthetic features and a label with a known
generative structure. Sample hundreds of millions of such tables, each a fresh little prediction
problem, and train one transformer to solve all of them *in-context*. What the model learns is not
any particular dataset but the **algorithm** of tabular prediction: given some labeled rows, infer
the rule and apply it to a new row. Meeting your real table at inference time is then just another
draw from a distribution it has already seen a hundred million cousins of. (If synthetic-data
priors interest you, the same generate-to-learn logic shows up in [set diffusion](/articles/set-diffusion).)

## The architecture, in three stages

TabFM's body is a pipeline that respects tabular structure in three moves. Step through them:

<PipelineStages />

1. **Column attention.** A **Set Transformer** attends *across the features* of a row. Because a
   set has no order, the model is **permutation-invariant** over columns — shuffle your feature
   order and nothing changes. Numeric values, which transformers otherwise handle poorly, are
   embedded with **Fourier features** (sin/cos of the value at several frequencies) so the network
   can represent magnitude and periodicity.
2. **Row compression.** Each row, now a set of attended feature embeddings, is pooled down to a
   single **CLS token** — one vector per row. **RoPE** positional encoding gives the rows an order
   to work with, so the sequence of rows is something the next stage can index into.
3. **In-context transformer.** A **24-block** transformer reads the sequence of row tokens — the
   labeled context rows *and* the test row — and does the actual in-context learning, emitting the
   test row's predicted label. This is the stage that "learns" your dataset, and it does so with
   attention, in one pass, weights frozen.

The paper's own diagram lays out the same three stages — the alternating row/column attention that
feeds row compression, then the in-context stack that predicts the missing label:

<Figure
  src="/articles/tabular-foundation-model/fig1.png"
  alt="TabFM architecture. Left: a table of training rows and one test row, features in columns with a Label column; the test row's label is a green cell marked with a question mark. Middle: 'Alternating Row and Column Attention', drawn as grids of blue nodes with curved attention arrows within rows and down columns. Right: three 'Row Embedding' blocks under 'In-Context Learning', connected top to bottom, with the final block emitting 'Predict Missing Label' in green."
  caption="Column and row attention build a per-row embedding; a stack of in-context blocks reads the labeled rows plus the test row and predicts its missing label — no gradient training on the table (model card, Figure 1)."
/>

## In-context vs fit-a-tree: the shape of the work

It helps to set TabFM beside the thing it wants to replace. A boosted tree has to *fit* your table
first — a sequential loop of boosting rounds — before it can predict. TabFM has no fit loop for
your data at all; the table enters as context and the answer comes out of one forward pass. Drag
the rounds and watch the sequential work pile up on one side and vanish on the other:

<IclVsTrain />

This is a claim about **workflow**, not accuracy. "No training loop" is a real ergonomic win — you
stand up a predictor in one pass — but whether it *predicts better* than a tuned tree is a separate
question, and one worth being careful about.

## The numbers, and what they actually compare

TabFM is evaluated on **TabArena**, a leaderboard that scores tabular methods with a **relative Elo**
across a suite of datasets. On classification, TabFM lands at **1727** Elo, and a small ensemble of
TabFM runs (**TabFM-Ensemble**) reaches **1815** — ahead of the strongest AutoGluon configuration
and the TabPFN/TabICL baselines:

<BenchBars
  title="TabArena · classification Elo (higher = better, relative)"
  bars={[
    { label: "TabFM-Ensemble", value: 1815, highlight: true },
    { label: "TabFM", value: 1727, highlight: true },
    { label: "AutoGluon 1.5", value: 1666 },
    { label: "TabPFN-3", value: 1639 },
    { label: "TabICLv2", value: 1576 },
  ]}
/>

The regression picture is wider: TabFM at **1940**, the ensemble at **2125**, clear of the field.

<BenchBars
  title="TabArena · regression Elo (higher = better, relative)"
  bars={[
    { label: "TabFM-Ensemble", value: 2125, highlight: true },
    { label: "TabFM", value: 1940, highlight: true },
    { label: "TabPFN-3", value: 1802 },
    { label: "AutoGluon 1.5", value: 1786 },
    { label: "TabDPT", value: 1722 },
  ]}
/>

Here is the load-bearing caveat. The banner "beats GBDTs" rests on beating **AutoGluon** — an
*AutoML ensemble* that stacks and tunes many models — not a single tuned XGBoost or LightGBM. In
the headline chart there is **no standalone GBDT bar**; the strongest tree-based entries are the
AutoGluon configurations. So the honest framing is **ensemble-vs-ensemble**: TabFM-Ensemble edges
AutoGluon, and single TabFM is competitive with it. That is a genuinely strong result — but it is
not "TabFM beats your tuned XGBoost," which the chart does not measure. The paper's full TabArena
chart shows exactly this — AutoGluon as the tree-side comparison, TabFM and its ensemble on top:

<Figure
  src="/articles/tabular-foundation-model/fig2.png"
  alt="Two horizontal-titled bar charts of TabArena Elo. Top, classification: TabFM-Ensemble 1815 and TabFM 1727 lead, then AutoGluon 1.5 (extreme, 4h) 1666, TabPFN-3 1639, AutoGluon 1.4 1623, TabPFN-2.6 1585, TabICLv2 1576, RealTabPFN-2.5 1566, RealMLP 1469, TabM 1440. Bottom, regression: TabFM-Ensemble 2125 and TabFM 1940 lead, then TabPFN-3 1802, AutoGluon 1.5 1786, RealTabPFN-2.5 1752, TabPFN-2.6 1751, TabDPT 1722, AutoGluon 1.4 1688, TabICLv2 1687, RealMLP 1654."
  caption="TabArena Elo for classification (top) and regression (bottom). The tree-based comparison is AutoGluon's AutoML ensemble, not a standalone GBDT; TabFM-Ensemble leads and single TabFM is competitive (model card, Figure 2)."
/>

<Callout type="warn">
**Read the fine print before you reach for it.** (1) **License:** the released weights are
**non-commercial** — fine for research, not for shipping a product. (2) **Ensemble-vs-ensemble:**
the "beats GBDTs" comparison is against AutoGluon's AutoML ensemble; there is no standalone
XGBoost/LightGBM bar in the headline chart. (3) **~10-class ceiling:** the classifier handles up to
about 10 classes. (4) **~500-feature ceiling:** it does not scale to very wide tables. (5) **Elo is
relative** — a ranking on TabArena's specific dataset suite, not an absolute accuracy guarantee on
*your* data. Validate on a held-out slice of your own table before trusting it.
</Callout>

## The take

TabFM is the tabular field finally getting its "just add context" moment. The mechanism is worth
internalizing even if the weights' license keeps you from deploying them: prediction reframed as
**in-context inference** over a synthetic prior, with an architecture that takes tables seriously —
order-invariant column attention, Fourier-embedded numerics, row-to-CLS compression, and a
24-block stack that does the learning at inference time. It inherits this from
[TabPFN and TabICL](/articles/how-transformers-attention-works) and pushes the scale, and the
TabArena numbers say the bet largely pays off: state-of-the-art *among ensembles*, and a real
challenger to the AutoML pipelines that have owned tabular ML.

What it is not, yet, is a drop-in XGBoost replacement for production. The non-commercial license,
the ~10-class and ~500-feature ceilings, and the ensemble-framed comparison all matter, and the
Elo is a leaderboard ranking, not a promise about your dataset. But as a research artifact it moves
the frontier: the fit-a-tree-per-dataset era now has a serious foundation-model rival, and the
interesting question is no longer "can transformers do tables" but "how far does one forward pass
go before you still need to train."

---

*Built on Google's **TabFM** — the [Hugging Face model card](https://huggingface.co/google/tabfm)
and Google's accompanying [blog post](https://research.google/blog/). There is no primary arXiv
paper; figures are from the model card, and the interactive diagrams are our own illustrations of
the mechanism. TabFM descends from [TabPFN](https://arxiv.org/abs/2207.01848) and TabICL; Elo scores
are quoted from TabArena.*