2026-07-04 · 8 min · tabular · foundation-models · in-context-learning · explainer · transformers
For a decade the honest answer to "what should I use on this spreadsheet?" has been a gradient-boosted tree — XGBoost or LightGBM — fit fresh on each dataset. Deep learning kept losing to it on tabular data. TabFM, Google's tabular foundation model, is a bet that the transformer recipe finally transfers, but not by training a network on your table. Instead it does in-context learning: you hand it your labeled rows as context, and it predicts new rows in a single forward pass — no gradient descent on your data at all. The knowledge to do this was baked in ahead of time, by pre-training on hundreds of millions of synthetic tables.
The cleanest way to feel what that means is to watch it. Feed the model labeled example rows and it answers a test row; add rows and the prediction firms up; take them all away and it can only guess. Crucially, no training happens at any point below — the "training set" is just context:
Nothing is trained. The labeled rows are the “training set,” but they enter as context — like examples in a prompt — and the model answers the test row in a single forward pass. Add rows and the prediction firms up; drop them all and it can only guess. This is TabFM’s inheritance from TabPFN and TabICL: prediction as in-context inference, not gradient fitting.
If that shape looks familiar, it's because TabFM descends from TabPFN and TabICL — the line of prior-fitted / in-context tabular transformers that reframed prediction as inference over a prompt rather than a fit loop. TabFM scales that idea up: a bigger model, a richer synthetic prior, and an architecture built specifically for the two things tables have that text doesn't — columns with no natural order, and rows that are exchangeable examples.
The synthetic prior: learning to predict, from causal make-believe
TabFM never sees your data in training. It is trained on tables generated from structural causal models (SCMs) — random causal graphs that emit synthetic features and a label with a known generative structure. Sample hundreds of millions of such tables, each a fresh little prediction problem, and train one transformer to solve all of them in-context. What the model learns is not any particular dataset but the algorithm of tabular prediction: given some labeled rows, infer the rule and apply it to a new row. Meeting your real table at inference time is then just another draw from a distribution it has already seen a hundred million cousins of. (If synthetic-data priors interest you, the same generate-to-learn logic shows up in set diffusion.)
The architecture, in three stages
TabFM's body is a pipeline that respects tabular structure in three moves. Step through them:
Column attention: a Set Transformer attends across features (order-invariant); numeric values enter via Fourier features.
- Column attention. A Set Transformer attends across the features of a row. Because a set has no order, the model is permutation-invariant over columns — shuffle your feature order and nothing changes. Numeric values, which transformers otherwise handle poorly, are embedded with Fourier features (sin/cos of the value at several frequencies) so the network can represent magnitude and periodicity.
- Row compression. Each row, now a set of attended feature embeddings, is pooled down to a single CLS token — one vector per row. RoPE positional encoding gives the rows an order to work with, so the sequence of rows is something the next stage can index into.
- In-context transformer. A 24-block transformer reads the sequence of row tokens — the labeled context rows and the test row — and does the actual in-context learning, emitting the test row's predicted label. This is the stage that "learns" your dataset, and it does so with attention, in one pass, weights frozen.
The paper's own diagram lays out the same three stages — the alternating row/column attention that feeds row compression, then the in-context stack that predicts the missing label:
In-context vs fit-a-tree: the shape of the work
It helps to set TabFM beside the thing it wants to replace. A boosted tree has to fit your table first — a sequential loop of boosting rounds — before it can predict. TabFM has no fit loop for your data at all; the table enters as context and the answer comes out of one forward pass. Drag the rounds and watch the sequential work pile up on one side and vanish on the other:
The difference is the shape of the work, not a quality claim. A boosted tree must fit your table — a sequential loop of rounds — before it predicts; TabFM folds the table in as context and answers in a single pass, no per-dataset training. Which one predicts better is a separate question, and the honest answer lives in the Elo chart below, where the comparison is ensemble-vs-ensemble.
This is a claim about workflow, not accuracy. "No training loop" is a real ergonomic win — you stand up a predictor in one pass — but whether it predicts better than a tuned tree is a separate question, and one worth being careful about.
The numbers, and what they actually compare
TabFM is evaluated on TabArena, a leaderboard that scores tabular methods with a relative Elo across a suite of datasets. On classification, TabFM lands at 1727 Elo, and a small ensemble of TabFM runs (TabFM-Ensemble) reaches 1815 — ahead of the strongest AutoGluon configuration and the TabPFN/TabICL baselines:
The regression picture is wider: TabFM at 1940, the ensemble at 2125, clear of the field.
Here is the load-bearing caveat. The banner "beats GBDTs" rests on beating AutoGluon — an AutoML ensemble that stacks and tunes many models — not a single tuned XGBoost or LightGBM. In the headline chart there is no standalone GBDT bar; the strongest tree-based entries are the AutoGluon configurations. So the honest framing is ensemble-vs-ensemble: TabFM-Ensemble edges AutoGluon, and single TabFM is competitive with it. That is a genuinely strong result — but it is not "TabFM beats your tuned XGBoost," which the chart does not measure. The paper's full TabArena chart shows exactly this — AutoGluon as the tree-side comparison, TabFM and its ensemble on top:
The take
TabFM is the tabular field finally getting its "just add context" moment. The mechanism is worth internalizing even if the weights' license keeps you from deploying them: prediction reframed as in-context inference over a synthetic prior, with an architecture that takes tables seriously — order-invariant column attention, Fourier-embedded numerics, row-to-CLS compression, and a 24-block stack that does the learning at inference time. It inherits this from TabPFN and TabICL and pushes the scale, and the TabArena numbers say the bet largely pays off: state-of-the-art among ensembles, and a real challenger to the AutoML pipelines that have owned tabular ML.
What it is not, yet, is a drop-in XGBoost replacement for production. The non-commercial license, the ~10-class and ~500-feature ceilings, and the ensemble-framed comparison all matter, and the Elo is a leaderboard ranking, not a promise about your dataset. But as a research artifact it moves the frontier: the fit-a-tree-per-dataset era now has a serious foundation-model rival, and the interesting question is no longer "can transformers do tables" but "how far does one forward pass go before you still need to train."
Built on Google's TabFM — the Hugging Face model card and Google's accompanying blog post. There is no primary arXiv paper; figures are from the model card, and the interactive diagrams are our own illustrations of the mechanism. TabFM descends from TabPFN and TabICL; Elo scores are quoted from TabArena.