Can a Foundation Model Match XGBoost on Tabular Data?

Deep learning no longer loses tabular by default.

[Ideas are mine. Blog written by AI and reviewed by the author]

No gradient boosting. No feature engineering worth the name. No training loop at all. One tabular foundation model, $3.30 of cloud compute, and a validation ROC AUC of 0.779 on a credit-risk problem — inside the band where carefully tuned XGBoost pipelines usually live.

This is a write-up of a small experiment I ran for an internal modeling competition, and an argument that deep learning has quietly become a serious contender on tabular data — the one domain it was never supposed to win.

The problem

The task is classic credit risk: predict early installment delinquency for loan applicants. The data is the standard shape for this industry:

• 246,008 training applications, 8.1% positive rate

• 61,503 test applications to score

• a main application table (~120 columns: demographics, income, credit amounts, external bureau scores, housing stats)

• six auxiliary tables — bureau records, monthly bureau balances, prior applications, POS/cash monthly snapshots, card monthly snapshots, installment payment history

• scored by ROC AUC on a held-out leaderboard

If you have worked on problems of this shape, you know the playbook: spend days engineering hundreds of features from the auxiliary tables, feed them to LightGBM or XGBoost, tune, ensemble, repeat. The feature engineering is the modeling. Gradient-boosted trees have owned this territory for a decade, and every serious leaderboard for this kind of data is a monument to them.

The contender

I used TabFM (v1.0.0, the PyTorch release) — a tabular foundation model. The idea, for anyone who has not met this family: the model is a large transformer (6.5GB of fp32 weights) pre-trained across a vast space of synthetic tabular tasks. You do not fine-tune it. “Fitting” means handing it a sample of labeled training rows as context; prediction is a single forward pass in which the model attends from your query rows to that context and emits class probabilities. In-context learning, but for tables.

Two knobs matter:

• context rows — how many labeled examples the model sees per forward pass (I went up to 6,000)

• n_estimators — how many independent context subsamples get ensembled (I went up to 8)

That is the entire hyperparameter surface I touched. random_state=42, defaults everywhere else.

Scaling works, and features still matter

I climbed a small ladder of configurations, validating each on a stratified holdout:

Two things jump out.

First, the scaling is monotonic. More ensemble members and more context rows kept buying AUC, and the curve had not flattened when I stopped — I stopped because of memory ceilings, not because the model did.

Second, the biggest single jump (+0.027) came from the laziest feature engineering imaginable. I did not craft domain features. I wrote one 60-line script that applies the identical recipe to every auxiliary table: group by case ID, take mean, max, and last of every numeric column, plus a row count. No ratios, no time windows, no domain knowledge, no per-table tuning. That took the feature count from ~120 to 306 and the AUC from 0.752 to 0.779.

In the GBDT playbook, the gap between mechanical aggregates and expert features is where competitions are won and months are spent. The foundation model closed a large chunk of that gap on its own — it seems to need the information joined into the row, but it finds the interactions itself.

So does it match XGBoost?

Honest answer: I did not run a tuned XGBoost head-to-head in this budget, so I will not claim a controlled victory. But context is useful. On public problems of exactly this shape, strong GBDT solutions with heavy, hand-crafted feature engineering typically land in the high 0.77s to low 0.79s AUC. A zero-shot model with deliberately dumb features validating at 0.779, with an unflattened scaling curve, is in that conversation — and it got there in one afternoon of setup rather than weeks of feature iteration.

The claim I am comfortable making: deep learning no longer loses tabular by default. The trade has changed from “GBDT wins, use it” to a genuine engineering choice:

• GBDT: cheap millisecond inference, mature tooling, but the accuracy lives in your feature pipeline.

• Tabular FM: near-zero modeling effort and no training, but you pay at inference time — every prediction is a large-transformer forward pass over thousands of context rows.

Some Other Projects on the Same Dataset