TabFM Evaluation
PythonPyTorch + JAXXGBoost + OptunaMulti-GPU testing
I evaluated Google’s TabFM across ten public datasets and three machines, then followed the failures far enough to find a bug in the public prediction path.
What I wanted to know
TabFM makes a useful promise: one pretrained model can handle classification and regression on new tables without task-specific training. I wanted to know where that held up, how it compared with a tuned XGBoost baseline, and what the memory and latency costs looked like outside the authors’ setup.
What I did
- Built a reproducible benchmark across CPU, single-GPU, and dual-GPU machines.
- Compared TabFM with tuned XGBoost on ten classification and regression datasets.
- Tested context scaling, accelerator memory, and latency rather than reporting accuracy alone.
- Re-ran the strongest and weakest results across multiple seeds before treating them as real.
What changed
The public predict API crashed on every multi-GPU host I tested. I traced the issue to a device-placement mismatch, wrote a focused fix and a CPU regression test, and sent it upstream. One of TabFM’s authors reviewed the pull request, and Google Research merged it.
The evaluation also clarified the practical trade-off. TabFM was competitive across the benchmark and stable across seeds, but the pinned JAX path reserved roughly 17 GB even for small contexts. The later PyTorch path used much less memory and scaled to larger tables.
The point of the project was not to declare a winner. It was to turn a new research release into evidence a team could use: where it works, what it costs, and what still breaks.