Large Models for Small Tables: Adapting Tabular Foundation Models to EHR Data

Electronic Health Records (EHR) contain abundant structured tabular data, yet developing accurate predictive models on small clinical datasets remains challenging. Foundation models have transformed natural language and imaging tasks by leveraging pretraining on large-scale data, and they are increasingly explored in healthcare as a route toward generalist medical AI. However, this paradigm remains largely unexplored for structured EHR data. In this study, we explore the adaptation of a tabular foundation model to EHR predictive tasks. We fine-tune TabPFN, a Transformer-based model pre-trained on diverse synthetic tabular datasets, to capture complex patterns in clinical data that traditional models might miss. Our results demonstrate that the fine-tuned foundation model consistently out-performs conventional methods, achieving higher performance across tabular clinical prediction tasks and suggesting that large models for small tables can advance healthcare AI by combining rich prior knowledge with domain-specific fine-tuning.