TabPFN Opens New Avenues for Small-Data Tabular Learning in Drug Discovery

Area of Science:

• Computational chemistry
• Machine learning in drug discovery
• Bioinformatics

Background:

• Early-stage drug discovery faces challenges with limited data and out-of-distribution (OOD) shifts, impacting predictive model reliability.
• Gradient-boosted decision trees (GBDTs) like XGBoost have dominated tabular modeling but show limited robustness in small-sample and OOD scenarios.
• Deep learning advances representation learning, but tabular models are crucial for specific data conditions.

Purpose of the Study:

• To evaluate the performance and robustness of TabPFN, a transformer-based tabular foundation model, in the context of molecular data and drug discovery.
• To compare TabPFN against traditional GBDTs (e.g., XGBoost) in classification and regression tasks, particularly under data-scarce and OOD conditions.
• To analyze the inductive biases and data efficiency of TabPFN through feature/data ablations and embedding analyses.

Main Methods:

• Application of TabPFN to diverse molecular datasets for classification and regression tasks.
• Comparative analysis of TabPFN against XGBoost on various data sizes and OOD evaluation settings.
• Feature and data ablation studies (10-90%) to assess model robustness.
• Embedding analysis to investigate model interpretability and structure-property relationships.

Main Results:

• TabPFN achieved performance on par with XGBoost in classification tasks.
• TabPFN demonstrated clear and stable advantages in regression tasks, especially on small/medium datasets and under OOD evaluations.
• Model robustness was confirmed through graceful performance degradation in ablation studies, showing less sensitivity than tree ensembles.
• On quantum tasks, TabPFN was competitive on QM7 but faced challenges with the larger QM8 dataset.

Conclusions:

• TabPFN presents a robust and data-efficient alternative for tabular learning in drug discovery, particularly beneficial for small-data and OOD challenges.
• The transformer-based architecture of TabPFN exhibits favorable inductive biases, leading to smoother structure-property relationships and enhanced class separability without overfitting.
• Further research can leverage TabPFN for improved predictive modeling in early-stage drug discovery where data limitations are prevalent.