Benchmarking foundation cell models for post-perturbation RNA-seq prediction.

Area of Science:

• Computational Biology
• Genomics
• Systems Biology

Background:

• Accurate prediction of cellular responses to perturbations is vital for understanding cell behavior in health and disease.
• Foundation cell models pre-trained on large-scale single-cell gene expression data are state-of-the-art for predicting post-perturbation profiles.
• However, robust benchmarking of these models remains a significant challenge due to data limitations.

Purpose of the Study:

• To benchmark the performance of recently developed foundation cell models (scGPT, scFoundation) against baseline models for predicting gene expression after cellular perturbations.
• To identify limitations in current benchmarking methodologies and benchmark datasets for evaluating post-perturbation gene expression prediction models.

Main Methods:

• Comparative benchmarking of scGPT and scFoundation against various baseline models, including simple statistical methods and machine learning models incorporating biological features.
• Evaluation of model performance on existing Perturb-Seq benchmark datasets.

Main Results:

• Surprisingly, the simplest baseline model (mean of training examples) outperformed both scGPT and scFoundation.
• Machine learning models incorporating biologically meaningful features significantly outperformed scGPT.
• Perturb-Seq benchmark datasets were found to have low perturbation-specific variance, rendering them suboptimal for model evaluation.

Conclusions:

• Current foundation cell models may not offer significant advantages over simpler methods for post-perturbation gene expression prediction.
• Existing benchmark datasets and evaluation strategies are insufficient for accurately assessing the performance of these advanced models.
• Further research is needed to develop more effective benchmarking approaches and datasets for robust evaluation of cellular response prediction models.