Evaluating the Utilities of Foundation Models in Single-Cell Data Analysis.

Area of Science:

• Computational Biology
• Genomics
• Artificial Intelligence

Background:

• Foundation Models (FMs) are increasingly used in scientific research.
• Single-cell sequencing data analysis is a complex field with growing data volumes.

Purpose of the Study:

• To comprehensively evaluate Foundation Models for single-cell sequencing data analysis.
• To compare FM performance against task-specific methods and assess their necessity.
• To provide guidelines for training and fine-tuning single-cell FMs.

Main Methods:

• Experimental evaluation of ten single-cell FMs across eight downstream tasks.
• Comparison of FM performance with established task-specific methods.
• Analysis of hyperparameter effects, initial settings, and training stability using the scEval framework.

Main Results:

• scGPT, Geneformer, and CellFM identified as top-performing FMs based on performance and accessibility.
• Single-cell FMs did not consistently outperform task-specific methods across all evaluated tasks.
• Identified key factors influencing single-cell FM training and performance.

Conclusions:

• The consistent superiority of single-cell FMs over task-specific methods is not yet established, questioning their universal necessity.
• Guidelines for pre-training and fine-tuning are provided to enhance single-cell FM performance.
• An open-source evaluation pipeline is offered for benchmarking and advancing single-cell analysis methods.