Evaluating the Utilities of Foundation Models in Single-cell Data Analysis.

Area of Science:

• Computational Biology
• Genomics
• Artificial Intelligence

Background:

• Foundation Models (FMs) are increasingly used in scientific research.
• Their application to single-cell sequencing data analysis is an emerging area.

Purpose of the Study:

• To comprehensively evaluate the performance of FMs for single-cell sequencing data analysis.
• To compare FM performance against task-specific methods.
• To provide guidelines for training and evaluating single-cell FMs.

Main Methods:

• Experimental evaluation of ten single-cell FMs across eight downstream tasks.
• Comparison with established task-specific methods.
• Development and application of the scEval framework for hyper-parameter and stability analysis.

Main Results:

• scGPT, Geneformer, and CellPLM identified as top-performing and accessible FMs.
• Single-cell FMs do not consistently outperform task-specific methods across all evaluated tasks.
• scEval framework provides insights into training stability and hyper-parameter effects.

Conclusions:

• The necessity of developing single-cell FMs is questioned due to performance variability compared to task-specific methods.
• Guidelines for pre-training and fine-tuning FMs are proposed to enhance performance.
• A freely available evaluation pipeline (scEval) is offered for benchmarking and developing new single-cell FMs.