Zero-shot evaluation reveals limitations of single-cell foundation models

Area of Science:

• Computational biology
• Single-cell genomics
• Artificial intelligence in life sciences

Background:

• Foundation models (e.g., scGPT, Geneformer) are increasingly used in single-cell genomics.
• Their performance without fine-tuning (zero-shot) is critical for discovery applications with unknown labels.
• Rigorous evaluation of zero-shot capabilities is lacking.

Purpose of the Study:

• To evaluate the zero-shot performance of scGPT and Geneformer.
• To compare their reliability against simpler methods in single-cell data analysis.
• To emphasize the importance of zero-shot assessments for foundation models in this field.

Main Methods:

• Assessed scGPT and Geneformer models in a zero-shot learning scenario.
• Compared their performance on single-cell datasets without task-specific fine-tuning.
• Benchmarked against traditional or simpler analytical approaches.

Main Results:

• Zero-shot performance of scGPT and Geneformer exhibited reliability challenges in certain contexts.
• These advanced models were sometimes outperformed by less complex methods.
• The effectiveness varied depending on the specific single-cell data and task.

Conclusions:

• Zero-shot evaluation is crucial for understanding the practical utility of foundation models in single-cell research.
• Current foundation models may not always be reliable or superior in zero-shot scenarios.
• Future development and deployment should prioritize robust zero-shot performance assessments.