Harnessing the Power of Single-Cell Large Language Models with Parameter Efficient Fine-Tuning using scPEFT

Area of Science:

• Computational Biology
• Genomics
• Artificial Intelligence

Background:

• Single-cell large language models (scLLMs) offer powerful insights from single-cell atlases.
• However, scLLMs exhibit limitations in out-of-context applications, leading to unreliable zero-shot predictions.

Purpose of the Study:

• To introduce a novel framework, single-cell parameter-efficient fine-tuning (scPEFT), to enhance scLLM performance.
• To improve the adaptability and accessibility of scLLMs for diverse biological research.

Main Methods:

• Implemented scPEFT by integrating low-dimensional adapters into scLLMs.
• Froze the backbone model, updating only adapter parameters for efficient task adaptation with limited data.
• Reduced parameter tuning by over 96% and GPU memory usage by over 50%.

Main Results:

• scPEFT demonstrated superior performance compared to zero-shot and traditional fine-tuning methods across various datasets.
• Successfully applied to disease-specific, cross-species, and under-characterized cell population analyses.
• Attention-mechanism analysis identified COVID-related genes and novel blood cell subpopulations.

Conclusions:

• scPEFT provides an efficient and accessible solution for adapting scLLMs to specific biological tasks.
• The framework enhances the utility of scLLMs for general single-cell analyses, particularly in resource-constrained settings.
• scPEFT facilitates condition-specific biological interpretations and discoveries.