A comparison of scRNA-seq annotation methods based on experimentally labeled immune cell subtype dataset

Area of Science:

• Computational Biology
• Immunology
• Bioinformatics

Background:

• Accurate cell-type annotation is vital for single-cell data analysis.
• Existing reference datasets often rely on computational labels, introducing potential biases.
• A need exists for robust evaluation of diverse cell annotation tools.

Purpose of the Study:

• To systematically evaluate 18 cell annotation methods using an experimentally labeled immune cell-subtype dataset.
• To assess method performance across various scenarios, including intra-dataset, inter-dataset, and unsupervised tasks.
• To identify superior methods and highlight limitations of current approaches.

Main Methods:

• Construction of a same-batch, experimentally labeled immune cell-subtype single-cell dataset.
• Systematic evaluation of 18 cell annotation methods under five distinct validation scenarios.
• Utilized Accuracy and Adjusted Rand Index (ARI) as primary evaluation metrics.

Main Results:

• Support Vector Machine (SVM), scBERT, and scDeepSort demonstrated superior performance as supervised methods.
• Seurat excelled in unsupervised clustering but showed limitations in capturing true cell-type distributions.
• Experimentally labeled data exposed weaknesses in unsupervised clustering techniques.

Conclusions:

• Experimentally validated datasets are essential for unbiased evaluation of cell annotation tools.
• Supervised methods like SVM, scBERT, and scDeepSort show promise for accurate cell-type identification.
• The study provides a valuable resource for selecting appropriate cell annotation strategies.