Limitations of cell embedding metrics assessed using drifting islands

Area of Science:

• Computational biology
• Single-cell genomics
• Bioinformatics

Background:

• Single-cell data analysis relies on embeddings to represent complex biological information.
• Assessing the quality of these embeddings is crucial but remains a significant challenge.
• Existing evaluation metrics may not fully capture the fidelity of biological structures within embeddings.

Purpose of the Study:

• To evaluate the performance of current embedding evaluation metrics.
• To introduce a novel deep learning model, Islander, for single-cell embedding.
• To develop a new metric, scGraph, for assessing embedding quality and detecting distortions.

Main Methods:

• Trained a three-layer perceptron model named Islander.
• Evaluated Islander against leading embedding methods on diverse cell atlases.
• Developed and applied the scGraph metric to identify distortions in biological structures within embeddings.

Main Results:

• Islander demonstrated superior performance compared to existing embedding methods across various cell atlases.
• Islander was observed to distort crucial biological structures, limiting its utility for discovery.
• The scGraph metric effectively flagged these distortions, indicating its potential for quality control.

Conclusions:

• Current evaluation metrics for single-cell embeddings are insufficient.
• While Islander excels in embedding tasks, its distortion of biological data necessitates careful quality assessment.
• The scGraph metric offers a valuable tool for ensuring the biological relevance of single-cell embeddings.