Deep Learning-Based Quality Control Using Subcellular RNA Spatial Distribution Patterns for Cell Segmentation in...

Area of Science:

• Computational Biology
• Genomics
• Bioinformatics

Background:

• Sequencing-based spatial transcriptomics (sST) offers high-resolution transcriptome analysis at the subcellular level.
• Accurate cell segmentation remains a challenge in sST data analysis, hindering precise RNA spot assignment.
• Existing methods lack robust quality control for cell segmentation in sST.

Purpose of the Study:

• To develop a deep learning-based quality control (QC) method for cell segmentation in sST data.
• To improve the accuracy and reliability of cell segmentation results.
• To enhance the overall analysis of spatial transcriptomics data.

Main Methods:

• A deep neural network was designed to analyze subcellular RNA distribution patterns.
• The method identifies partially segmented and merged cells based on RNA spatial characteristics.
• The QC approach was integrated with a Transformer-based segmentation model, using it to refine training datasets.

Main Results:

• The deep learning method effectively assesses the quality of segmented cells in sST data.
• Partially segmented and merged cells were accurately identified, addressing common segmentation issues.
• Integrating the QC method with Transformer-based segmentation improved overall cell segmentation performance.

Conclusions:

• The proposed deep learning method provides a novel approach for quality control and enhancement of cell segmentation in sST.
• This technique addresses key challenges in assigning RNA spots to cells, improving data interpretation.
• The method demonstrates significant potential for advancing spatial transcriptomics research using real and synthetic datasets.