Enhanced Precision of Fluorescence In Situ Hybridization (FISH) Analysis Using Neural Network-Based Nuclear...

Area of Science:

• Digital pathology
• Biomedical imaging
• Genomics

Background:

• Accurate nuclear segmentation is crucial for interpreting fluorescence in situ hybridization (FISH) results.
• Traditional 2D automated algorithms struggle with dense or overlapping nuclei in samples like lymphomas, losing vital spatial depth information.
• This limitation impacts the reliability of diagnostic FISH analyses.

Purpose of the Study:

• To evaluate if AI-based 3D nuclear segmentation can enhance the accuracy, reproducibility, and diagnostic reliability of FISH analysis.
• To compare the performance of different AI algorithms (NucleAIzer, StarDist, Cellpose) and traditional methods for nuclear segmentation in FISH.

Main Methods:

• Formalin-fixed follicular lymphoma sections were labeled for BCL2 gene rearrangements using FISH.
• Sections were scanned in multilayer Z-stacks to capture 3D information.
• AI algorithms (NucleAIzer, StarDist, Cellpose) and FISHQuant were compared against manual eye control for nuclear segmentation accuracy.

Main Results:

• 2D segmentation methods and FISHQuant showed limitations with dense nuclei and low-intensity signals.
• AI-driven 3D segmentation improved nuclear separation and signal localization across focal planes.
• NucleAIzer and StarDist demonstrated superior precision, reduced variance (VP/VS ≈ 0.96), and strong gene spot correlation (r > 0.82).

Conclusions:

• Inaccurate nuclear segmentation hinders automated FISH signal evaluation.
• Deep learning 3D segmentation models, specifically NucleAIzer and StarDist, overcome limitations of 2D methods.
• These AI approaches enhance nuclear detection consistency, leading to improved classification of gene aberrations in automated digital pathology.