Modality Attention and Sampling Enables Deep Learning with Heterogeneous Marker Combinations in Fluorescence...

Area of Science:

• Biomedical imaging
• Computational biology
• Machine learning applications

Background:

• Fluorescence microscopy is crucial for cellular and anatomical analysis, often requiring quantitative image analysis.
• Deep learning (DL) in fluorescence microscopy is underexploited due to the high cost of training marker-specific models.
• Existing DL models are typically limited to specific marker combinations and experimental setups.

Purpose of the Study:

• To develop a flexible and efficient deep learning approach for fluorescence microscopy image analysis.
• To enable training DL models with heterogeneous datasets containing various marker combinations.
• To allow prospective application of trained models to arbitrary subsets of markers.

Main Methods:

• Proposed 'Marker Sampling and Excite' (MSE), a novel neural network approach.
• Incorporated a modality sampling strategy and a novel attention module within the MSE framework.
• Trained a single neural network on heterogeneous datasets with varying marker combinations.

Main Results:

• The single MSE network achieved performance comparable to an ensemble of networks trained for each marker combination separately.
• Demonstrated feasibility in high-throughput analysis by revising bone marrow vasculature characterization.
• Validated the approach on a distinct dataset of microvessels in fetal liver tissues.

Conclusions:

• The MSE approach significantly improves the utility of deep learning in fluorescence microscopy.
• This framework offers a solution for handling incomplete data acquisitions and missing modalities.
• The method has potential applications beyond microscopy in fields with similar data challenges.