Multimodal Gated Mixture of Experts Using Whole Slide Image and Flow Cytometry for Multiple Instance Learning...

Area of Science:

• Digital Pathology
• Computational Biology
• Machine Learning

Background:

• Malignant lymphoma diagnosis relies on integrating various data types.
• Flow cytometry (FCM) provides valuable auxiliary information for classifying lymphoma subtypes.
• Digital pathology utilizes whole slide images (WSIs) for detailed morphological analysis.

Purpose of the Study:

• To develop a multimodal deep learning classification method for lymphoma diagnosis.
• To enhance diagnostic explainability by mimicking pathologist workflows.
• To leverage both WSIs and FCM data for improved classification accuracy.

Main Methods:

• A deep learning framework combining Mixture of Experts (MoE) and Multiple Instance Learning (MIL).
• MoE network with a gating network for superclass and expert networks for subclass classification.
• MIL effectively handles WSIs, while MoE integrates hierarchical data structures.

Main Results:

• Achieved 72.3% classification accuracy on a six-class lymphoma dataset.
• Outperformed methods using only images (70.2%) or a simple combination of FCM and images (69.5%).
• The model provides visualizations of cellular and tumor regions, enhancing interpretability.

Conclusions:

• The proposed multimodal deep learning method significantly improves lymphoma classification accuracy.
• Integration of WSIs and FCM data enhances diagnostic explainability.
• The method shows promise for real-world lymphoma diagnosis with potential for expansion to more classes.