Weakly supervised multi-modal contrastive learning framework for predicting the HER2 scores in breast cancer

Area of Science:

• Computational pathology
• Artificial intelligence in oncology
• Biomarker quantification

Background:

• Human Epidermal growth factor Receptor 2 (HER2) is a crucial biomarker for breast cancer prognosis and treatment selection.
• Current HER2 scoring relies on pathologist observation of immunohistochemistry (IHC) images, which is subjective and labor-intensive.
• Existing computational methods often use unimodal data (H&E or IHC) and fail to effectively integrate multi-modal information for improved HER2 scoring.

Purpose of the Study:

• To develop a novel weakly supervised multi-modal contrastive learning (WSMCL) framework for accurate HER2 score prediction at the whole slide image (WSI) level.
• To leverage joint learning from H&E and IHC modalities under weak WSI label supervision.
• To enhance feature learning by integrating information from different imaging modalities.

Main Methods:

• Feature extraction from patches within H&E and IHC WSIs.
• Application of multi-head self-attention (MHSA) to capture global patch dependencies within each modality.
• Selection of top-k and bottom-k attention score patches as candidates for multi-modal learning.
• Implementation of a multi-modal attentive contrastive learning (MACL) module for semantic alignment of features across modalities.

Main Results:

• The proposed WSMCL framework demonstrates superior HER2 scoring performance compared to existing methods.
• Experimental results validate the effectiveness of multi-modal joint learning for HER2 score prediction.
• The WSMCL approach achieves better accuracy by integrating information from both H&E and IHC images.

Conclusions:

• The WSMCL framework offers a robust and accurate solution for automated HER2 scoring in breast cancer.
• Integrating multi-modal data through contrastive learning significantly improves HER2 scoring performance.
• This approach has the potential to reduce pathologist workload and improve diagnostic consistency in breast cancer management.