Shared-Specific Feature Learning With Bottleneck Fusion Transformer for Multi-Modal Whole Slide Image Analysis.

Area of Science:

• Computational pathology
• Artificial intelligence in medicine
• Precision medicine

Background:

• Accurate prediction of lymph node metastasis (LNM) in papillary thyroid carcinoma is crucial for treatment decisions and avoiding unnecessary surgeries.
• Integrating whole slide histopathological images (WSIs) with tabular clinical data offers potential for improved LNM prediction.
• High dimensionality of WSIs poses challenges for information alignment in multi-modal analysis.

Purpose of the Study:

• To develop a novel transformer-guided multi-modal multi-instance learning framework for predicting LNM.
• To effectively fuse information from WSIs and tabular clinical data for enhanced diagnostic accuracy.
• To improve pre-surgical prediction of LNM to guide surgical interventions.

Main Methods:

• Proposed a siamese attention-based feature grouping (SAG) scheme to create low-dimensional embeddings from high-dimensional WSIs.
• Developed a bottleneck shared-specific feature transfer (BSFT) module with bottleneck tokens for cross-modal knowledge transfer.
• Incorporated modal adaptation and orthogonal projection to enhance learning of shared and specific features, aggregated via attention for prediction.

Main Results:

• The proposed framework achieved an Area Under the Curve (AUC) of 97.34% on a collected LNM dataset.
• The framework outperformed existing state-of-the-art methods by over 1.27%.
• Experimental results validated the efficiency of the proposed SAG and BSFT components.

Conclusions:

• The developed transformer-guided multi-modal framework effectively integrates WSI and clinical data for accurate LNM prediction.
• The novel SAG and BSFT modules are key to handling high-dimensional WSI data and facilitating cross-modal feature transfer.
• This approach shows significant potential for improving precision medicine by enabling more accurate pre-surgical LNM assessment.