A pathology-attention multi-instance learning framework for multimodal classification of colorectal lesions.

Area of Science:

• Digital Pathology
• Computational Pathology
• Artificial Intelligence in Medicine

Background:

• Colorectal cancer diagnosis relies on accurate pathological assessment of whole slide images (WSIs).
• Deep learning shows potential but struggles with multimodal diagnostic processes and generalization due to staining variability and tissue heterogeneity.
• Existing weakly supervised methods lack integration of visual features and pathological knowledge.

Purpose of the Study:

• To develop a multimodal weakly supervised learning framework for five-class WSI classification of colorectal cancer.
• To address limitations of current methods by integrating visual analysis with pathological knowledge.
• To improve model generalization across different datasets and mitigate staining variability.

Main Methods:

• Proposed PAT-MIL (Pathology-Attention-MIL), a multimodal weakly supervised learning framework.
• Integrated dynamic attention mechanisms with expert-defined text prototypes for semantic guidance.
• Employed a refinement strategy for adaptive prototype distribution and a loss balancing method for optimizing visual clustering and semantic alignment.

Main Results:

• Achieved 86.45% accuracy (AUC = 0.9624) on an internal five-class dataset, outperforming ABMIL and DSMIL.
• Reached 95.78% and 84.09% accuracy on external datasets CRS-2024 and UniToPatho, respectively.
• Demonstrated superior performance over baseline methods on both internal and external validation sets.

Conclusions:

• PAT-MIL effectively mitigates staining variability and enhances cross-center generalization.
• The framework achieves robust colorectal lesion classification without pixel-level annotations.
• This work advances multimodal pathological image analysis by combining visual and textual data.