mTREE: Multi-Level Text-Guided Representation End-to-End Learning for Whole Slide Image Analysis

Area of Science:

• Computational pathology
• Digital pathology
• Medical image analysis

Background:

• Multi-modal learning struggles with high-resolution histopathology images (gigapixel Whole Slide Images - WSIs).
• Existing methods often use manual labeling or multi-stage processes, lacking seamless end-to-end integration of multi-scale image and text data.
• Effective integration of multi-scale image representations with text data in an end-to-end framework is needed.

Purpose of the Study:

• To introduce a novel end-to-end learning framework, Multi-Level Text-Guided Representation End-to-End Learning (mTREE), for histopathology image and text analysis.
• To enable seamless integration of multi-scale Whole Slide Image (WSI) representations with textual pathology information.
• To leverage textual information for both localization of key areas and feature integration within a unified model.

Main Methods:

• Developed mTREE, a text-guided approach for capturing multi-scale WSI representations.
• Utilized textual pathology information as an attention map to identify key areas in WSIs.
• Integrated textual features with image representations in a unified, end-to-end learning framework, combining global-to-local and local-to-global strategies.
• Employed a dual role for text: localization via attention and feature integration.

Main Results:

• mTREE demonstrated effectiveness in quantitative analyses for classification and survival prediction tasks.
• The proposed mTREE approach showed significant superiority over existing baseline methods.
• Code and trained models are publicly available.

Conclusions:

• mTREE offers an effective solution for integrating multi-scale histopathology image data with textual information.
• The novel text-guided, end-to-end framework significantly improves performance in WSI analysis tasks.
• This approach advances multi-modal learning applications in digital pathology.