Multimodal text guided network for chest CT pneumonia classification

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Respiratory Medicine

Background:

• Pneumonia is a widespread respiratory illness with significant global health impact.
• Deep learning shows promise for automated pneumonia diagnosis from medical images, but current methods face limitations.
• Slice-based and sequence-based classification methods struggle with spatial context, labor-intensive annotations, and multi-modal information integration.

Purpose of the Study:

• To develop an advanced deep learning model for accurate pneumonia classification using chest CT sequences.
• To address the limitations of existing methods by effectively integrating multi-modal information (CT images and textual reports).
• To enhance feature learning by simulating clinical diagnostic processes and leveraging semantic information from textual descriptions.

Main Methods:

• Proposed a Multi-modal Text-Guided Network (MTGNet) incorporating a sequential graph pooling network for CT sequence encoding.
• Developed a CT description encoder to learn from textual reports and a modal transfer module to generate simulated textual features.
• Employed cross-modal attention for fusing sequence-level and simulated textual representations, alongside contrastive learning for discriminative feature extraction.

Main Results:

• The MTGNet model demonstrated significant improvements in pneumonia classification performance on a self-constructed dataset.
• Effective integration of multi-modal information enhanced the model's ability to capture critical spatial and semantic features.
• The proposed approach successfully simulated clinical diagnostic workflows, leading to superior classification outcomes.

Conclusions:

• The developed MTGNet offers a powerful new approach for pneumonia classification using chest CT sequences.
• Integrating textual data with CT imaging significantly enhances diagnostic accuracy.
• The model's ability to learn discriminative features and simulate clinical reasoning holds promise for advancing automated medical image analysis.