Driven by textual knowledge: A Text-View Enhanced Knowledge Transfer Network for lung infection region segmentation

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Computer Vision

Background:

• Lung infections are a leading cause of death, necessitating accurate segmentation of infected areas for effective treatment.
• Current imaging-based segmentation methods lack sufficient accuracy.
• Integrating expert knowledge from text data offers a promising approach to improve segmentation, but challenges remain in encoding and cross-modal transfer.

Purpose of the Study:

• To develop a novel network, Text-View Enhanced Knowledge Transfer Network (TVE-Net), for improved lung infection segmentation.
• To enhance the model's perception of infected lung locations by leveraging spatial information from textual data.
• To address semantic space inconsistency between text and image features for better cross-modal information transfer.

Main Methods:

• TVE-Net generates a 'text view' by modeling infected area locations using a positional probability function, explicitly integrating spatial information.
• A unified image encoder maps both text and image features into the same space.
• A self-supervised constraint and a multi-stage knowledge transfer module with cross-attention are employed for robustness and feature correlation learning.

Main Results:

• TVE-Net significantly outperforms unimodal and multimodal methods in lung infection segmentation.
• The network achieves substantial improvements in both fully supervised and semi-supervised settings.
• Experiments were validated on the QaTa-COV19 and MosMedData+ datasets.

Conclusions:

• The proposed TVE-Net effectively integrates textual spatial information with imaging data for superior lung infection segmentation.
• This approach enhances the accuracy and robustness of segmentation models, particularly in semi-supervised scenarios.
• TVE-Net represents a significant advancement in leveraging multimodal data for medical image analysis.