ConTEXTual Net: A Multimodal Vision-Language Model for Segmentation of Pneumothorax

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Natural Language Processing

Background:

• Radiology reports contain descriptive text about disease characteristics.
• Multimodal learning shows promise in integrating text and image data.
• Accurate pneumothorax segmentation is crucial for patient diagnosis and treatment.

Purpose of the Study:

• To develop a novel vision-language model, ConTEXTual Net, for improved pneumothorax segmentation.
• To leverage descriptive text from radiology reports to guide medical image analysis.
• To evaluate the performance of ConTEXTual Net against existing models and human variability.

Main Methods:

• Proposed ConTEXTual Net, a vision-language model integrating free-form radiology reports with chest radiographs.
• Utilized a pre-trained language model for feature extraction from reports.
• Implemented cross-attention between language features and a convolutional neural network's intermediate embeddings.
• Trained and validated on the CANDID-PTX dataset comprising 3196 positive pneumothorax cases.

Main Results:

• ConTEXTual Net achieved a Dice score of 0.716±0.016, comparable to inter-reader variability (0.712±0.044).
• Outperformed vision-only models (e.g., Swin UNETR, nnUNet) and a competing vision-language model (LAVT).
• Ablation studies confirmed text information as the primary driver of performance gains.

Conclusions:

• ConTEXTual Net effectively utilizes radiology report text to enhance pneumothorax segmentation accuracy.
• The model's performance approaches human expert agreement levels.
• Image-text concordance is vital, and certain augmentations can negatively impact performance.