ConTEXTual Net 3D: Vision-Language Modeling in PET/CT for Visual Grounding of Positive Findings

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Radiology

Background:

• Vision-language models (VLMs) enable visual grounding, linking text descriptions to image locations.
• VLMs have potential in radiology reporting, but lack sufficient annotated PET/CT datasets.
• Developing effective VLMs for PET/CT requires novel approaches to data generation and model training.

Purpose of the Study:

• To develop an automated pipeline for generating weak image-text labels from PET/CT reports.
• To train a 3D visual grounding model (ConTEXTual Net 3D) using the generated weak labels.
• To evaluate the performance of ConTEXTual Net 3D against existing models and radiologists.

Main Methods:

• An automated pipeline identified sentences with SUVmax and slice numbers in PET/CT reports to create weak labels.
• Lesion masks were automatically generated and paired with corresponding text descriptions.
• A 3D visual grounding model, ConTEXTual Net 3D, was trained on 11,356 PET/CT image-text pairs.

Main Results:

• The weak-labeling pipeline achieved 98% accuracy in identifying lesion locations.
• ConTEXTual Net 3D achieved an F1 score of 0.80, significantly outperforming LLMSeg (0.22) and a 2.5D model (0.53).
• Performance varied by radiotracer, with higher scores for 18F-FDG and DCFPyL compared to DOTATATE and 18F-fluciclovine.

Conclusions:

• The novel weak-labeling pipeline successfully created an annotated PET/CT dataset.
• ConTEXTual Net 3D demonstrates strong performance, surpassing other automated methods but not yet matching expert radiologists.
• Larger datasets are likely necessary to bridge the performance gap between AI models and human experts in PET/CT interpretation.