OpenVocabCT: Toward Universal Text-Driven CT Image Segmentation.

Area of Science:

• Medical Imaging and Artificial Intelligence
• Computer Vision in Healthcare
• Machine Learning for Medical Diagnosis

Background:

• Deep learning models (CNNs, ViTs) improve CT analysis but struggle with diverse clinical data.
• Foundation models are adaptable but require extensive voxel-level annotations, which are scarce for medical images.
• Prompting-based models offer solutions, but visual prompts (SAM) need manual input, and text-prompt models (CLIP-Driven) are limited by training data.

Purpose of the Study:

• To introduce OpenVocabCT, a novel vision-language model for universal text-driven segmentation in 3D CT images.
• To address the limitations of current models in handling diverse clinical data and complex segmentation tasks.
• To enable adaptable and versatile medical image analysis through text prompts.

Main Methods:

• Developed OpenVocabCT, a vision-language model pretrained on large-scale 3D CT images.
• Utilized the CT-RATE dataset to decompose diagnostic reports into fine-grained, organ-level descriptions.
• Employed large language models for multi-granular contrastive learning to enhance model understanding.

Main Results:

• OpenVocabCT demonstrated superior performance on downstream segmentation tasks compared to existing methods.
• Evaluated on 14 public and 1 institutional dataset for both organ and tumor segmentation.
• Achieved state-of-the-art results, showcasing the model's effectiveness across various segmentation challenges.

Conclusions:

• OpenVocabCT offers a versatile and clinically relevant approach to medical image segmentation using text prompts.
• The model's pretraining on large-scale CT data and multi-granular contrastive learning contribute to its superior performance.
• Public release of code, datasets, and models aims to foster further research and development in AI-driven medical imaging.