[Small bowel video keyframe retrieval based on multi-modal contrastive learning]

Area of Science:

• Medical Imaging
• Computer Vision
• Machine Learning

Context:

• Accurate localization of pathological regions in small intestine videos is crucial for diagnosis.
• Directly training on raw video data is computationally intensive and slow.
• Learning visual representations from image-text datasets can lead to computational inconsistencies.

Purpose:

• To propose a novel framework, KRCL (small bowel video keyframe retrieval based on multi-modal contrastive learning), for efficient and accurate keyframe retrieval from small intestine videos.
• To leverage textual information from video category labels to learn text-relevant video features.
• To model temporal information within a pre-trained image-text model and transfer knowledge to the medical video domain.

Summary:

• The KRCL framework utilizes multi-modal contrastive learning to connect video data with textual labels, enabling effective keyframe retrieval.
• It integrates temporal modeling and knowledge transfer from pre-trained image-text models to enhance video feature learning.
• The approach facilitates interaction between medical videos, images, and text data for improved analysis.

Impact:

• KRCL achieves state-of-the-art performance on the Hyper-Kvasir and MSR-VTT datasets, demonstrating its effectiveness and robustness.
• The method offers a computationally efficient alternative to direct video training.
• This advancement has the potential to significantly improve the accuracy and efficiency of gastrointestinal disease detection and diagnosis.