Self-supervised Contrastive Video-Speech Representation Learning for Ultrasound.

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Machine Learning

Background:

• Manual annotations for deep learning models in medical imaging are costly and often inaccessible.
• Developing methods to learn from raw data without annotations is crucial for scaling AI in healthcare.

Purpose of the Study:

• To develop a self-supervised representation learning framework using multi-modal ultrasound video and speech data.
• To leverage the inherent correlation between visual and auditory information in medical procedures.

Main Methods:

• Proposed a framework for unsupervised learning of representations from raw ultrasound video and speech.
• Introduced cross-modal contrastive learning to model video-audio correspondence.
• Implemented an affinity-aware self-paced learning scheme to enhance correlation modeling.

Main Results:

• The model successfully learned meaningful representations by identifying correlations between ultrasound videos and sonographer speech.
• The learned representations demonstrated strong transferability to downstream tasks.
• Achieved significant performance improvements in standard plane detection and eye-gaze prediction tasks.

Conclusions:

• Self-supervised learning from multi-modal raw data is a viable alternative to annotation-dependent methods in medical imaging.
• The proposed framework effectively utilizes cross-modal correlations for robust representation learning.
• This approach has the potential to reduce the reliance on manual annotations in medical AI development.