A visually grounded language model for fetal ultrasound understanding

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Obstetrics

Background:

• Freehand fetal ultrasound requires significant expertise.
• AI-powered tools can assist in complex medical procedures.

Purpose of the Study:

• To develop an AI assistant, Sonomate, for real-time support during fetal ultrasound examinations.
• To improve the accuracy and efficiency of fetal ultrasound procedures through AI.

Main Methods:

• Developed Sonomate using coarse-grained video-text and fine-grained image-sentence alignment.
• Incorporated anatomy-aware alignment and context label correction for heterogeneous data.
• Built a visually grounded language model for understanding ultrasound videos.

Main Results:

• Sonomate effectively detects anatomy in fetal ultrasound images without retraining.
• Demonstrated promising performance in visual question answering for fetal ultrasound images and videos.
• Ensured safety through built-in guardrails for deployment.

Conclusions:

• Sonomate offers AI-assistive technology for sonography training and enhanced diagnostics.
• This AI assistant can improve the quality and accessibility of fetal ultrasound examinations.
• The visually grounded language model shows potential for real-time medical applications.