Voice-Assisted Image Labeling for Endoscopic Ultrasound Classification Using Neural Networks

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Machine Learning

Background:

• Ultrasound imaging is vital for real-time patient anatomy visualization but suffers from high operator dependency and a steep learning curve.
• Deep learning (DL) offers potential solutions for ultrasound training and interpretation, yet requires large, accurately labeled datasets.
• Manual labeling of ultrasound data is challenging due to the lack of 3D spatial context and the retrospective assignment of labels.

Purpose of the Study:

• To develop and evaluate a multi-modal deep learning model for automatic endoscopic ultrasound (EUS) image classification.
• To investigate the efficacy of using raw verbal comments from clinicians to label EUS images.
• To reduce the burden of manual data annotation for DL applications in medical imaging.

Main Methods:

• A multi-modal convolutional neural network (CNN) with two branches (voice and image data) was designed.
• The CNN was trained to predict image labels based on spoken anatomical landmark names from recorded expert verbal comments.
• The model was evaluated on a dataset of 5 different EUS image labels.

Main Results:

• The proposed multi-modal CNN achieved a prediction accuracy of 76% at the image level.
• The model successfully utilized verbal commentaries to classify EUS images.
• The results demonstrate the feasibility of automated labeling using audio data.

Conclusions:

• Incorporating spoken commentaries significantly enhances ultrasound image classification performance.
• This method alleviates the need for extensive manual labeling of large EUS datasets for DL.
• The approach shows promise for improving ultrasound training and aiding interpretation in complex cases.