[Cross-modal retrieval method for thyroid ultrasound image and text based on generative adversarial network]

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Computer Vision

Background:

• Thyroid examinations commonly utilize ultrasonic imaging, generating both visual data and textual reports.
• Effective retrieval of correlated thyroid ultrasound images and text reports is currently lacking, hindering clinical utility.
• Cross-modal retrieval methods offer potential for integrating image and text data in medical diagnostics.

Purpose of the Study:

• To develop an advanced cross-modal retrieval method for correlating thyroid ultrasound images with their associated text reports.
• To enhance the accuracy and efficiency of searching and accessing relevant thyroid imaging data and clinical notes.
• To address the existing gap in methods for linking visual and textual information in thyroid ultrasonography.

Main Methods:

• A deep learning approach utilizing an improved cross-modal generative adversarial network (GAN).
• Modification of weight sharing constraints to cosine similarity for better common representation learning between image and text data.
• Integration of a fully connected layer before the cross-modal discriminator for enhanced semantic regularization and data fusion.

Main Results:

• The proposed method achieved a mean average precision (mAP) of 0.508 for cross-modal retrieval of thyroid ultrasound images and text reports.
• This performance is significantly superior to traditional cross-modal retrieval techniques.
• Demonstrated effective learning of common representations between different data modalities.

Conclusions:

• The developed deep learning-based cross-modal retrieval method offers a significant advancement for thyroid ultrasound image and text report correlation.
• This novel approach provides a valuable new tool for medical professionals, improving data accessibility and diagnostic support.
• The findings highlight the potential of improved GAN architectures for complex medical data retrieval tasks.