Radiological Image and Text-Based Medical Concept Detection in Social Networks Using Hybrid Deep Learning

Nowadays, the presence of health-related content on social networks is rapidly increasing. With the effect of these networks, a large number of medical images, diagnosed and interpreted by various experts, are shared online. Therefore, concept detection and image classification from medical images remains a challenging task. In recent years, deep learning-based models have become increasingly popular for addressing these challenges. The primary objective of this study is to perform multi-label classification of radiological images shared on a social network by automatically assigning relevant medical concepts. These concepts are derived from the Unified Medical Language System (UMLS). In this study, Convolutional Neural Network (CNN) combined with feed forward neural networks and various image encoders, including VGG-19, DenseNet-121, ResNet-101, Xception, Efficient-B7, to predict the appropriate concepts. The proposed hybrid deep learning models were trained and evaluated using the ImageCLEF 2019 dataset. Further evaluation was performed using a custom dataset (Rdpd_Test_Ds) composed of radiological images and their associated comments collected from a social network. The performance of the models was assessed using precision, recall, and F1-score metrics. The evaluation results are promising, demonstrating high performance. To the best of our knowledge, this research is the first to apply deep learning-based models to radiological data collected from a social network, representing a novel and impactful contribution to the field.