Related Experiment Video
Updated: Nov 3, 2025
Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
Pay attention to doctor-patient dialogues: Multi-modal knowledge graph attention image-text embedding for COVID-19
Wenbo Zheng1,2, Lan Yan2,3, Chao Gou4
1School of Software Engineering, Xi'an Jiaotong University, Xi'an 710049, China.
Insights
This study introduces a novel multi-modal approach for diagnosing coronavirus disease 2019 (COVID-19) by integrating doctor-patient dialogues and medical images. The method enhances diagnostic accuracy and robustness, addressing data limitations in current deep learning models.
Area of Science:
- Medical Informatics
- Artificial Intelligence in Healthcare
- Computational Biology
Background:
- The rapid rise in coronavirus disease 2019 (COVID-19) cases strains global healthcare systems.
- Accurate and early clinical assessment of COVID-19 severity is crucial for effective patient management.
- Existing deep learning models face challenges with multimodal data adequacy and integrating diverse information sources.
Purpose of the Study:
- To develop and evaluate a novel multi-modal knowledge graph attention embedding method for improved COVID-19 diagnosis.
- To address the limitations of current deep learning approaches in handling multimodal data for disease assessment.
- To leverage both textual (doctor-patient dialogues) and visual (medical images) data for more accurate COVID-19 inferences.
Main Methods:
- Proposed a multi-modal knowledge graph attention embedding technique for COVID-19 diagnosis.
- Incorporated medical knowledge through an attention mechanism to enhance classifier performance.
- Constructed a new COVID-19 multi-modal dataset comprising doctor-patient dialogues and various medical imaging modalities (X-ray, CT, ultrasound).
Main Results:
- The proposed approach significantly outperformed existing state-of-the-art techniques in classification performance.
- Demonstrated robustness across different modalities within the multi-modal data.
- The newly constructed dataset includes 1393 COVID-19 patient dialogues and 3706 images, alongside 607 non-COVID-19 patient dialogues and 10754 images, with fine-grained labels.
Conclusions:
- The multi-modal knowledge graph attention embedding method offers a promising solution for accurate COVID-19 diagnosis.
- Integrating doctor-patient dialogues with medical images enhances diagnostic capabilities beyond image-only analysis.
- This work encourages a shift towards holistic data utilization in AI-driven medical diagnostics.
Abstract:
The sudden increase in coronavirus disease 2019 (COVID-19) cases puts high pressure on healthcare services worldwide. At this stage, fast, accurate, and early clinical assessment of the disease severity is vital. In general, there are two issues to overcome: (1) Current deep learning-based works suffer from multimodal data adequacy issues; (2) In this scenario, multimodal (e.g., text, image) information should be taken into account together to make accurate inferences. To address these challenges, we propose a multi-modal knowledge graph attention embedding for COVID-19 diagnosis. Our method not only learns the relational embedding from nodes in a constituted knowledge graph but also has access to medical knowledge, aiming at improving the performance of the classifier through the mechanism of medical knowledge attention. The experimental results show that our approach significantly improves classification performance compared to other state-of-the-art techniques and possesses robustness for each modality from multi-modal data. Moreover, we construct a new COVID-19 multi-modal dataset based on text mining, consisting of 1393 doctor-patient dialogues and their 3706 images (347 X-ray 2598 CT 761 ultrasound) about COVID-19 patients and 607 non-COVID-19 patient dialogues and their 10754 images (9658 X-ray 494 CT 761 ultrasound), and the fine-grained labels of all. We hope this work can provide insights to the researchers working in this area to shift the attention from only medical images to the doctor-patient dialogue and its corresponding medical images.
Related Concept Videos
01:17Classification of Illness
An illness is a response to a disease in which the person's level of functioning is changed compared with a previous level. The general classification of illness includes acute and chronic.
Acute illness is severe...
01:13Patient-centered Care
00:59Hospitals-II
Nurses that work in...
01:29Issues And Trends In Healthcare Delivery System
Cost Containment
Payment for healthcare services has historically promoted adoption of costly and often unnecessary or inefficient...
01:26Healthcare Associated Infections I: Iatrogenic, Exogenic and Endogenic
HAIs significantly increase the cost of health care. Extended stays in healthcare institutions, increased disability, increased costs of medications, including specialized antibiotics, and prolonged recovery times add to the patient's expenses and the healthcare institution and funding bodies.