Related Experiment Video
Updated: Jun 29, 2026
DNA Virus Detection System Based on RPA-CRISPR/Cas12a-SPM and Deep Learning
Published on: May 10, 2024
Intelligent Attention-Driven Deep Learning for Hip Disease Diagnosis: Fusing Multimodal Imaging and Clinical Text for
Jinming Zhang1,2, He Gong1,2, Pengling Ren1,2
1Medical Engineering & Engineering Medicine Innovation Center, Hangzhou International Innovation Institute, Beihang University, Hangzhou 311115, China.
Insights
Multimodal deep learning integrating clinical data, X-rays, and CT scans significantly enhances the early detection and classification of hip joint disorders like osteoarthritis and osteonecrosis of the femoral head (ONFH). This approach offers improved diagnostic accuracy.
Area of Science:
- Orthopedic imaging analysis
- Artificial intelligence in medicine
- Deep learning for medical diagnosis
Background:
- Hip joint disorders present overlapping radiological features, challenging single-modality imaging and early diagnosis.
- Isolated imaging or clinical data may not fully capture disease-specific pathological characteristics.
Purpose of the Study:
- To develop and evaluate a multimodal deep learning framework for improved diagnosis of hip joint disorders.
- To assess the diagnostic performance of integrating radiographs, CT volumes, and clinical texts.
Main Methods:
- A retrospective study of 605 hip joints (normal, osteoarthritis, osteonecrosis of the femoral head (ONFH), femoroacetabular impingement (FAI)) from Center A, with external validation on 24 hips from Center B.
- Development of a deep learning framework using ResNet50, 3D-ResNet50, and BERT for feature extraction.
- Attention-based fusion of multimodal features for four-class classification.
Main Results:
- The combined Clinical+X-ray+CT model achieved an AUC of 0.949, outperforming single-modality models.
- Consistent improvements in accuracy, sensitivity, and specificity were observed.
- Grad-CAM visualizations confirmed the model's focus on clinically relevant anatomical regions.
Conclusions:
- Attention-based multimodal feature fusion significantly enhances diagnostic performance for hip joint diseases.
- The developed framework is interpretable and clinically applicable for early detection and precise classification in orthopedic imaging.
Abstract:
Background and Objectives: Hip joint disorders exhibit diverse and overlapping radiological features, complicating early diagnosis and limiting the diagnostic value of single-modality imaging. Isolated imaging or clinical data may therefore inadequately represent disease-specific pathological characteristics. Materials and Methods: This retrospective study included 605 hip joints from Center A (2018-2024), comprising normal hips, osteoarthritis, osteonecrosis of the femoral head (ONFH), and femoroacetabular impingement (FAI). An independent cohort of 24 hips from Center B (2024-2025) was used for external validation. A multimodal deep learning framework was developed to jointly analyze radiographs, CT volumes, and clinical texts. Features were extracted using ResNet50, 3D-ResNet50, and a pretrained BERT model, followed by attention-based fusion for four-class classification. Results: The combined Clinical+X-ray+CT model achieved an AUC of 0.949 on the internal test set, outperforming all single-modality models. Improvements were consistently observed in accuracy, sensitivity, specificity, and decision curve analysis. Grad-CAM visualizations confirmed that the model attended to clinically relevant anatomical regions. Conclusions: Attention-based multimodal feature fusion substantially improves diagnostic performance for hip joint diseases, providing an interpretable and clinically applicable framework for early detection and precise classification in orthopedic imaging.
