Hybrid deep learning system combining radiological and clinical data for improved tuberculosis diagnosis

Area of Science:

• Artificial Intelligence in Medical Imaging
• Deep Learning for Disease Diagnosis
• Multimodal Data Fusion for Healthcare

Background:

• Tuberculosis (TB) diagnosis is complex, requiring integration of imaging and clinical data.
• Existing methods using convolutional neural networks (CNNs) struggle to model interactions between imaging and clinical data effectively.
• Previous attempts to combine clinical scores with imaging improved diagnostic accuracy but had limitations.

Purpose of the Study:

• To develop a hybrid deep learning model that jointly learns from chest imaging and structured clinical data for enhanced TB diagnosis.
• To improve the accuracy and interpretability of TB diagnosis by effectively modeling cross-modal interactions.
• To provide a viable diagnostic pathway for TB, particularly in resource-limited settings.

Main Methods:

• A hybrid deep learning model employing a cross-modal transformer architecture with contrastive pre-training.
• Utilizing a vision transformer for chest images and a tabular transformer for clinical variables.
• Implementing a cross-modal attention module for token interaction and contrastive learning for representation alignment.

Main Results:

• The proposed model achieved an Area Under the Curve (AUC) of 0.96 and an accuracy of 0.90 on two independent cohorts.
• Demonstrated a significant improvement of approximately 10% over CNN-based baselines.
• Attention and SHAP analyses revealed that clinical factors like HIV status and low BMI influence model predictions.

Conclusions:

• The hybrid deep learning model effectively integrates radiological and clinical data for superior TB diagnosis.
• This approach offers enhanced accuracy and interpretability compared to traditional methods.
• The model presents a promising and viable solution for TB diagnosis in resource-constrained environments.