Enhancing Document Classification Through Multimodal Image-Text Classification: Insights from Fine-Tuned CLIP and...

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Computer Vision

Background:

• Foundation models trained on general data struggle in clinical settings due to domain shift.
• Multimodal deep learning shows promise for medical diagnosis by integrating images and text.
• The optimal approach for domain adaptation—fine-tuning large models or training task-specific ones—remains unclear.

Purpose of the Study:

• Introduce PairDx, a balanced dataset for evaluating multimodal medical AI models.
• Compare fine-tuning large vision-language models versus training lighter, task-specific architectures for clinical domain adaptation.
• Assess the efficiency-performance trade-offs of different multimodal approaches in healthcare.

Main Methods:

• Curated PairDx dataset (22,665 image-caption pairs) across six medical document classes.
• Developed and evaluated PairDxCLIP (fine-tuned CLIP) and PairDxFusion (custom hybrid model).
• Established baselines including zero-shot CLIP and BiomedCLIP for comparison.

Main Results:

• Both PairDxCLIP (93% accuracy) and PairDxFusion (94% accuracy) significantly outperformed baselines.
• PairDxFusion achieved high accuracy with substantially faster training (17 min 55 s) compared to PairDxCLIP (65 min 52 s).
• PairDxFusion also demonstrated efficient testing time, outperforming BiomedCLIP.

Conclusions:

• Domain-specific datasets and lightweight multimodal fusion effectively bridge the domain gap in medical AI.
• Custom fusion models offer a practical balance of high performance and reduced computational cost for clinical applications.
• This approach enhances healthcare decision support systems through efficient and accurate AI models.