A transformer-based representation-learning model with unified processing of multimodal input for clinical...

Area of Science:

• Artificial Intelligence in Medicine
• Deep Learning for Healthcare
• Clinical Decision Support Systems

Background:

• Clinicians integrate multimodal data (chief complaint, images, lab results) for diagnosis.
• Current deep learning diagnostic aids often fail to leverage this multimodal information effectively.

Purpose of the Study:

• To develop a unified, transformer-based representation-learning model for clinical diagnostic assistance.
• To enable a single model to process and integrate diverse data types for a holistic patient representation.

Main Methods:

• Utilized embedding layers to convert images, unstructured text, and structured data into unified visual and text tokens.
• Employed bidirectional transformer blocks with intramodal and intermodal attention for holistic representation learning.
• Processed radiographs, chief complaint, clinical history, laboratory results, and demographic information.

Main Results:

• The unified multimodal model significantly outperformed image-only and non-unified multimodal models.
• Achieved a 12% higher accuracy in pulmonary disease identification compared to image-only models.
• Demonstrated a 29% improvement in predicting adverse COVID-19 outcomes versus image-only models.

Conclusions:

• Unified multimodal transformer models offer a powerful approach to clinical diagnostic aid.
• These models can streamline patient triaging and enhance clinical decision-making processes.
• Future applications may include broader integration of diverse clinical data for improved patient care.