A Balanced Multimodal Multi-Task Deep Learning Framework for Robust Patient-Specific Quality Assurance.

Area of Science:

• Artificial Intelligence
• Medical Physics
• Radiotherapy

Background:

• Multimodal deep learning is vital for automated patient-specific quality assurance (PSQA) in radiotherapy.
• Integrating image and tabular data improves prediction of quality indicators like Gamma Passing Rate (GPR) and dose difference (DD).
• Modality imbalance, where tabular data dominates, reduces model robustness, especially under task heterogeneity.

Purpose of the Study:

• To introduce BMMQA (Balanced Multi-modal Quality Assurance), a novel framework to achieve modality balance in multimodal deep learning for PSQA.
• To enhance predictive accuracy and robustness in radiotherapy quality assurance by addressing modality imbalance.

Main Methods:

• Proposed BMMQA framework with modality-specific loss factors for controlled convergence.
• Implemented task-specific fusion strategies (softmax-weighted attention, spatial cascading).
• Utilized Shapley values for modality contribution quantification and a modality-contribution-based task weighting scheme.

Main Results:

• BMMQA outperformed existing fusion baselines under standard and stricter GPR criteria.
• Achieved a 15.7% reduction in mean absolute error (MAE) for dose difference prediction.
• Enhanced robustness in critical failure cases and achieved a peak SSIM of 0.964 for dose distribution prediction.

Conclusions:

• Explicit modality balancing significantly improves predictive accuracy and clinical trustworthiness in PSQA.
• Mitigating overreliance on single modalities is crucial for robust AI in radiotherapy.
• BMMQA establishes a pioneering framework for multi-task multimodal learning in medical AI.