Clinically-Inspired Multi-Agent Transformers for Disease Trajectory Forecasting From Multimodal Data

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Computational Biology

Background:

• Deep neural networks commonly automate medical diagnosis from images.
• Predicting disease progression is clinically crucial but challenging with current methods.
• Existing prognosis tools often require domain expertise and are complex to implement.

Purpose of the Study:

• To develop a clinically relevant method for disease trajectory forecasting.
• To address prognosis prediction as a one-to-many prediction problem.
• To improve the accuracy and applicability of disease progression prediction.

Main Methods:

• A novel framework using two transformer-based components inspired by clinical decision-making.
• One transformer analyzes imaging data; the second integrates imaging features with auxiliary clinical data.
• Temporal disease dynamics are modeled within transformer states, enabling multi-task classification with a new loss function.

Main Results:

• Demonstrated effectiveness in predicting knee osteoarthritis structural changes.
• Successfully forecasted Alzheimer's disease clinical status using multi-modal data.
• Outperformed state-of-the-art baselines in both predictive performance and calibration.

Conclusions:

• The proposed two-transformer method offers a powerful and accessible approach to disease prognosis.
• The model accurately predicts disease progression from raw multi-modal data.
• This advancement has significant implications for clinical decision-making and patient care.