End-to-End Prediction of Knee Osteoarthritis Progression With Multimodal Transformers

Knee Osteoarthritis (KOA) is a prevalent chronic musculoskeletal condition with no currently available treatment. Predicting its progression is difficult due to its varied manifestation. Recent studies highlight the potential of using multimodal data and Deep Learning (DL) for prediction, though evidence is still emerging. In our study, we leveraged DL, specifically, a Transformer model, to fuse multimodal knee imaging data. We analyzed its performance across different progression horizons - from short-term to long-term - using a large dataset (n = 3967/2421) from the Osteoarthritis Initiative. We show that structural knee MRI allows identifying radiographic KOA progressors on par with multimodal fusion approaches, achieving an area under the ROC curve (ROC AUC) of 0.70-0.76 and Average Precision (AP) of 0.15-0.54 in horizons from 2 to 8 years. Multimodal approach using X-ray, structural and compositional MR images was more effective for predicting 1-year progression, achieving a ROC AUC of 0.76 (0.04) and AP of 0.13 (0.04). Our follow-up analysis suggests that prediction from the imaging data is particularly accurate for post-traumatic cases, and we further investigate which subject subgroups may benefit the most. The study offers new insights into multimodal imaging of KOA and brings a unified data-driven framework for studying its progression end-to-end, providing new tools to enhance clinical trial design. The source code of our framework and the pre-trained models are made publicly available.