Robust machine learning challenge: An AIFM multicentric competition to spread knowledge, identify common pitfalls and...

Area of Science:

• Medical Physics
• Radiomics
• Machine Learning

Background:

• Radiomics shows promise in distinguishing lung tumors from metastases.
• Machine learning models are increasingly used in medical imaging analysis.
• Assessing model robustness and implementation pitfalls is crucial for clinical translation.

Purpose of the Study:

• To foster knowledge and identify robust methods in machine learning for the medical physics community.
• To evaluate the potential of radiomics in differentiating primary lung tumors and metastases.
• To highlight potential pitfalls in machine learning model development and implementation.

Main Methods:

• A public dataset of 41 radiomic features from 535 patients was used.
• Participants developed base models (all features) and robust models (robust features only).
• Models were validated using cross-validation and unseen data, with Population Stability Index (PSI) assessing implementation issues.

Main Results:

• PSI detected implementation errors in 70% of models.
• The average Gmean for participant models (0.67) significantly outperformed the reference Gmean (0.50).
• Robust models showed slightly worse performance than base models but better stability on the training set.

Conclusions:

• Machine learning models developed through this challenge significantly improved diagnostic performance (Gmean).
• Excluding less-robust features did not enhance model robustness and should be avoided without confounding factors.
• Collaborative learning and knowledge exchange among medical physicists led to improved machine learning models.