Integration of nested cross-validation, automated hyperparameter optimization, high-performance computing to reduce...

Area of Science:

• Medical imaging
• Deep learning
• Computational science

Background:

• Deep learning model performance benchmarking in medical imaging suffers from variability and bias, hindering real-world trust.
• Current methods using fixed test sets inadequately capture performance metric variance.
• This necessitates robust evaluation frameworks for reliable medical AI deployment.

Purpose of the Study:

• Introduce NACHOS (Nested and Automated Cross-validation and Hyperparameter Optimization using Supercomputing) to reduce and quantify performance metric variance.
• Develop a framework for trustworthy deep learning model evaluation in medical imaging.
• Enhance the reliability of AI tools in clinical settings.

Main Methods:

• NACHOS integrates Nested Cross-Validation (NCV) and Automated Hyperparameter Optimization (AHPO) within a parallelized high-performance computing (HPC) framework.
• Demonstrated on chest X-ray and Optical Coherence Tomography (OCT) datasets.
• Introduced DACHOS (Deployment with Automated Cross-validation and Hyperparameter Optimization using Supercomputing) for final model building on full datasets.

Main Results:

• NCV is crucial for quantifying and reducing estimation variance.
• AHPO ensures consistent hyperparameter optimization across test folds.
• HPC guarantees the computational feasibility of the proposed framework.

Conclusions:

• NACHOS and DACHOS offer a scalable, reproducible, and trustworthy framework for deep learning model evaluation and deployment in medical imaging.
• The open-source codebase is publicly available to promote adoption and further research.
• This framework addresses critical needs for reliable AI in healthcare.