Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health...

Area of Science:

• Health Informatics
• Artificial Intelligence
• Machine Learning Validation

Background:

• Cross-validation is crucial for developing and validating AI in healthcare.
• Existing tutorials lack practical, comparative examples on real-world health data.
• Electronic health records (EHRs) present unique challenges for model validation.

Purpose of the Study:

• To provide a practical tutorial comparing various cross-validation techniques.
• To demonstrate these methods using the accessible Medical Information Mart for Intensive Care-III (MIMIC-III) dataset.
• To offer best practices and reproducible code for AI modeling with EHR data.

Main Methods:

• Comparison of K-fold cross-validation and nested cross-validation.
• Application to classification (mortality prediction) and regression (length of stay prediction) tasks.
• Utilized the MIMIC-III dataset for real-world applicability.

Main Results:

• Nested cross-validation effectively reduces optimistic bias in model performance estimates.
• Nested cross-validation introduces increased computational complexity.
• Identified advantages and disadvantages of each method for specific predictive tasks.

Conclusions:

• Nested cross-validation is recommended for robust AI model validation in healthcare, despite computational demands.
• The tutorial provides reproducible resources to improve understanding and application of cross-validation techniques.
• Encourages the research community to adopt these validated methods for EHR-based AI development.