Beyond Random Splitting: Evaluating the Impact of Data Partitioning Strategies on Ventilator-Associated Pneumonia...

Ventilator-Associated Pneumonia (VAP) significantly impacts critical care outcomes, yet prediction models often over-look healthcare data's meronomic structure. Using MIMIC-III data, we developed a multi-source extraction approach integrating structured data with clinical notes, identifying 679 VAP and 3,207 non-VAP cases. We compared four data splitting strategies: Ventilator Session-Based Split, Ventilator Session-Based Split on Single ICU Stays, Hospital Admission-Based Split, and Hospital Admission-Based Split on Single ICU Stays. Evaluating four machine learning models revealed that conventional random splitting yielded moderately high performance (AUROC: 76-81%) while restricting to single ICU stays surprisingly improved performance (AUROC: 86-87%). Admission-based approaches showed realistic results (AUROC: 72-76%). Feature analysis identified mechanical ventilation hours, systolic blood pressure, and urine counts as consistently important predictors. These findings demonstrate that robust VAP prediction requires evaluation frameworks respecting healthcare data's meronomic nature.