Tracheostomy weaning in patients with severe acquired brain injury: External validation of machine learning models

Our primary aim was to externally validate previously developed machine-learning (ML) models for predicting the probability of tracheostomy decannulation after 3 months from admission to rehabilitation inpatient in patients with severe Acquired Brain Injury (sABI) using a new external and temporally-independent multicentric prospective dataset. A secondary aim was to evaluate the timing of decannulation and to assess model calibration and clinical net benefit.

External validation data was collected within the PRABI study, comprising 435 sABI patients admitted between January 2020 and April 2024 across four centers. A previously trained ensemble model and a AdaBoost SVR model were used to predict decannulation probability and timing on such external dataset, respectively. Performance was assessed using metrics such as accuracy, Area Under the Receiver Operating Characteristic curve (AUROC), sensitivity, and median absolute error.

The external validation dataset included 402 patients. The ensemble model achieved an accuracy of 81.8 % and an AUROC of 0.85 for decannulation probability, with sensitivity and specificity of 76.4 % and 86.2 %, respectively. The AdaBoost SVR model predicted decannulation timing with a median absolute error of 26.2 days and an accuracy of 76.2 % when predictions were dichotomized at the 90-days threshold.

This study successfully validated the ML models on an independent dataset, demonstrating their robustness and generalizability. Accurate prediction of decannulation probability and timing is crucial for optimizing the management of sABI patients, reducing infection risks, enhancing recovery, and facilitating smoother transitions to home care. External validation is a critical step for ensuring the reliability of ML models in diverse clinical settings, paving the way for their integration into clinical practice.