Development and internal validation of machine learning models for personalized survival predictions in spinal cord
Mert Karabacak1, Alexander J Schupper1, Matthew T Carr1
1Department of Neurosurgery, Mount Sinai Health System, 1468 Madison Ave, New York, NY, USA.
Background Context:
Numerous factors have been associated with the survival outcomes in patients with spinal cord gliomas (SCG). Recognizing these specific determinants is crucial, yet it is also vital to establish a reliable and precise prognostic model for estimating individual survival outcomes.
Objective:
The objectives of this study are twofold: first, to create an array of interpretable machine learning (ML) models developed for predicting survival outcomes among SCG patients; and second, to integrate these models into an easily navigable online calculator to showcase their prospective clinical applicability.
Study Design:
This was a retrospective, population-based cohort study aiming to predict the outcomes of interest, which were binary categorical variables, in SCG patients with ML models.
Patient Sample:
The National Cancer Database (NCDB) was utilized to identify adults aged 18 years or older who were diagnosed with histologically confirmed SCGs between 2010 and 2019.
Outcome Measures:
The outcomes of interest were survival outcomes at three specific time points postdiagnosis: 1, 3, and 5 years. These outcomes were formed by combining the "Vital Status" and "Last Contact or Death (Months from Diagnosis)" variables. Model performance was evaluated visually and numerically. The visual evaluation utilized receiver operating characteristic (ROC) curves, precision-recall curves (PRCs), and calibration curves. The numerical evaluation involved metrics such as sensitivity, specificity, accuracy, area under the PRC (AUPRC), area under the ROC curve (AUROC), and Brier Score.
Methods:
We employed five ML algorithms-TabPFN, CatBoost, XGBoost, LightGBM, and Random Forest-along with the Optuna library for hyperparameter optimization. The models that yielded the highest AUROC values were chosen for integration into the online calculator. To enhance the explicability of our models, we utilized SHapley Additive exPlanations (SHAP) for assessing the relative significance of predictor variables and incorporated partial dependence plots (PDPs) to delineate the influence of singular variables on the predictions made by the top performing models.
Results:
For the 1-year survival analysis, 4,913 patients [5.6% with 1-year mortality]; for the 3-year survival analysis, 4,027 patients (11.5% with 3-year mortality]; and for the 5-year survival analysis, 2,854 patients (20.4% with 5-year mortality) were included. The top models achieved AUROCs of 0.938 for 1-year mortality (TabPFN), 0.907 for 3-year mortality (LightGBM), and 0.902 for 5-year mortality (Random Forest). Global SHAP analyses across survival outcomes at different time points identified histology, tumor grade, age, surgery, radiotherapy, and tumor size as the most significant predictor variables for the top-performing models.
Conclusions:
This study demonstrates ML techniques can develop highly accurate prognostic models for SCG patients with excellent discriminatory ability. The interactive online calculator provides a tool for assessment by physicians (https://huggingface.co/spaces/MSHS-Neurosurgery-Research/NCDB-SCG). Local interpretability informs prediction influences for a given individual. External validation across diverse datasets could further substantiate potential utility and generalizability. This robust, interpretable methodology aligns with the goals of precision medicine, establishing a foundation for continued research leveraging ML's predictive power to enhance patient counseling.
