Related Experiment Videos
Temporal validation of machine-learning models for shingles vaccine nonvaccination among U.S. adults aged 50 years
1Department of Mathematics and Statistics, Texas Tech University, Lubbock, TX 79409, United States.
Background:
Herpes zoster vaccination is recommended for U.S. adults aged 50 years and older, yet many remain unvaccinated. Prior National Health Interview Survey (NHIS) studies have described vaccination coverage and associated factors, but temporal validation of models predicting future-year unvaccinated status using interpretable machine-learning methods is limited.
Methods:
The 2019-2024 NHIS Sample Adult data were analyzed for adults aged 50 years and older with complete vaccination, predictor, and survey-design data (N = 96,663). Models were developed using 2019-2023 data and evaluated in the held-out 2024 survey year. Survey-weighted logistic regression and weighted XGBoost used the same 15 harmonized predictors. Performance was assessed using area under the receiver operating characteristic curve (AUC), Brier score, calibration, training-derived classification thresholds, and model-importance measures. Age-stratified and sensitivity analyses were also conducted.
Results:
The survey-weighted proportion unvaccinated declined from 73.7% in 2019 to 56.1% in 2024. In the held-out 2024 sample, logistic regression achieved an AUC of 0.764 (95% CI, 0.757-0.771) and XGBoost an AUC of 0.771 (95% CI, 0.764-0.778); the paired AUC difference was 0.0065 (p = 1.10 × 10-9). Brier scores were 0.210 and 0.207, respectively. Both models systematically overpredicted the probability of being unvaccinated in 2024, while calibration slopes remained near 1. Influenza vaccination, age, education, and race/ethnicity were the four highest-ranked predictors in both models. Survey-weighted nonvaccination was 71.4% among adults aged 50-59 years, 59.6% among those aged 60-64 years, and 44.6% among those aged 65 years and older. Restricting model development to 2022-2023 produced similar discrimination but reduced systematic overprediction.
Conclusions:
Shingles nonvaccination declined substantially but remained common among U.S. adults aged 50 years and older. XGBoost provided only a small improvement in discrimination over survey-weighted logistic regression. Temporal calibration drift in both models and its reduction with more recent training data emphasize the importance of temporal validation, model updating, and external validation before applied public-health use.
