OWL: an optimized and independently validated machine learning prediction model for lung cancer screening based on
Zoucheng Pan1, Ruyang Zhang1, Sipeng Shen1
1Department of Biostatistics, Center for Global Health, School of Public Health, Nanjing Medical University, Nanjing, 211166, China.
Background:
A reliable risk prediction model is critically important for identifying individuals with high risk of developing lung cancer as candidates for low-dose chest computed tomography (LDCT) screening. Leveraging a cutting-edge machine learning technique that accommodates a wide list of questionnaire-based predictors, we sought to optimize and validate a lung cancer prediction model.
Methods:
We developed an Optimized early Warning model for Lung cancer risk (OWL) using the XGBoost algorithm with 323,344 participants from the England area in UK Biobank (training set), and independently validated it with 93,227 participants from UKB Scotland and Wales area (validation set 1), as well as 70,605 and 66,231 participants in the Prostate, Lung, Colorectal, and Ovarian cancer screening trial (PLCO) control and intervention subpopulations, respectively (validation sets 2 & 3) and 23,138 and 18,669 participants in the United States National Lung Screening Trial (NLST) control and intervention subpopulations, respectively (validation sets 4 & 5). By comparing with three competitive prediction models, i.e., PLCO modified 2012 (PLCOm2012), PLCO modified 2014 (PLCOall2014), and the Liverpool Lung cancer Project risk model version 3 (LLPv3), we assessed the discrimination of OWL by the area under receiver operating characteristic curve (AUC) at the designed time point. We further evaluated the calibration using relative improvement in the ratio of expected to observed lung cancer cases (RIEO), and illustrated the clinical utility by the decision curve analysis.
Findings:
For general population, with validation set 1, OWL (AUC = 0.855, 95% CI: 0.829-0.880) presented a better discriminative capability than PLCOall2014 (AUC = 0.821, 95% CI: 0.794-0.848) (p < 0.001); with validation sets 2 & 3, AUC of OWL was comparable to PLCOall2014 (AUCPLCOall2014-AUCOWL < 1%). For ever-smokers, OWL outperformed PLCOm2012 and PLCOall2014 among ever-smokers in validation set 1 (AUCOWL = 0.842, 95% CI: 0.814-0.871; AUCPLCOm2012 = 0.792, 95% CI: 0.760-0.823; AUCPLCOall2014 = 0.791, 95% CI: 0.760-0.822, all p < 0.001). OWL remained comparable to PLCOm2012 and PLCOall2014 in discrimination (AUC difference from -0.014 to 0.008) among the ever-smokers in validation sets 2 to 5. In all the validation sets, OWL outperformed LLPv3 among the general population and the ever-smokers. Of note, OWL showed significantly better calibration than PLCOm2012, PLCOall2014 (RIEO from 43.1% to 92.3%, all p < 0.001), and LLPv3 (RIEO from 41.4% to 98.7%, all p < 0.001) in most cases. For clinical utility, OWL exhibited significant improvement in average net benefits (NB) over PLCOall2014 in validation set 1 (NB improvement: 32, p < 0.001); among ever smokers of validation set 1, OWL (average NB = 289) retained significant improvement over PLCOm2012 (average NB = 213) (p < 0.001). OWL had equivalent NBs with PLCOm2012 and PLCOall2014 in PLCO and NLST populations, while outperforming LLPv3 in the three populations.
Interpretation:
OWL, with a high degree of predictive accuracy and robustness, is a general framework with scientific justifications and clinical utility that can aid in screening individuals with high risks of lung cancer.
Funding:
National Natural Science Foundation of China, the US NIH.
