Explainable machine learning for early diagnosis of esophageal cancer: A feature-enriched Light Gradient Boosting...

Area of Science:

• Oncology
• Data Science
• Machine Learning

Background:

• Esophageal cancer is a rapidly spreading malignancy with critical needs for early detection.
• Traditional statistical methods are common for risk assessment, but prediction models offer enhanced capabilities.
• Few studies have explored advanced prediction models for esophageal cancer risk.

Purpose of the Study:

• To develop and evaluate a machine learning model for accurate and interpretable esophageal cancer risk prediction.
• To identify key clinical and demographic features contributing to esophageal cancer risk.
• To assess the utility of advanced machine learning techniques for early cancer detection.

Main Methods:

• Utilized a Kaggle dataset of 3985 esophageal cancer patient records with 85 features.
• Implemented a data-engineering pipeline including imputation, encoding, standardization, and feature selection (Mutual Information).
• Trained and evaluated six machine learning classifiers (LightGBM, SVM, GNB, KNN, AdaBoost, MLP) using stratified 10-fold cross-validation and hyperparameter optimization (GridSearchCV).
• Assessed model interpretability using SHAP and LIME, and evaluated performance with reduced dimensionality.

Main Results:

• LightGBM demonstrated superior performance, achieving 99.74% accuracy post-tuning, with high precision, recall, F1-score, and AUC.
• Explainability analyses revealed tumor staging, smoking factors, and follow-up indicators as significant predictors.
• A reduced model using top 20 SHAP features maintained high predictive performance (99.76%), confirming robustness.

Conclusions:

• The LightGBM model offers exceptional predictive accuracy and interpretability for esophageal cancer.
• The model's ability to identify key risk factors supports its potential for early detection strategies.
• Machine learning approaches, like the proposed LightGBM model, show significant promise in improving esophageal cancer prevention and patient outcomes.