Efficient learning from big data for cancer risk modeling: A case study with melanoma.

Area of Science:

• Computational biology
• Machine learning in healthcare
• Real-world data analysis

Background:

• Building accurate cancer risk models from real-world data presents significant challenges in data preprocessing, representation, and computational efficiency.
• Electronic health records (EHRs) offer a valuable resource for developing such models, but require careful handling of de-identified data.

Purpose of the Study:

• To present a cloud-based computational approach for learning melanoma risk prediction models from de-identified EHR data.
• To demonstrate the effectiveness of this approach in overcoming data preprocessing and performance challenges.
• To evaluate the impact of data sampling on computational performance and model accuracy.

Main Methods:

• A hybrid cloud computing strategy was employed, utilizing Apache Spark for distributed data preprocessing and scikit-learn for non-distributed machine learning model training.
• The study explored the effects of training dataset sampling on computational performance.
• Risk factors were identified and evaluated using regression weights and tree SHAP values.

Main Results:

• A gradient-boosted classifier achieved high predictive performance (AUC=0.799, Sensitivity=0.753, Specificity=0.688) on a dataset of over 4 million patients.
• A significantly smaller sampled dataset (two orders of magnitude smaller) yielded statistically similar or superior performance compared to the original data.
• The sampled dataset approach reduced training time and cost by over 99%.

Conclusions:

• A hybrid computing infrastructure combined with data sampling effectively predicts melanoma risk in a diverse US dermatology population.
• Data sampling substantially accelerated machine learning model development time without compromising predictive accuracy.
• The optimal model incorporated over 300 identified risk factors.