A divide-and-conquer method for sparse risk prediction and evaluation.

Area of Science:

• Statistical modeling
• Machine learning
• Bioinformatics

Background:

• Divide-and-conquer (DAC) algorithms are used for large datasets but can be computationally intensive.
• Existing DAC methods for sparse regression lack inference capabilities for risk prediction accuracy.
• Massive datasets require efficient algorithms for sparse predictive modeling.

Purpose of the Study:

• To propose a computationally efficient algorithm, SOLID, for fitting sparse logistic regression to massive datasets.
• To develop a modified cross-validation (MCV) procedure for accurate risk prediction model assessment.
• To enable inference on the accuracy of predictive models using a novel approach.

Main Methods:

• Developed the screening and one-step linearization infused DAC (SOLID) algorithm.
• Integrated screening and linearization within the DAC framework for penalized estimation.
• Introduced modified cross-validation (MCV) utilizing SOLID's intermediate results for computational efficiency.

Main Results:

• SOLID and MCV significantly outperform existing DAC methods in computational speed.
• The proposed methods achieve statistical efficiency comparable to full sample-based estimators.
• MCV is the first DAC procedure to provide valid inference on predictive model accuracy.

Conclusions:

• SOLID and MCV offer a computationally efficient and statistically sound approach for analyzing massive datasets in sparse logistic regression.
• The developed inference procedure provides valid interval estimators for risk prediction accuracy.
• The SOLID procedure was successfully applied to a clinical notes-based disease diagnosis classification model.