Distributed non-disclosive validation of predictive models by a modified ROC-GLM

Area of Science:

• Biostatistics
• Data Privacy
• Machine Learning Evaluation

Background:

• Distributed statistical analyses offer a privacy-preserving method for analyzing data across multiple databases by using summary statistics.
• Evaluating discrimination models requires assessing prognostic or predictive performance on independent data.
• Receiver Operating Characteristics (ROC) and Area Under the Curve (AUC) are key measures for binary classification model performance.

Purpose of the Study:

• To calculate ROC and AUC for binary classification in a distributed, privacy-preserving manner.
• To assess calibration-in-the-large indicators within a distributed framework.
• To investigate the impact of differential privacy on distributed model validation.

Main Methods:

• Utilized DataSHIELD for distributed analysis and a novel algorithm for privacy-preserving ROC analysis.
• Employed a generalized linear model (GLM) approximation (ROC-GLM) for ROC and AUC determination.
• Incorporated differential privacy (DP) by adding noise, with DP parameter impact studied via simulations.

Main Results:

• Distributed AUC measures showed differences from true AUC, heavily influenced by differential privacy parameters.
• Accuracy of the distributed AUC estimator can be negatively impacted by excessive noise from DP.
• Recommendations include simulating and checking distributed AUC estimator accuracy with appropriate DP parameter choices.

Conclusions:

• The algorithms' applicability hinges on the statistical model's sensitivity.
• Approximation errors were acceptable in most simulations, but higher sensitivity models require adjusted privacy parameters.
• Complex measures like AUC can be effectively used for validation in distributed settings while maintaining individual privacy.