Machine learning methods for estimating personalized treatment effects-insights on validity from two large trials

Area of Science:

• Biostatistics
• Medical Informatics
• Machine Learning

Background:

• Machine learning (ML) offers potential for personalized medicine by estimating individualized treatment effects.
• Formal validation of ML methods in empirical settings is limited, creating uncertainty about their reliability.
• Current ML applications in precision medicine require robust validation to ensure generalizability.

Purpose of the Study:

• To evaluate the internal and external validity of 17 causal heterogeneity ML methods.
• To assess the generalizability of ML-derived heterogeneous treatment effects.
• To identify limitations in current ML validation for precision medicine.

Main Methods:

• Utilized data from two large randomized controlled trials: International Stroke Trial (n=19,435) and Chinese Acute Stroke Trial (n=21,106).
• Evaluated 17 causal heterogeneity ML methods, including metalearners, tree-based, and deep learning approaches.
• Assessed ML model performance using three visual and three quantitative metrics for internal and external validity.

Main Results:

• None of the evaluated ML methods consistently demonstrated reliable internal or external validity.
• Heterogeneous treatment effects estimated from training data did not generalize to test data.
• Poor generalizability was observed even without significant distribution shifts between training and test sets.

Conclusions:

• Current ML methods show limitations in reliably estimating and generalizing personalized treatment effects.
• The findings raise concerns regarding the current applicability of ML models in precision medicine.
• There is a critical need for enhanced validation strategies to ensure the generalizability of ML models in healthcare.