How to select predictive models for decision-making or causal inference.

Area of Science:

• Causal Inference
• Machine Learning
• Health Services Research

Background:

• Standard model selection procedures may not identify the most reliable predictive models for explaining intervention effects.
• Accurate model selection is crucial for supporting evidence-based decision-making in healthcare.

Purpose of the Study:

• To identify the most trustworthy model selection procedure for predictive models used to explain intervention effects.
• To compare various model selection strategies under practical, finite-sample conditions without assuming well-specified models.

Main Methods:

• Evaluated standard cross-validation and internal validation alongside advanced causal risk metrics.
• Investigated causal risk estimation using nuisance reweighting on observed data.
• Conducted an extensive empirical study with simulations and real-world healthcare datasets.

Main Results:

• Mean squared error, a common predictive metric, performed poorly for causal effect estimation.
• Propensity score reweighting offered limited improvement.
• The R-risk metric, incorporating outcome and propensity score models, demonstrated superior performance.
• Flexible estimators like super learners optimized nuisance estimation.

Conclusions:

• Predictive models for intervention effects necessitate evaluation beyond standard predictive settings.
• The R-risk metric from causal inference is recommended for selecting trustworthy models in these contexts.