Multi-Source Conformal Inference Under Distribution Shift.

Area of Science:

• Machine Learning
• Statistical Inference
• Data Science

Background:

• Increasing use of complex machine learning models across diverse data sources.
• Challenges in multi-source environments: distribution shifts, privacy concerns, lack of uncertainty quantification.
• Need for valid inferences and reliable predictions in heterogeneous data settings.

Purpose of the Study:

• To develop distribution-free prediction intervals for a target population using multiple, potentially biased data sources.
• To enable valid inferences in multi-source environments despite distribution shifts and privacy concerns.
• To quantify uncertainty in machine learning predictions for improved decision-making.

Main Methods:

• Derivation of efficient influence functions for quantiles in target and source populations.
• Incorporation of machine learning algorithms for nuisance function estimation, achieving parametric convergence rates.
• Development of a data-adaptive strategy to upweight informative and downweight non-informative data sources when conditional outcome invariance is violated.

Main Results:

• The proposed method achieves nominal coverage probabilities, demonstrating robustness and efficiency across various conformal scores and data-generating mechanisms.
• The data-adaptive strategy effectively handles violations of conditional outcome invariance, improving efficiency and reducing bias.
• Successful application to hospital length of stay prediction for pediatric cardiac surgery patients highlights practical utility.

Conclusions:

• The methodology provides a robust framework for generating reliable prediction intervals from multi-source, potentially biased data.
• It effectively addresses key challenges in machine learning for multi-source environments, enhancing generalizability and uncertainty quantification.
• The approach offers a valuable tool for data-driven decision-making in complex, real-world scenarios.