Real-World Molecular Out-Of-Distribution: Specification and Investigation

Area of Science:

• Drug discovery
• Computational chemistry
• Machine learning

Background:

• Molecular out-of-distribution (MOOD) generalization is critical for reliable AI in drug discovery.
• Real-world deployment exposes models to data shifts, impacting performance and uncertainty calibration.
• Existing methods often fail to account for these deployment-related data distribution shifts.

Purpose of the Study:

• To establish a rigorous framework for investigating MOOD generalization in molecular scoring.
• To quantify the impact of covariate shifts on model performance and uncertainty.
• To propose and validate a splitting protocol for realistic evaluation.

Main Methods:

• Defined MOOD generalization through problem specification based on sample distance distributions.
• Developed a novel splitting protocol to bridge the gap between training/deployment and testing.
• Conducted a comprehensive investigation into model design, selection, and dataset characteristics.

Main Results:

• Covariate shifts can degrade performance by up to 60% and uncertainty calibration by up to 40%.
• Appropriate molecular representations significantly enhance MOOD performance.
• Algorithms with built-in uncertainty estimation are vital for robust predictions.

Conclusions:

• The proposed framework and protocol enable meaningful benchmarking of MOOD generalization.
• Model design and uncertainty estimation are critical for reliable molecular scoring in drug discovery.
• This work opens new avenues for developing robust AI models for real-world applications.