Evaluating Machine Learning Models for Molecular Property Prediction: Performance and Robustness on...

Area of Science:

• * Cheminformatics
• * Computational Chemistry
• * Machine Learning

Background:

• * Machine learning models are widely used for predicting molecular properties.
• * Performance evaluation typically uses in-distribution (ID) data, but real-world use requires out-of-distribution (OOD) data.
• * Assessing model performance on OOD data is crucial for reliable predictions in novel chemical spaces.

Purpose of the Study:

• * To investigate and evaluate machine learning model performance on OOD molecular data.
• * To define OOD data generation strategies in molecular property prediction.
• * To analyze the relationship between in-distribution (ID) and out-of-distribution (OOD) performance.

Main Methods:

• * Evaluated 14 machine learning models, including random forests and graph neural networks (GNNs).
• * Utilized eight datasets and ten splitting strategies for OOD data generation.
• * Employed Bemis-Murcko scaffolds and UMAP-based clustering (ECFP4 fingerprints) for OOD splitting.

Main Results:

• * Bemis-Murcko scaffold splitting showed models performed well, similar to random splitting.
• * UMAP-based chemical similarity clustering presented the most challenging OOD scenario.
• * The correlation between ID and OOD performance varied significantly with splitting strategy (Pearson's r ~0.9 for scaffolds, ~0.4 for clusters).

Conclusions:

• * OOD data generation strategy critically influences model performance and ID-OOD correlation.
• * Scaffold-based splitting is less challenging than similarity-based clustering for OOD evaluation.
• * Model selection requires careful consideration of OOD performance aligned with specific application domains.