The Fragility of Bioactivity Prediction: Rigorous Dataset Splits Expose the Illusion of ML Accuracy

Area of Science:

• Computational chemistry
• Cheminformatics
• Machine learning

Background:

• Machine learning (ML) is widely used for predicting biological activity from molecular structure.
• Concerns exist regarding the generalizability of ML models beyond their training data (chemical space).
• Simple similarity methods sometimes match complex ML models, questioning the learning of transferable structure-activity relationships (SAR).

Purpose of the Study:

• To systematically evaluate the impact of dataset-splitting strategies on ML model performance.
• To compare the generalizability of k-nearest neighbors (k-NN) against modern ML models.
• To assess the robustness of predictive models when encountering out-of-distribution (OOD) data.

Main Methods:

• Systematic examination of various dataset-splitting strategies.
• Performance evaluation of k-nearest neighbors (k-NN) and representative modern ML models.
• Comparison of model performance using standard molecular fingerprints and 3D geometric descriptors.

Main Results:

• k-NN models performed comparably to state-of-the-art ML methods across all splitting strategies.
• Predictive accuracy for all models sharply deteriorated as dataset splits became more stringent (i.e., more OOD).
• Performance collapse occurred even with 3D descriptors, suggesting inflated accuracies in prior studies using less rigorous splits.

Conclusions:

• Current ML models, including sophisticated ones, exhibit limited generalizability to novel chemical structures.
• Rigorous evaluation standards are crucial for assessing the true predictive power of ML models in drug discovery.
• Development of new molecular representations is needed to achieve true out-of-distribution generalization.